Vehicle, vehicle control method, and computer program

Abstract

To appropriately determine an abnormal physical condition of a subject.SOLUTION: A vehicle includes: extraction means for extracting an amount of feature of a subject from an image of the subject in the vehicle; determination means for determining an abnormal physical condition of the subject by using the amount of feature; and display means having a first area capable of displaying information on the abnormal physical condition, including at least one of a cause and a counter-measure of the abnormal physical condition, and a second area capable of displaying at least one of information on the subject and an instruction to the subject. The first area and the second area are provided side by side.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the technical fields of vehicles, vehicle control methods, and computer programs. [Background technology]

[0002] Known systems of this type determine the state of a subject based on an image of the subject. For example, Patent Document 1 discloses a technology for estimating a driver's state from the face position, face orientation, gaze direction, etc. acquired by analyzing a facial image. Patent Document 2 discloses a technology for detecting the face position from a facial image and estimating the driver's state using the degree of eye opening / closing, gaze direction, face orientation, etc.

[0003] As another related technique, for example, Patent Document 3 discloses a technique for detecting a feature point vector from a plurality of feature points extracted from face image data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 209225 [Patent Document 2] International Publication No. 2017 / 208529 [Patent Document 3] International Publication No. 2016 / 013090 Summary of the Invention [Problem to be solved by the invention]

[0005] If an attempt is made to determine the state of a subject by directly using features such as the position of the face, the direction of the face, and the direction of gaze, it is expected that the state will be affected by, for example, individual differences and environmental variations such as lighting conditions. In this case, the accuracy of the determination may decrease depending on the environment when the state is determined. In other words, the technologies described in Patent Documents 1, 2, and 3 have a technical problem in that the state of the subject cannot be accurately determined depending on the situation.

[0006] This disclosure has been made in consideration of the above-mentioned problems, and aims to provide a vehicle, a vehicle control method, and a computer program that can appropriately determine abnormalities in the physical condition of a subject. [Means for solving the problem]

[0007] One aspect of the vehicle disclosed herein comprises an extraction means for extracting features of a subject from an image of the subject inside the vehicle, a determination means for determining an abnormal physical condition of the subject using the features, and a display means having a first area capable of displaying information about the abnormal physical condition, including at least one of the cause of the abnormal physical condition or a countermeasure, and a second area capable of displaying at least one of information about the subject or instructions to the subject, the first area and the second area being arranged side by side.

[0008] One aspect of the vehicle control method of this disclosure is a vehicle control method equipped with a display means, which extracts features of a subject from an image of the subject inside the vehicle, determines whether the subject has an abnormal physical condition using the features, displays information regarding the abnormal physical condition, including at least one of a cause of the abnormal physical condition or a countermeasure, in a first area of the display means, and displays at least one of information regarding the subject or instructions to the subject in a second area of the display means and arranged alongside the first area.

[0009] One aspect of the computer program of this disclosure is to cause at least one computer to execute a vehicle control method equipped with a display means, which extracts features of a subject inside the vehicle from an image of the subject, determines whether the subject has an abnormal physical condition using the features, displays information regarding the abnormal physical condition, including at least one of a cause of the abnormal physical condition or a countermeasure, in a first area of the display means, and displays at least one of information regarding the subject or instructions to the subject in a second area of the display means that is arranged alongside the first area. [Effects of the Invention]

[0010] According to one aspect of each of the above-described vehicle, vehicle control method, and computer program, it is possible to appropriately determine whether a subject has an abnormal condition. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing the overall configuration of an abnormal physical condition determination system according to a first embodiment. [Figure 2] FIG. 10 is a block diagram showing the overall configuration of a modified example of a system for determining abnormal physical condition. [Figure 3] 1 is a block diagram showing the hardware configuration of an abnormal physical condition determination system according to a first embodiment. [Figure 4] 3 is a flowchart showing the flow of operations of the abnormal physical condition determination system according to the first embodiment. [Figure 5] 10 is a table showing specific examples of feature vectors. [Figure 6] FIG. 10 is a block diagram showing the overall configuration of an abnormal physical condition determination system according to a second embodiment. [Figure 7] 10 is a schematic diagram showing the interior configuration of a vehicle to which a physical condition abnormality determination system according to a second embodiment is applied. FIG. [Figure 8] FIG. 2 is a diagram illustrating an example of a display area of a display unit. [Figure 9] FIG. 10 is a diagram (part 1) showing an example of a display in the display area when car sickness occurs. [Figure 10]FIG. 2 is a diagram (part 2) showing an example of a display in the display area when car sickness occurs. [Figure 11] FIG. 10 is a diagram showing an example of an icon indicating the degree of car sickness. [Figure 12] 10A and 10B are diagrams showing other display examples in the display area of the display unit. [Figure 13] FIG. 10 is a block diagram showing the overall configuration of an abnormal physical condition determination system according to a third embodiment. [Figure 14] FIG. 10 is a block diagram showing the overall configuration of an abnormal physical condition determination system according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of a system for determining an abnormal physical condition, a method for operating the system for determining an abnormal physical condition, a computer program, and a vehicle will be described with reference to the drawings.

[0013] First Embodiment An abnormal physical condition determination system according to a first embodiment will be described with reference to Figures 1 to 5. In the following, an example will be described in which the abnormal physical condition determination system is a device that determines abnormal physical conditions (specifically, motion sickness) of a vehicle passenger.

[0014] (System Configuration) First, the overall configuration of an abnormal physical condition determination system according to the first embodiment will be described with reference to Figures 1 and 2. Figure 1 is a block diagram showing the overall configuration of an abnormal physical condition determination system according to the first embodiment. Figure 2 is a block diagram showing the overall configuration of an abnormal physical condition determination system according to a modified example.

[0015] As shown in Fig. 1, the abnormal physical condition determination system 1 according to the first embodiment is mounted on a vehicle 10 equipped with an in-vehicle camera 50. The abnormal physical condition determination system 1 includes, as functional blocks for realizing its functions, an image acquisition unit 101, a subject detection unit 102, a feature extraction unit 103, a feature accumulation unit 104, a feature relation calculation unit 105, and an abnormal physical condition determination unit 106.

[0016] The image acquisition unit 101 is configured to be able to acquire images captured by the in-vehicle camera 50 (specifically, images showing passengers in the vehicle 10). The images acquired by the image acquisition unit 101 are configured to be output to the target person detection unit 102.

[0017] The subject detection unit 102 is configured to be able to detect a subject for whom an abnormal physical condition is to be determined from the image acquired by the image acquisition unit 101. The subject detection unit 102 detects, for example, an area in which the subject is captured from the captured image (for example, the subject's face area, etc.). Note that as a specific method for detecting the subject from the captured image, existing technology can be appropriately adopted, and therefore a detailed description thereof will be omitted here. The detection result of the subject detection unit 102 is configured to be output to the feature extraction unit 103.

[0018] The feature extraction unit 103 is configured to be able to extract feature amounts for determining abnormal physical conditions of the subject from the area detected by the subject detection unit 102. Examples of feature amounts extracted by the feature extraction unit 103 include feature amounts related to the position of the occupant's eyes, face direction, gaze direction, facial color, position of facial feature points, and eyelid opening degree. The feature extraction unit 103 extracts multiple types of feature amounts from the occupant. The feature amounts extracted by the feature extraction unit 103 are configured to be output to each of the feature accumulation unit 104 and the feature relation calculation unit 105.

[0019] The feature accumulation unit 104 is configured to accumulate the feature amounts extracted by the feature extraction unit 103 as time-series data. The feature accumulation unit 104 accumulates the feature amounts, for example, in units of frames arranged in time series. The feature amounts accumulated in the feature accumulation unit 104 are configured to be output to the feature relation calculation unit 105 as appropriate.

[0020] The feature relation calculation unit 105 is configured to be able to calculate the relationship between each feature (i.e., the relationship between different types of feature) from time-series data of multiple types of feature. More specifically, the feature relation calculation unit 105 calculates the relationship between each feature from the feature extracted by the feature extraction unit 103 (in other words, the current feature) and the feature read from the feature accumulation unit 104 (in other words, the past feature). The relationship between the feature is calculated as, for example, a feature vector having multiple dimensions. The relationship between the feature calculated by the feature relation calculation unit 105 is configured to be output to the abnormal physical condition determination unit 106.

[0021] The abnormal physical condition determination unit 106 determines whether the passenger is experiencing an abnormal physical condition based on the relationship between the feature quantities calculated by the feature relation calculation unit 105. The abnormal physical condition determination unit 106 determines, for example, whether the passenger is experiencing an abnormal physical condition or the degree of the passenger's abnormal physical condition. The abnormal physical condition determination unit 106 can be determined by pre-setting the correlation between the relationship between the feature quantities and the abnormal physical condition. For example, a threshold value for the feature vector may be set, and if the calculated feature vector exceeds the threshold value, it may be determined that an abnormal physical condition has occurred. The abnormal physical condition determination unit 106 may also be configured to learn the threshold value for determination. For example, the abnormal physical condition determination unit 106 may perform learning using correct answer data obtained by input from the passenger.

[0022] 2, the abnormal physical condition determination system 1 may be provided in an external server 20 located outside the vehicle 10. In other words, the abnormal physical condition determination system 1 does not necessarily have to be mounted on the vehicle 10. Note that, instead of all the components of the abnormal physical condition determination system 1, some of the components of the abnormal physical condition determination system 1 may be provided in the external server 20.

[0023] (Hardware configuration) Next, the hardware configuration of the abnormal physical condition determination system 1 according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the hardware configuration of the abnormal physical condition determination system according to the first embodiment.

[0024] 3, the abnormal physical condition determination system 1 according to the first embodiment includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and a storage device 14. The abnormal physical condition determination system 1 may further include an input device 15 and an output device 16. The CPU 11, RAM 12, ROM 13, storage device 14, input device 15, and output device 16 are connected via a data bus 17.

[0025] The CPU 11 reads a computer program. For example, the CPU 11 is configured to read a computer program stored in at least one of the RAM 12, the ROM 13, and the storage device 14. Alternatively, the CPU 11 may read a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The CPU 11 may obtain (i.e., read) the computer program from a device (not shown) located outside the abnormal physical condition determination system 1 via a network interface. The CPU 11 controls the RAM 12, the storage device 14, the input device 15, and the output device 16 by executing the read computer program. In particular, in this embodiment, when the CPU 11 executes the read computer program, a functional block for determining abnormal physical condition is realized within the CPU 11.

[0026] The RAM 12 temporarily stores computer programs executed by the CPU 11. The RAM 12 temporarily stores data that is temporarily used by the CPU 11 while the CPU 11 is executing the computer programs. The RAM 12 may be, for example, a D-RAM (Dynamic RAM).

[0027] The ROM 13 stores computer programs executed by the CPU 11. The ROM 13 may also store fixed data. The ROM 13 may be, for example, a P-ROM (Programmable ROM).

[0028] The storage device 14 stores data that is to be saved over the long term by the abnormal physical condition determination system 1. The storage device 14 may operate as a temporary storage device for the CPU 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0029] The input device 15 is a device that receives input instructions from a user of the abnormal physical condition determination system 1. The input device 15 may include, for example, at least one of a keyboard, a mouse, and a touch panel.

[0030] The output device 16 is a device that outputs information related to the abnormal physical condition determination system 1 to the outside. For example, the output device 16 may be a display device (for example, a display) that can display information related to the abnormal physical condition determination system 1.

[0031] (Operation flow) Next, the flow of operations of the abnormal physical condition determination system 1 according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of operations of the abnormal physical condition determination system according to the first embodiment.

[0032] 4, during operation of the abnormal physical condition determination system 1 according to the first embodiment, the image acquisition unit 101 first acquires an image of the occupant captured by the in-vehicle camera 50 (step S101). Then, the subject detection unit 102 detects a face region from the image of the occupant (step S102). Note that when detecting feature amounts from an area other than the face region, it is sufficient to detect another area corresponding to the feature amount.

[0033] Next, the feature extraction unit 103 extracts feature amounts from the face region (step S103), and the feature accumulation unit 104 accumulates (that is, stores) the extracted feature amounts (step S104).

[0034] Next, the feature relation calculation unit 105 determines whether feature quantities for a predetermined number of frames have been accumulated in the feature 2201 accumulation unit 104 (step S105). Note that the "predetermined number of frames" here refers to a value corresponding to a sufficient number of feature quantities for calculating the relationship between feature quantities, and an optimal value (for example, the number of frames for 5 seconds) is set in advance. If it is determined that feature quantities for the predetermined number of frames have not been accumulated (step S105: NO), the process is executed again from step S101. That is, the process of extracting feature quantities from the image of the passenger and accumulating them is repeated.

[0035] On the other hand, if it is determined that the feature amounts have been accumulated for the predetermined number of frames (step S105: YES), the feature relation calculation unit 105 calculates the relation between the feature amounts (step S106).Then, the physical condition abnormality determination unit 106 determines the physical condition abnormality (here, motion sickness) of the passenger based on the calculated relation between the feature amounts (step S107).

[0036] The above-described series of processes are executed while the vehicle 10 is traveling. The abnormal physical condition determination system 1 may start operating, for example, when the vehicle 10 starts traveling. Alternatively, the system may start operating when a predetermined time (the time when it is estimated that the passenger's physical condition will begin to become abnormal) has elapsed since the vehicle 10 started traveling. Here, the predetermined time may be set arbitrarily. Alternatively, the time from the start of traveling until the passenger's physical condition begins to become abnormal may be stored as historical information and set based on this historical information. In this way, unnecessary processing during periods when the passenger does not experience any abnormal physical condition can be reduced. Alternatively, the abnormal physical condition determination system 1 may start operating when the passenger's face can be detected normally or when the passenger gets into the vehicle. In this way, it is possible to track the passenger's face, thereby preventing a situation in which the passenger's face cannot be detected even though the passenger is experiencing an abnormal physical condition. Alternatively, the system may start operating based on vehicle behavior information (for example, when the threshold value of the acceleration sensor becomes equal to or greater than a predetermined threshold). In this way, unnecessary processing can be reduced when the vehicle is behaving in a manner that is unlikely to cause an abnormal physical condition.

[0037] Furthermore, the abnormal physical condition determination system 1 ends its operation, for example, when the vehicle 10 stops traveling. Alternatively, it may end its operation based on vehicle behavior information (for example, when the threshold value of the acceleration sensor becomes equal to or less than a predetermined threshold value).

[0038] (Example of feature vector) Next, the feature vectors used in the abnormal physical condition determination system 1 according to the first embodiment (i.e., the relationships between the feature quantities generated in step S106 in FIG. 4) will be specifically described with reference to FIG. 5. FIG. 5 is a table showing specific examples of feature vectors.

[0039] 5, it is assumed that the feature extraction unit 103 extracts the pupil positions of the passenger's right and left eyes, the facial direction angle (pan, tilt, and roll directions), and the gaze angle (horizontal and vertical directions) as feature quantities. In this case, the feature relation calculation unit 105 calculates the relationship between the above-mentioned multiple types of feature quantities as a multidimensional feature vector.

[0040] Specifically, the feature relation calculation unit 105 calculates an eight-dimensional feature vector including (1) the correlation between the face position and the right eye pupil position, (2) the correlation between the face position and the left eye pupil position, (3) the correlation between the face direction pan direction and the horizontal gaze direction, (4) the correlation between the face direction tilt direction and the vertical gaze direction, (5) the ratio of the movement speed between the face center position and the right eye position, (6) the ratio of the movement speed between the face center position and the left eye position, (7) the ratio of the angular velocity between the face direction pan direction and the horizontal gaze direction, and (8) the ratio of the angular velocity between the face direction tilt direction and the vertical gaze direction. Note that, taking (1) as an example, the correlation is the value obtained by dividing the covariance of the face position and the right eye pupil by the product of the standard deviation of the face position and the standard deviation of the right eye pupil.

[0041] Each element of the feature vector may be weighted appropriately. In this case, the weight may be determined according to the state of the face estimated from the image of the face region. Specifically, if the passenger is wearing glasses, the accuracy of the feature values acquired regarding the eyes is likely to be reduced, so the weight of the feature values regarding the eyes may be reduced. Furthermore, the feature vector does not need to be eight-dimensional, and may have an appropriately set number of dimensions.

[0042] The abnormal physical condition determination unit 106 determines that the passenger is in abnormal physical condition based on the relationship between the calculated feature quantities. Specifically, if the correlations and ratios calculated by the feature relation calculation unit 105 are equal to or greater than predetermined values, the passenger is determined to be in abnormal physical condition. When an eight-dimensional feature vector including the above (1) to (8) is calculated, the passenger may be determined to be in abnormal physical condition if any one or more correlations or ratios are equal to or greater than predetermined values, or the passenger may be determined to be in abnormal physical condition if all eight correlations and ratios are equal to or greater than their corresponding predetermined values.

[0043] When motion sickness occurs, the passenger experiences rotational vertigo and dizziness when opening the eyes. Meanwhile, the above-described feature vectors (1) to (8) are calculated as the relationship between pairs of feature quantities that normally change in the same way. Therefore, the above-described feature vectors can identify situations where the facial and eye movements are no longer in sync due to dizziness, and can appropriately determine the occurrence of motion sickness.

[0044] The above-described feature vectors are merely examples, and other feature relationship may be used. For example, a pair of feature relationships for calculating the relationship may be determined depending on the type of abnormal physical condition to be determined.

[0045] (Technical Effects) Next, the technical effects obtained by the abnormal physical condition determination system 1 according to the first embodiment will be described.

[0046] As described in FIGS. 1 to 5, the physical condition abnormality determination system 1 according to the first embodiment can determine whether a passenger is experiencing motion sickness based on feature amounts extracted from an image of the passenger. In particular, in this embodiment, the extracted feature amounts are not directly used, but rather motion sickness is determined based on the relationship between multiple feature amounts. This makes it possible to suppress the influence of environmental variations such as individual differences and lighting conditions, and to determine whether a passenger is experiencing motion sickness with high accuracy. Furthermore, because the feature amounts are time-series data, it is possible to take into account changes in the feature amounts per unit time, unlike when feature amounts are extracted from only one image. This allows for more accurate motion sickness determination.

[0047] In the above-described embodiment, an example of determining motion sickness in a passenger of the vehicle 10 has been described. However, motion sickness in a passenger of a moving body other than the vehicle 10, such as a ship or an airplane, may also be determined. Furthermore, a physical abnormality other than motion sickness may also be determined. Specifically, a determination may be made regarding dizziness (so-called VR sickness) that may occur when using a VR (Virtual Reality) device. By using features related to the subject's face as in this embodiment, physical abnormalities, particularly those accompanied by dizziness, can be appropriately determined. However, by utilizing the relationship between appropriate features, this method can also be applied to determining other physical abnormalities not accompanied by dizziness.

[0048] Second Embodiment Next, an abnormal physical condition determination system 1 according to a second embodiment will be described with reference to Fig. 6 to Fig. 12. The second embodiment differs from the first embodiment already described only in part of its configuration and operation, and the remaining parts are generally similar. Therefore, the following will describe in detail the parts that differ from the first embodiment, and will omit a description of the other overlapping parts as appropriate.

[0049] (System Configuration) First, the overall configuration of the abnormal physical condition determination system 1 according to the second embodiment will be described with reference to Figures 6 and 7. Figure 6 is a block diagram showing the overall configuration of the abnormal physical condition determination system according to the second embodiment. Figure 7 is a schematic diagram showing the interior configuration of a vehicle to which the abnormal physical condition determination system according to the second embodiment is applied. In Figure 6, the same components as those shown in Figure 1 are assigned the same reference numerals.

[0050] As shown in FIG. 6, the abnormal physical condition determination system 1 according to the second embodiment includes a determination result output unit 107 in addition to the components of the abnormal physical condition determination system 1 according to the first embodiment (see FIG. 1).

[0051] The abnormal physical condition determination unit 106 may identify the cause of the abnormal physical condition of the passenger from information indicating the behavior of the vehicle 10. In this case, the abnormal physical condition determination unit 106 may be configured to be able to acquire information indicating the behavior of the vehicle 10 from a sensor (e.g., an acceleration sensor, etc.) not shown that is provided on the vehicle 10. The abnormal physical condition determination unit 106 may also identify the cause of the abnormal physical condition of the passenger from the analysis results of an image of the passenger. In this case, the abnormal physical condition determination unit 106 may be configured to be able to extract feature quantities that can determine the cause of the abnormal physical condition from an image including the entire body of the passenger. In addition, an abnormal physical condition cause determination unit not shown may identify the cause of the abnormal physical condition of the passenger.

[0052] For example, if a passenger is determined to be in an abnormal physical condition, the abnormal physical condition determination unit 106 acquires the acceleration of the vehicle at the time of the determination. The abnormal physical condition determination unit 106 compares the acquired acceleration with a pre-stored threshold (e.g., an acceleration that is likely to cause an abnormal physical condition) to determine whether the passenger's abnormal physical condition is caused by the acceleration of the vehicle 10. The acquired acceleration may be the acceleration at a timing a predetermined period before the time when the passenger is determined to be in an abnormal physical condition. Alternatively, it may be the acceleration at a predetermined period before the time when the passenger is determined to be in an abnormal physical condition. Furthermore, instead of or in addition to the acceleration, at least one of the velocity, angular velocity, yaw angular velocity, yaw angular acceleration, pitch angular velocity, pitch angular acceleration, roll angular velocity, and roll angular acceleration may be acquired. The cause may be identified based on at least one of these, or may be identified based on all of them.

[0053] The abnormal physical condition determination unit 106 may identify the cause of the abnormal physical condition without comparing the detected acceleration, etc. with a threshold value. For example, it can be determined that the acceleration detected at the time when the abnormal physical condition is determined is the cause of the abnormal physical condition. Therefore, the acceleration, etc. detected at the time when the abnormal physical condition is determined may be identified as the cause of the abnormal physical condition as it is.

[0054] Furthermore, for example, if a passenger is determined to be in an abnormal physical condition, the abnormal physical condition determination unit 106 identifies the passenger's behavior (e.g., reading, eating, drinking, etc.) from the camera image at the time of the determination. The abnormal physical condition determination unit 106 determines whether the passenger's abnormal physical condition is caused by the passenger's behavior based on whether the identified behavior corresponds to a pre-stored cause of the abnormal physical condition (e.g., table information on behaviors that are likely to cause abnormal physical conditions). The camera image used to identify the cause may be one taken a predetermined period before the time when the passenger is determined to be in an abnormal physical condition. Alternatively, the camera image may be one taken during a predetermined period before the time when the passenger is determined to be in an abnormal physical condition. Furthermore, the table information on behaviors that are likely to cause abnormal physical conditions may associate a degree of abnormal physical condition with each behavior. Alternatively, multiple behaviors may be acquired, and the cause may be identified based on at least one of these, or all of these.

[0055] The determination result output unit 107 is configured to be able to output the determination result of the abnormal physical condition determination unit 106 to the display unit 301 mounted on the vehicle 10. That is, the determination result output unit 107 outputs information relating to the abnormal physical condition of the passenger to present to the passenger or a fellow passenger. Note that the determination result output unit 107 may be configured to be able to output information relating to the abnormal physical condition of the passenger to a speaker or the like (not shown).

[0056] 7, display unit 301 is configured as a display or the like provided at the front of the vehicle. In-vehicle camera 50 is provided as camera 50a capable of capturing images of the faces of passengers 201 in the driver's seat and passenger seat, and camera 50b capable of capturing images of the faces of passengers 202 in the rear seat.

[0057] (Example of display on the display unit) Next, examples of display by the abnormal physical condition determination system 1 according to the second embodiment (i.e., specific examples of display on the display unit 301) will be described with reference to Figs. 8 to 12. Fig. 8 is a diagram showing an example of the display area of the display unit. Fig. 9 is a diagram (part 1) showing an example of display in the display area when car sickness occurs. Fig. 10 is a diagram (part 2) showing an example of display in the display area when car sickness occurs. Fig. 11 is a diagram showing an example of an icon indicating the degree of car sickness. Fig. 12 is a diagram showing another example of display in the display area of the display unit.

[0058] The display area of the display unit 301 may include a cause and countermeasure information display area. Furthermore, the display area of the display unit 301 may also include a passenger information display area. An example is shown in FIG. 8. The cause and countermeasure information display area on the left displays at least one of the cause and countermeasure information depending on the motion sickness diagnosis result. If no passengers are experiencing motion sickness, the cause and countermeasure information display area need not display anything. The passenger information display area on the right displays information about passengers in the vehicle 10 and information about instructions for the passengers. In this example, there are three passengers, one in the driver's seat, one in the passenger seat, and one in the left rear seat, and therefore, an icon indicating the presence of a passenger (circle icon in the figure) is displayed in each seat.

[0059] In the example shown in FIG. 9, the passenger in the left rear seat is experiencing motion sickness. Therefore, the icon corresponding to the passenger in the left rear seat in the passenger information display area is changed to an icon indicating motion sickness. In addition, the arrow icon (icon at the front of the vehicle) in the passenger information display area is lit. This icon indicates that it is recommended to stop the vehicle on the shoulder of the road. In such a case, a driving route for parking the vehicle 10 on the shoulder of the road or in a parking space may be displayed in conjunction with a navigation device.

[0060] The cause and countermeasure information display area displays the cause of motion sickness (here, braking and reading) of the passenger in the left rear seat. As described above, the cause of motion sickness can be identified by the abnormal physical condition determination unit 106 or a physical condition abnormality cause identification unit (not shown). If there are multiple passengers suffering from motion sickness, the cause and countermeasure information display area can be divided according to the number of passengers, and corresponding displays can be made for each passenger. Although not shown here, more specific instructions, such as "Driver, please be careful not to brake" or "Passenger in the left rear seat, please stop reading," can be displayed as countermeasures for the abnormal physical condition based on the identified cause of the abnormal physical condition.

[0061] The display of the cause and the countermeasures may use information from a template prepared in advance. The template information may be at least one piece of display information, such as the cause of the cause, such as "sudden braking" or "reading," or instructions, such as "driver, please drive carefully" or "passengers, please refrain from reading, eating, or drinking." The template information may also be an icon image indicating the cause or the countermeasures. Furthermore, when an abnormality in the passenger's physical condition is detected, the template information may be used without identifying the cause of the abnormality.

[0062] Alternatively, in addition to or instead of the display on the display unit 301, audio guidance may be provided.

[0063] The example shown in Fig. 10 is a display example after the situation shown in Fig. 9 occurs and the vehicle 10 has stopped on the shoulder. In the passenger information display area, the arrow icon that was lit in Fig. 9 is turned off because the vehicle 10 has stopped on the shoulder. Meanwhile, the arrow icon near the passenger in the left rear seat is lit. This arrow icon is an icon that recommends that the passenger in the left rear seat get out of the vehicle 10.

[0064] The cause and countermeasure information display area displays the instruction "Go outside and take a deep breath." In other words, countermeasures to reduce motion sickness are displayed. Note that the display example here is merely an example, and other countermeasures may be displayed. For example, the display may encourage the passenger to look out the window, adjust their seat so that they can see outside more easily, move to the passenger seat, or take motion sickness medication. Alternatively, the display may be linked to a navigation device and display a driving route suitable for implementing countermeasures. Specifically, the display may display an option for the passenger as to whether motion sickness medication is necessary and accept the passenger's response (the question may be asked by voice and the passenger's response may be accepted using voice recognition). If the passenger responds that motion sickness medication is necessary, the display may display a driving route to a nearby pharmacy.

[0065] One of the causes of motion sickness is the inability of passengers other than the driver to predict the behavior of the vehicle 10. To address this, if there is a passenger experiencing motion sickness, a display or audio guidance regarding the behavior of the vehicle 10 may be provided. For example, audio guidance may be provided on the vehicle 10 as to which direction the vehicle 10 will turn in a certain number of seconds, based on information about the road the vehicle is traveling on. Alternatively, the direction the vehicle 10 will turn may be displayed on the display unit 301, or the number of seconds remaining until the vehicle 10 will move may be displayed.

[0066] As shown in FIG. 11, the passenger icon displayed in the passenger information display area changes depending on the degree of motion sickness. For example, in Example 1, the facial expression changes depending on the degree of motion sickness. In Example 2, the icon's shade and color change depending on the degree of motion sickness. When changing the color of the icon, it is preferable to use a color that allows the passenger to intuitively grasp the urgency. Specifically, the color may be changed to "green" if the degree of motion sickness is mild, "yellow" if it is moderate, and "red" if it is severe. Note that although an example in which the icon changes in three stages is shown here, the number of types of icons may be increased or decreased depending on the degree, for example, to five stages.

[0067] As shown in Fig. 12, the cause and countermeasure information display area may be displayed as icons. The icons in the figure indicate the causes of motion sickness, and the icons in the top row indicate, from left to right, "reading," "sudden braking," and "not looking outside." The icons in the bottom row indicate, from left to right, "eating and drinking," "sudden acceleration," and "vibration." In the example of Fig. 12, "reading" and "sudden braking" are estimated to be the causes of motion sickness, so these icons are lit.

[0068] (Technical Effects) Next, the technical effects obtained by the abnormal physical condition determination system 1 according to the second embodiment will be described.

[0069] As described in Figures 6 to 12, the abnormal physical condition determination system 1 according to the second embodiment can accurately inform the passengers of the vehicle 10 of their current condition by displaying the motion sickness determination result. In addition, by displaying the causes of motion sickness and countermeasures, it becomes easier to alleviate motion sickness. Note that, like the first embodiment, the second embodiment can also be applied to motion sickness that occurs in moving objects other than the vehicle 10, and to physical abnormalities other than motion sickness.

[0070] <Third embodiment> Next, an abnormal physical condition determination system 1 according to a third embodiment will be described with reference to Fig. 13. The third embodiment differs only in part of the configuration and operation from the first and second embodiments already described, and the remaining parts are generally similar. Therefore, the following will describe in detail the parts that differ from the first and second embodiments, and will omit a description of the other overlapping parts as appropriate.

[0071] (System Configuration) First, the overall configuration of an abnormal physical condition determination system 1 according to the third embodiment will be described with reference to Fig. 13. Fig. 13 is a block diagram showing the overall configuration of an abnormal physical condition determination system according to the third embodiment. In Fig. 13, the same components as those shown in Fig. 1 are denoted by the same reference numerals.

[0072] As shown in FIG. 13, the abnormal physical condition determination system 1 according to the third embodiment includes a control information output unit 108 in addition to the components of the abnormal physical condition determination system 1 according to the first embodiment (see FIG. 1).

[0073] The control information output unit 108 generates and outputs control information for controlling the vehicle 10 based on the result of the abnormal physical condition determination. The control information output from the control information output unit 108 is input to a vehicle control unit 302 of the vehicle 10. The vehicle control unit 302 is a control unit capable of controlling each part of the vehicle 10, and executes control in accordance with the control information.

[0074] (Control content) Next, the control content executed based on the control information output from the control information output unit 108 (in other words, the control content executed by the vehicle control unit 302) will be specifically described.

[0075] The control information output unit 108 outputs, for example, control information related to the autonomous driving of the vehicle 10. Specifically, the control information output unit 108 may output control information that removes low-frequency motion in a motion control unit that controls the motion of the vehicle 10. In this way, it is possible to suppress minute vibrations that cause motion sickness. The control information output unit 108 may output control information that avoids sudden braking or sudden acceleration. The control information output unit 108 may output control information that causes the vehicle 10 to slowly stop on the shoulder of the road. When the vehicle 10 is traveling at autonomous driving level 3 or higher (i.e., a level at which the driver is almost not involved in driving), the control information output unit 108 may output control information that lowers the autonomous driving level to one of 0 to 2 (i.e., a level at which the driver is involved in driving) and switches to a state where the driver can stop on the shoulder of the road.

[0076] The control information output unit 108 may also output control information for controlling the air conditioning in the vehicle cabin. That is, information for changing the air condition in the vehicle cabin to a state that is less likely to cause motion sickness may be output. Alternatively, the control information output unit 108 may output control information for adjusting the position of a movable navigation display unit or a movable smartphone holder in the vehicle cabin to a height that allows the passenger to see the outside scenery when looking at the display screen. That is, information for changing the passenger's field of vision to a state that is less likely to cause motion sickness may be output.

[0077] (Technical Effects) Next, the technical effects obtained by the abnormal physical condition determination system 1 according to the third embodiment will be described.

[0078] 13, the abnormal physical condition determination system 1 according to the third embodiment can realize a situation in which motion sickness is easily alleviated or is less likely to occur by executing control according to the motion sickness determination result. Note that, like the first and second embodiments, the third embodiment can also be applied to motion sickness occurring in moving objects other than the vehicle 10, and to physical condition abnormalities other than motion sickness.

[0079] <Fourth embodiment> Next, an abnormal physical condition determination system 1 according to a fourth embodiment will be described with reference to Fig. 14. The fourth embodiment differs only in part of the configuration and operation from the first to third embodiments already described, and the remaining parts are generally similar. Therefore, the following will describe in detail the parts that differ from the first to third embodiments, and will omit a description of the other overlapping parts as appropriate.

[0080] (System Configuration) First, the overall configuration of an abnormal physical condition determination system 1 according to the fourth embodiment will be described with reference to Fig. 14. Fig. 14 is a block diagram showing the overall configuration of an abnormal physical condition determination system according to the fourth embodiment. In Fig. 14, the same components as those shown in Fig. 1 are denoted by the same reference numerals.

[0081] 14, the abnormal physical condition determination system 1 according to the fourth embodiment includes a feature extraction unit 103, a feature accumulation unit 104, a feature relation calculation unit 105, and an abnormal physical condition determination unit 106. That is, the abnormal physical condition determination system 1 according to the fourth embodiment does not include the image acquisition unit 101 and the subject detection unit 102, which are components of the abnormal physical condition determination system 1 according to the first embodiment (see FIG. 1).

[0082] The feature extraction unit 103 according to the fourth embodiment is configured to acquire an image captured by the in-vehicle camera 50 and extract, from the image, feature amounts for determining whether the subject has an abnormal physical condition. Note that the components other than the feature extraction unit 103 (i.e., the feature accumulation unit 104, the feature relation calculation unit 105, and the abnormal physical condition determination unit 106) are configured in the same manner as those in the first to third embodiments.

[0083] In particular, in the fourth embodiment, unlike the first to third embodiments already described, the subject detection unit 102 does not detect the subject (for example, detect the subject's facial area, etc.). For this reason, it is preferable that the image acquired by the feature extraction unit 103 includes an area suitable for extracting features (for example, an image captured by enlarging the area around the subject's face). To capture such an image, for example, the imaging range of the in-vehicle camera 50 may be adjusted in advance to an appropriate range, or the in-vehicle camera 50 may be provided with a function for automatically adjusting the imaging range. Alternatively, the feature extraction unit 103 may have a function for automatically detecting an area suitable for extracting features from the captured image and extracting features from that area.

[0084] (Technical Effects) Next, the technical effects obtained by the abnormal physical condition determination system 1 according to the fourth embodiment will be described.

[0085] As explained in Fig. 14, the abnormal physical condition determination system 1 according to the fourth embodiment can determine whether a passenger is experiencing motion sickness based on the relationship between feature amounts extracted from an image of the passenger, similar to the first embodiment. Furthermore, the system configuration is particularly simplified in the fourth embodiment, making it possible to reduce the cost and size of the device. Similar to the first to third embodiments, the fourth embodiment can also be applied to motion sickness occurring in moving objects other than the vehicle 10, and to physical abnormalities other than motion sickness.

[0086] <Additional Notes> The above-described embodiment may be further described as follows, but is not limited to the following.

[0087] (Appendix 1) The abnormal physical condition determination system described in Appendix 1 is characterized by comprising an extraction means for extracting multiple features indicating the condition of a subject from an image of the subject, a storage means for storing the multiple features as time-series data, a calculation means for calculating the relationship between each of the multiple features stored in the storage means, and a determination means for determining the abnormal physical condition of the subject based on the relationship.

[0088] (Appendix 2) The abnormal physical condition determination system described in Appendix 2 is the abnormal physical condition determination system described in Appendix 1, characterized in that the calculation means calculates the relationship from the degree of change per unit time of the multiple feature amounts.

[0089] (Appendix 3) The abnormal physical condition determination system described in Appendix 3 is the abnormal physical condition determination system described in Appendix 1 or 2, characterized in that the calculation means calculates the correlation or ratio between different feature amounts as the relationship.

[0090] (Appendix 4) The abnormal physical condition determination system described in Appendix 4 is the abnormal physical condition determination system described in any one of Appendixes 1 to 3, characterized in that the multiple features include features related to the subject's pupil position, facial direction, or gaze direction.

[0091] (Appendix 5) The abnormal physical condition determination system described in Appendix 5 is a system for determining abnormal physical condition described in any one of Appendixes 1 to 4, characterized in that the relationship includes at least one of a correlation between face position and pupil position, a correlation between face direction pan direction and horizontal gaze direction, and a correlation between face direction tilt direction and vertical gaze direction.

[0092] (Appendix 6) The abnormal physical condition determination system described in Appendix 6 is a system for determining abnormal physical condition described in any one of Appendixes 1 to 5, characterized in that the relationship includes at least one of the ratio of the movement speed between the face center of gravity position and the eye position, the ratio of the angular velocity between the face direction pan direction and the horizontal line of sight, and the ratio of the angular velocity between the face direction tilt direction and the vertical line of sight.

[0093] (Appendix 7) The abnormal physical condition determination system described in Appendix 7 is a system for determining abnormal physical condition described in any one of Appendixes 1 to 6, characterized in that the determination means determines and outputs the degree of the abnormal physical condition based on the relationship.

[0094] (Appendix 8) The abnormal physical condition determination system described in Appendix 8 is a system for determining abnormal physical condition described in any one of Appendixes 1 to 7, characterized in that it further includes a presentation means for presenting information regarding the abnormal physical condition of the subject according to the determination result of the determination means to at least one of the subject and people around the subject.

[0095] (Appendix 9) The abnormal physical condition determination system described in Appendix 9 is a system described in any one of Appendixes 1 to 8, characterized in that the subject is a passenger in a vehicle, and further comprises an output means for outputting control information for executing control of the vehicle in accordance with the abnormal physical condition based on the determination result of the determination means.

[0096] (Appendix 10) The abnormal physical condition determination system described in Supplementary Note 10 is the abnormal physical condition determination system described in any one of Supplementary Notes 1 to 9, characterized in that the abnormal physical condition is a state of motion sickness.

[0097] (Appendix 11) The method for determining an abnormal physical condition described in Appendix 11 is a method for determining an abnormal physical condition, characterized in that multiple feature amounts indicating the condition of a subject are extracted from an image of the subject, the multiple feature amounts are accumulated as time-series data, the relationship between each of the multiple accumulated feature amounts is calculated from the multiple accumulated feature amounts, and the abnormal physical condition of the subject is determined based on the relationship.

[0098] (Appendix 12) The computer program described in Appendix 12 is a computer program characterized by operating a computer to extract multiple feature amounts indicating a condition of a subject from an image of the subject, accumulate the multiple feature amounts as time-series data, calculate a relationship between each of the multiple accumulated feature amounts, and determine an abnormality in the subject's physical condition based on the relationship.

[0099] This disclosure may be modified as appropriate within the scope of the claims and the gist or idea of the invention that can be read from the entire specification, and vehicles, vehicle control methods, and computer programs that involve such modifications are also included in the technical idea of this disclosure. [Explanation of symbols]

[0100] 1. Abnormal physical condition detection system 10 vehicles 11 CPU 12 RAM 13 ROM 14 Storage device 15 Input Devices 16 Output Devices 17 Data Bus 20 External Servers 50 In-car camera 101 Image acquisition unit 102 Target detection unit 103 Feature Extraction Unit 104 Feature Storage Unit 105 Feature Relation Calculation Unit 106 Abnormal Physical Condition Judgment Unit 107 Judgment result output unit 108 Control information output unit 201,202 passengers 301 Display section 302 Vehicle control unit

Claims

1. extraction means for extracting feature amounts of a subject from an image of the subject inside a vehicle; a determination means for determining an abnormal physical condition of the subject using the feature amount; A display means having a first area capable of displaying information about the abnormal physical condition, including at least one of the causes of the abnormal physical condition and countermeasures, and a second area capable of displaying at least one of information about the subject or instructions to the subject; Equipped with The first region and the second region are provided side by side, the display means displays information about the abnormal physical condition after the vehicle has stopped. A vehicle characterized by:

2. The display means displays at least one of a cause of the abnormal physical condition or a countermeasure as information regarding the abnormal physical condition, together with at least one of information regarding the subject or instructions to the subject.

2. The vehicle according to claim 1 .

3. The information regarding the abnormal physical condition includes an option as to whether or not measures to address the abnormal physical condition are necessary.

3. The vehicle according to claim 1 or 2.

4. the display means, when it is selected that the countermeasure is necessary, displays a driving route suitable for implementing the countermeasure; 4. The vehicle according to claim 3.

5. The display means displays the traveling direction of the vehicle when the determination means determines that the subject is in an abnormal physical condition.

5. A vehicle according to claim 1, wherein the vehicle is a vehicle having a plurality of sprockets.

6. the determination means determines a cause of the abnormal physical condition based on information indicating the behavior of the vehicle or a movement of the subject.

6. A vehicle according to any one of claims 1 to 5.

7. A method for controlling a vehicle equipped with a display means, comprising: extracting a feature of the subject from an image of the subject inside the vehicle; determining an abnormal physical condition of the subject using the feature amount; displaying information about the abnormal physical condition, including at least one of a cause of the abnormal physical condition and a countermeasure, in a first area of the display means after the vehicle has stopped; displaying at least one of information about the subject or instructions to the subject in a second area provided in the display means and arranged alongside the first area; A vehicle control method comprising:

8. At least one computer A computer program for executing control of a vehicle equipped with a display means, extracting features of a subject from an image of the subject inside a vehicle; determining an abnormal physical condition of the subject using the feature amount; displaying information about the abnormal physical condition, including at least one of a cause of the abnormal physical condition and a countermeasure, in a first area of the display means after the vehicle has stopped; displaying at least one of information about the subject or instructions to the subject in a second area provided in the display means and arranged alongside the first area; A computer program that executes a vehicle control method.

Images