Facial temperature calculator

Abstract

To provide a face temperature calculation device capable of suppressing a temperature error due to disturbance and highly accurately calculating a face temperature.SOLUTION: A processor 162 of an ECU 160 includes a data acquisition part 162a for acquiring temperature distribution data in the vicinity of an occupant in a vehicle detected by an infrared sensor 140d, a face area determination part 162c determining a face area of the occupant based on the temperature distribution data, a division part 162d for dividing the face area into a plurality of blocks, a temperature determination part 162e for determining whether or not a temperature is within a prescribed area for every divided block based on the temperature distribution data, and a face temperature calculation part 162g for calculating a face temperature of the occupant based on the temperatures of the blocks, the temperatures of which are within the prescribed range.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a facial temperature calculation device. [Background technology]

[0002] Conventionally, in order to eliminate the influence of changes in the ambient temperature when measuring facial temperature with a thermograph, it is known to calibrate the subject's facial temperature based on the blackbody coverage of a calibrator photographed by a thermograph (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-254222 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when detecting the body temperature of a vehicle occupant using a thermograph, there is a problem that the detected body temperature of the occupant may contain errors due to exposure to sunlight or air from an air conditioner on the occupant. In addition, there is a problem that the detected body temperature of the occupant may contain errors due to vibrations caused by the vehicle behavior or changes in the posture of the occupant, etc. In the technology described in the above Patent Document 1, the measured value fluctuates due to the influence of these various disturbances, and the body temperature cannot be measured accurately.

[0005] In view of the above problems, an object of the present disclosure is to provide a facial temperature calculation device capable of suppressing temperature errors caused by disturbances and calculating facial temperature with high accuracy. [Means for solving the problem]

[0006] The gist of the present disclosure is as follows. (1) an acquisition unit that acquires temperature distribution data in the vicinity of a vehicle occupant detected by an infrared sensor; a face area determination unit that determines a face area of ​​an occupant based on the temperature distribution data; a dividing unit for dividing the face area into a plurality of blocks; a temperature determination unit that determines whether or not a temperature of each of the divided blocks is within a predetermined range based on the temperature distribution data; a face temperature calculation unit that calculates a face temperature of an occupant based on temperatures of the blocks whose temperatures are within the predetermined range; A facial temperature calculation device comprising:

[0007] (2) The face temperature calculation device described in (1) above, wherein the temperature determination unit uses the average value of the temperatures of all the blocks in the face area as a reference, and if the difference between the temperature of any block and the average value is less than or equal to a threshold value, determines that the temperature of the block is within the specified range.

[0008] (3) The face temperature calculation device described in (1) or (2) above, wherein the temperature determination unit obtains an average value or a median value of the temperature for each divided block, and determines whether the temperature for each block is within the specified range based on the average value or the median value.

[0009] (4) The face temperature calculation device according to any one of (1) to (3) above, wherein the face temperature calculation unit calculates the face temperature by averaging temperatures of the blocks within the predetermined range. Effect of the Invention

[0010] According to the present disclosure, a facial temperature calculation device is provided that is capable of suppressing temperature errors caused by disturbances and calculating facial temperature with high accuracy. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing a configuration of a vehicle control system mounted on a vehicle. [Diagram 2]FIG. 1 is a schematic diagram showing a state in which an image of a passenger sitting in a rear seat is captured by an infrared camera mounted on a headrest of a front seat of a vehicle. [Diagram 3] FIG. 3 is a schematic diagram showing a case where the reclining angle of the front seat in FIG. 2 is changed. [Figure 4] 1 is a schematic diagram showing a measurement range R1 of face surface temperature by an infrared sensor. [Diagram 5] 1 is a characteristic diagram showing how the maximum temperature value and the average temperature value in the measurement range R1 change over time. [Figure 6] 5 is a schematic diagram showing an example in which a measurement range R2 is provided in addition to the measurement range R1 shown in FIG. 4. FIG. [Figure 7] 1 is a characteristic diagram showing how the average temperature value in the measurement range R1 and the average temperature value in the measurement range R2 change over time. [Figure 8] 8 is a characteristic diagram showing a difference obtained by subtracting the average temperature value in the measurement range R2 from the average temperature value in the measurement range R1 in FIG. 7. [Figure 9] FIG. 2 is a schematic diagram showing functional blocks of a processor of an ECU provided in a vehicle. [Figure 10A] 4 and 6, this is a schematic diagram showing how an infrared sensor detects the temperature distribution on the surface of an occupant's face. [Figure 10B] FIG. 13 is an enlarged schematic diagram showing a measurement range R3. [Figure 10C] 13 is a schematic diagram showing how a measurement range R3 is divided into a plurality of blocks by a dividing section and how a temperature is determined for each block by a temperature determining section. FIG. [Figure 10D] 13 is a schematic diagram showing a block selected by a selection unit; FIG. [Figure 11A] 4 and 6, this is a schematic diagram showing how an infrared sensor detects the temperature distribution on the surface of an occupant's face. [Figure 11B] FIG. 13 is an enlarged schematic diagram showing a measurement range R3. [Figure 11C]13 is a schematic diagram showing how a measurement range R3 is divided into a plurality of blocks by a dividing section and how a temperature is determined for each block by a temperature determining section. FIG. [Figure 11D] 13 is a schematic diagram showing a block selected by a selection unit; FIG. [Figure 12] FIG. 1 is a schematic diagram showing an example of a neural network in which learning is performed based on a learning dataset to create a trained model. [Figure 13] 4 is a flowchart showing a process performed by a processor of the ECU. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, these descriptions are intended to merely exemplify preferred embodiments of the present invention, and are not intended to limit the present invention to such specific embodiments. In the following description, similar components are given the same reference numerals.

[0013] 1 is a schematic diagram showing the configuration of a vehicle control system 100 mounted on a vehicle. The vehicle control system 100 has a vehicle control device 120, one or more sensors 140, and an electronic control unit (ECU: Electronic Control Unit, hereinafter referred to as ECU) 160. The vehicle control device 120, the one or more sensors 140, and the ECU 160 are communicatively connected via an in-vehicle network conforming to standards such as a Controller Area Network (CAN) and Ethernet (registered trademark).

[0014] The vehicle control system 100 is mounted on a vehicle capable of driving with the driver releasing the steering wheel (hands-off driving), for example. For example, such driving is assumed to be at or above level 2 or level 3 as defined by the Society of Automotive Engineers (SAE). The vehicle may be an engine vehicle that uses an engine as a driving source, a plug-in hybrid vehicle (PHV), or an electric vehicle (EV).

[0015] The vehicle control device 120 is various devices related to vehicle control, and includes an engine 120a and a motor 120b as drive sources for running the vehicle, a transmission 120c, and a steering device 120d. The vehicle control device 120 may include a suspension capable of controlling damping force. Note that FIG. 1 illustrates an example of a PHV vehicle, and if the vehicle is an engine vehicle, the vehicle control device 120 does not include the motor 120b. Also, if the vehicle is an EV, the vehicle control device 120 does not include the engine 120a.

[0016] The one or more sensors 140 include a speed sensor 140a for detecting the speed of the vehicle, an acceleration sensor 140b for detecting the acceleration of the vehicle, a biometric sensor 140c for detecting biometric information of an occupant such as a driver of the vehicle, and a seating sensor 140e for detecting that the occupant is seated in a seat in the vehicle cabin. The biometric sensor 140c is a sensor for detecting biometric information such as an electrocardiogram, heart rate, body temperature, and amount of sweat of the occupant, and detects the biometric information in a contact or non-contact state with the occupant. The biometric sensor 140c is directed to an assumed position of the occupant in the vehicle, and includes an infrared sensor 140d (infrared thermography) for detecting the temperature distribution of the occupant's body temperature in a non-contact manner by detecting the amount of infrared energy emitted from the occupant. The one or more sensors 140 may include an imaging sensor for imaging the surroundings of the vehicle (for example, the front of the vehicle) or the occupant.

[0017] The ECU 160 includes a processor 162, a memory 164, and a communication interface 166. The processor 162 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 162 may further include other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. The memory 164 includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The communication interface 166 includes an interface circuit for connecting the ECU 160 to an in-vehicle network.

[0018] Vehicles that are capable of driving at Level 2 or Level 3 or higher as defined by the Society of Automotive Engineers (SAE) allow passengers, including the driver, to engage in activities other than driving. In this case, it is important that passengers can comfortably engage in activities other than driving in the vehicle without experiencing discomfort caused by driving the vehicle.

[0019] In this embodiment, a model is trained using machine learning to estimate the degree of discomfort (discomfort level) of an occupant caused by driving a vehicle based on biometric information representing the occupant's electrocardiogram, heart rate, breathing, body temperature, sweating, etc.

[0020] If the degree of discomfort of the occupant can be estimated by the trained model, the discomfort of the occupant can be suppressed by controlling the vehicle control device 120 to slow down the movement of the vehicle when the discomfort of the occupant is high. This allows the occupant to comfortably carry out activities other than driving inside the vehicle.

[0021] An example of discomfort felt by occupants while driving a vehicle is motion sickness. Another example of discomfort felt by occupants while driving a vehicle is discomfort caused by sudden acceleration, sudden deceleration, and sudden steering. The discomfort felt by occupants is likely to occur when there is a difference between the movement of the vehicle when the occupant manually drives the vehicle and the movement of the vehicle when the vehicle is automatically driven. In the following, the degree of motion sickness will be taken as an example of the discomfort felt by occupants while driving a vehicle, but the discomfort felt by occupants while driving a vehicle is not limited to the degree of motion sickness, and includes the discomfort felt by occupants caused by the difference in the movement between manual driving and automatic driving described above. In place of the discomfort felt by occupants, an index representing the comfort level of the occupants may be used.

[0022] Of the biometric information sensors 140c that detect the biometric information of the occupant, the infrared sensor 140d may be built in the infrared camera. Fig. 2 is a schematic diagram showing a state in which an occupant 212 seated in a rear seat 210 is photographed using an infrared camera 204 mounted on a headrest 202 of a front seat 200 of a vehicle. In the example shown in Fig. 2, a state in which a calibrator 216 mounted near a headrest 214 of the rear seat 210 is photographed together with the face of the occupant 212 seated in the rear seat 210 is shown.

[0023] In the configuration shown in Fig. 2, the infrared camera 204 captures an image of the face of the passenger 212 in the rear seat 210 as well as the calibrator 216. Meanwhile, Fig. 3 is a schematic diagram showing a case in which the reclining angle of the front seat 200 in Fig. 2 is changed. When the reclining angle of the front seat 200 is changed as shown in Fig. 3, or when the seat position of the front seat 200 is changed forward or backward, the calibrator 216 may fall outside the angle of view of the infrared camera 204. On the other hand, if the angle of view is widened so that the calibrator 216 does not fall outside the angle of view, an unnecessary heat source other than the face (sunlight, window, etc.) is captured, which causes a problem of decreasing the accuracy of temperature detection.

[0024] In addition, if a relative positional shift occurs between the temperature measurement range of infrared sensor 140d and the occupant's face due to vehicle vibration or a change in the occupant's posture, the measurement range may include the rear window or headrest behind the face, and the temperature of the rear window, headrest, etc. may be measured.

[0025] Furthermore, there are many temperature disturbances around the calibrator 216 due to radiant heat from sunlight, air conditioning, etc., and it is difficult to keep the calibrator 216 at a constant temperature inside a vehicle where such radiant heat is present. Note that radiant heat refers to all factors that cause errors in the measured temperature values, such as the amount of sunlight, the direction of sunlight, the outside air temperature, the temperature inside the vehicle, the air conditioning air volume, and the air conditioning air direction.

[0026] Under the above circumstances, even if the calibrator 216 performs calibration, the measured body temperature will contain an error. This may result in an erroneous estimation, such as an overestimation of the degree of motion sickness. In the following, a detailed description will be given of examples in which the measured body temperature contains an error, particularly when there is a relative positional shift between the temperature measurement range of the infrared sensor 140d and the face of the passenger, or when disturbance due to radiant heat occurs.

[0027] Fig. 4 is a schematic diagram showing a measurement range R1 of the face surface temperature by the infrared sensor 140d. In the example shown in Fig. 4, the face 300 of the passenger sitting in the back seat is photographed by the infrared camera 204 as in Fig. 2, and the temperature distribution of the face surface is detected by the infrared sensor 140d. In the example shown in Fig. 4, the temperature is detected within the measurement range R1.

[0028] The temperature detected within the measurement range R1 fluctuates due to changes in the relative position between the measurement range R1 and the face caused by vehicle vibrations, etc., and disturbances caused by radiant heat, etc. This causes a deviation between the maximum temperature detected within the measurement range R1 and the average temperature detected within the measurement range R1. Figure 5 is a characteristic diagram showing how the maximum temperature (characteristic C1 shown in Figure 5) and the average temperature (characteristic C2 shown in Figure 5) in the measurement range R1 change over time.

[0029] In the example shown in Fig. 4, solar radiation (sunlight) is hitting the right side of the occupant's face 300, and the temperature detected on the right side of the face 300 (the left side in the drawing) within the measurement range R1 is higher than the temperature detected on the left side of the face 300. For this reason, as shown in Fig. 5, the maximum temperature (characteristic C1) is generally higher than the average temperature (characteristic C2). Note that although Figs. 4 and 5 show a case where a calibrator is used, as will be described later, a calibrator may not be used when estimating the degree of motion sickness of an occupant using the amount of change in body temperature.

[0030] Also, as shown in FIG. 5, the maximum temperature (characteristic C1) includes variations due to noise caused by vehicle vibration, and the temperature is temporarily high (for example, around time 540 [sec], around time 570 [sec], around time 600 [sec], etc.). In this example, there is a rear window receiving sunlight behind the face 300, and when the face 300 moves relative to the measurement range R1 due to vehicle vibration, the measurement range R1 includes the rear window behind the face 300. Since the temperature of the rear window receiving sunlight is higher than the temperature of the face 300, as shown in FIG. 5, the maximum temperature includes noise and is temporarily high. As described above, when the face moves due to the influence of vehicle vibration, etc., noise occurs in the maximum temperature (mismeasurement of a strong heat source). Note that when the headrest is included in the measurement range R1 due to vehicle vibration, the maximum temperature temporarily decreases because the temperature of the headrest is lower than the body temperature.

[0031] On the other hand, in the average temperature value (characteristic C2), the effect of noise is reduced more than the maximum value (characteristic C1) because the measurement range R1 includes the area of ​​the face 300 in addition to the rear window. Therefore, by using the average value in the measurement range R1, the effect of noise caused by the relative positional shift between the temperature measurement range of the infrared sensor 140d and the face of the occupant is reduced.

[0032] However, even when the average temperature value (characteristic C2) is used, the influence of radiant heat cannot be eliminated. Moreover, the influence of radiant heat occurs even when there is no vehicle vibration. FIG. 6 is a schematic diagram showing an example in which a measurement range R2 is provided in addition to the measurement range R1 shown in FIG. 4. In the example shown in FIG. 6, solar radiation is also incident on the right side of the occupant's face 300. The measurement range R2 is set to be a range that is not affected by radiant heat due to solar radiation from the right side of the face 300.

[0033] FIG. 7 is a characteristic diagram showing how the average temperature value in measurement range R1 (characteristic C2 shown by a dashed line in FIG. 7) and the average temperature value in measurement range R2 (characteristic C3 shown by a solid line in FIG. 7) change over time. FIG. 8 is a characteristic diagram showing the difference obtained by subtracting the average temperature value in measurement range R2 (characteristic C3) from the average temperature value in measurement range R1 in FIG. 7 (characteristic C2). FIG. 8 shows how the difference obtained by subtracting characteristic C3 from characteristic C2 (characteristic C2' shown in FIG. 8) changes over time, with the average temperature value in measurement range R2 (characteristic C3) being used as the reference (=0).

[0034] As shown in Figures 7 and 8, the average temperature in the measurement range R1, which is affected by radiant heat, is higher than the average temperature in the measurement range R2, which is not affected by radiant heat. The value of the dashed line L in Figure 8 is the time-series average value of the characteristic C2', and is about 0.2°C higher than the average temperature in the reference measurement range R2. Therefore, it can be seen that if the measurement range includes a range affected by radiant heat, an error will be included in the detected body temperature. When estimating the degree of motion sickness of a passenger, if a measurement error of about 0.2°C occurs in the passenger's body temperature, the accuracy of estimating the motion sickness may decrease.

[0035] For this reason, in this embodiment, a correction process is performed on the temperature distribution data of the occupant's body temperature detected by the infrared sensor 140d, and data with large errors is excluded, thereby suppressing the effects of relative positional shift between the temperature measurement range of the infrared sensor 140d and the occupant's face and the effects of radiant heat, thereby improving the estimation accuracy of the facial surface temperature.

[0036] More specifically, in this embodiment, the face area detected by the infrared sensor 140d is divided into a plurality of blocks, and the temperature is acquired for each block. Then, the temperature of the passenger's face is calculated based on the temperature of the block whose temperature is within a predetermined range. By performing such processing, the face temperature is calculated based on the temperature of the block that is not affected by vehicle vibration and is not affected by radiant heat, so that the face temperature is calculated accurately and the accuracy of estimating the degree of the passenger's motion sickness is improved.

[0037] FIG. 9 is a schematic diagram showing functional blocks of the processor 162 of the ECU 160 provided in the vehicle. The processor 162 is one aspect of a face temperature calculation device, and includes a data acquisition unit 162a, a seating determination unit 162b, a face area determination unit 162c, a division unit 162d, a temperature determination unit 162e, a selection unit 162f, a face temperature calculation unit 162g, a motion sickness estimation unit 162h, and a vehicle control unit 162i. Each of these units in the processor 162 is a functional module realized by a computer program that runs on the processor 162, for example. That is, each of these units in the processor 162 is composed of the processor 162 and a program (software) for making it function. The program may be recorded in the memory 164 of the ECU 160 or a recording medium connected from the outside. Alternatively, each of these units in the processor 162 may be a dedicated arithmetic circuit provided in the processor 162.

[0038] The data acquisition unit 162a of the processor 162 acquires data on the temperature distribution in the vicinity of the occupant detected by the infrared sensor 140d. The seating determination unit 162b of the processor 162 determines whether or not the occupant, whose body temperature is to be measured, is seated in the seat, based on the output value of the seating sensor 140e.

[0039] The facial area determination unit 162c of the processor 162 determines whether or not there is a facial area of ​​an occupant (facial area) based on the temperature distribution data detected by the infrared sensor 140d. Based on the temperature distribution data, the facial area determination unit 162c determines an area where the temperature is around 36°C (for example, 36±2°C) and its surroundings as a facial area.

[0040] The dividing section 162d of the processor 162 divides the face region determined by the face region determining section 162c into a plurality of blocks of a predetermined size.

[0041] The temperature determination unit 162e of the processor 162 determines whether the temperature of each divided block is within a predetermined range based on the temperature distribution data. Specifically, the temperature determination unit 162e obtains a representative value of the temperature of each divided block (for example, the average or median of the temperatures in the block), and determines whether the temperature of each block is within a predetermined range based on the representative value.

[0042] For example, the temperature determination unit 162e uses the average value of the temperatures of all blocks in the face region as a reference, and determines that the temperature of a block in which the difference between the average value and the representative value of the temperature is equal to or less than a threshold value is within a predetermined range. In other words, when the difference between the temperature of any block and the average value is equal to or less than a threshold value, the temperature determination unit 162e determines that the temperature of the block is within a predetermined range. Also, for example, the temperature determination unit 162e classifies the representative values ​​of the temperatures of each block by clustering, and extracts a cluster with the best number of clusters on the dendrogram, or a cluster in which the distance between clusters on the dendrogram is equal to or greater than a predetermined value (or within a predetermined range). Then, of the clusters obtained in this way, the largest cluster, i.e., the cluster with the most representative values, is identified, and the temperature of the block in which the representative value belongs to the identified cluster is determined to be within a predetermined range.

[0043] The selection unit 162f of the processor 162 selects a block whose temperature is within a predetermined range based on the determination result of the temperature determination unit 162e. The face temperature calculation unit 162g of the processor 162 calculates the face temperature of the occupant based on the temperature of the block selected by the selection unit 162f. Specifically, the face temperature calculation unit 162g calculates the face temperature by averaging the temperatures of the blocks selected by the selection unit 162f.

[0044] 10A to 10D are schematic diagrams for explaining the processes performed by the dividing unit 162d, the temperature determining unit 162e, and the selecting unit 162f. FIG. 10A is a schematic diagram showing the state where the infrared sensor 140d detects the temperature distribution on the surface of the occupant's face 300, similar to FIG. 4 and FIG. 6. FIG. 10A shows the state where the occupant's face 300 is exposed to solar radiation (sunlight) from the right side, and the measurement range R3 of the face temperature includes the region 301 affected by the radiant heat due to the solar radiation, similar to FIG. 4 and FIG. 6. Note that the face region determined by the face region determining unit 162c is wider than the measurement range R3 shown in FIG. 10A and includes the entire face, but for convenience of explanation, the measurement range R3 shown in FIG. 10A shows a part of the face region. The process performed in the measurement range R3 described below is performed in the same manner in the entire face region.

[0045] Fig. 10B is a schematic diagram showing an enlarged view of measurement range R3. As shown in Fig. 10B, measurement range R3 includes face 300 and also includes area 301 on the surface of face 300 that is affected by radiant heat. For this reason, if the facial temperature is calculated based on the temperature distribution in measurement range R3, the calculated facial temperature will be higher than the actual facial temperature because the temperature of area 301 that is affected by radiant heat is higher than the actual facial temperature.

[0046] Fig. 10C is a schematic diagram showing how the measurement range R3 is divided into multiple blocks by the dividing unit 162d and the temperature is judged for each block by the temperature judging unit 162e. In Fig. 10C, the blocks marked with an OK mark are blocks whose temperatures are within a predetermined range, and the blocks marked with an NG mark are blocks whose temperatures exceed the predetermined range.

[0047] Fig. 10D is a schematic diagram showing blocks selected by the selection unit 162f. As shown in Fig. 10D, the selection unit 162f excludes blocks marked with NG and selects blocks marked with OK. The face temperature calculation unit 162g calculates the face temperature by averaging the temperatures of the blocks marked with OK selected by the selection unit 162f.

[0048] As described above, the blocks whose temperatures are not within the predetermined range are excluded, and the face temperature is calculated based on the temperatures of the blocks whose temperatures are within the predetermined range, so that the face temperature is calculated with high accuracy without being affected by radiant heat. Note that the area 301 affected by radiant heat does not occupy the entire measurement range R3, so the face temperature is calculated with high accuracy based on the temperatures of the blocks whose temperatures are within the predetermined range.

[0049] 11A to 11D are schematic diagrams for explaining another example of the processing performed by the dividing unit 162d, the temperature determining unit 162e, and the selecting unit 162f. FIG. 11A is a schematic diagram showing how the infrared sensor 140d detects the temperature distribution on the surface of the occupant's face 300, similar to FIG. 4 and FIG. 6. FIG. 11A shows how the face 300 is tilted to the left due to the vehicle behavior, and the background behind the face 300 (headrest 302) enters the measurement range R3 of the face temperature. Note that the face area determined by the face area determining unit 162c is wider than the measurement range R3 shown in FIG. 11A and includes the entire face, but for convenience of explanation, the measurement range R3 shown in FIG. 11A shows a part of the face area. The processing performed in the measurement range R3 described below is performed in the same manner in the entire face area.

[0050] Fig. 11B is a schematic diagram showing an enlarged view of measurement range R3. As shown in Fig. 11B, measurement range R3 includes face 300 and headrest 302 located to the right rear of face 300. Therefore, when the face temperature is calculated based on the temperature distribution in measurement range R3, the calculated face temperature will be lower than the actual face temperature because the temperature of headrest 302 is lower than the actual face temperature.

[0051] 11C is a schematic diagram showing how the measurement range R3 is divided into multiple blocks by the dividing unit 162d and the temperature is judged for each block by the temperature judging unit 162e. In FIG. 11C, the blocks marked with an OK mark are blocks whose temperatures are within a predetermined range, and the blocks marked with an NG mark are blocks whose temperatures exceed the predetermined range.

[0052] Fig. 11D is a schematic diagram showing blocks selected by the selection unit 162f. As shown in Fig. 11D, the selection unit 162f excludes blocks marked with NG and leaves them unselected, and selects blocks marked with OK. The face temperature calculation unit 162g calculates the face temperature by averaging the temperatures of the blocks marked with OK selected by the selection unit 162f.

[0053] As described above, blocks whose temperatures are not within the predetermined range are excluded and the face temperature is calculated based on the temperatures of blocks whose temperatures are within the predetermined range, so the face temperature can be calculated with high accuracy without being affected by the temperature of head restraint 302. Note that, since head restraint 302 does not occupy the entire measurement range R3, the face temperature can be calculated with high accuracy based on the temperatures of blocks whose temperatures are within the predetermined range.

[0054] The motion sickness estimation unit 162h of the processor 162 uses a trained model created in advance to estimate the degree of motion sickness of the occupant based on the biological information detected by the biological information sensor 140c. More specifically, the motion sickness estimation unit 162h estimates the degree of motion sickness of the occupant using a motion sickness index output by inputting the biological information into the trained model as an estimated value representing the degree of motion sickness of the occupant. Note that the face temperature calculated by the face temperature calculation unit 162g is used as the body temperature of the occupant included in the biological information.

[0055] FIG. 12 is a schematic diagram showing an example of a neural network (NN) in which learning is performed based on a learning data set and a trained model is created. The neural network shown in FIG. 12 is already known, so an overview will be given here. The circles in FIG. 12 represent artificial neurons, and in neural networks, these artificial neurons are usually called nodes or units (called nodes here). In FIG. 12, L=1 indicates an input layer, L=2 and L=3 indicate hidden layers, and L=4 indicates an output layer. As shown in FIG. 12, the input layer (L=1) is composed of five nodes, and input values ​​x1, x2, x3, x4, and x5 are input to each node of the input layer (L=1). Meanwhile, FIG. 12 shows two layers, a hidden layer (L=2) and a hidden layer (L=3) having six nodes, but the number of layers of these hidden layers can be one or any number, and the number of nodes present in these hidden layers can also be any number. z 21 , z 22 , z 23 , z 24 , z 25 and z 26 indicates the output value from each node in the hidden layer (L=2), and z 31 , z 32 , z 33 , z 34 , z 35 and z 36 indicates the output value from each node in the hidden layer (L = 3). The number of nodes in the output layer (L = 4) is set to one, and the output value from the node in the output layer is indicated by y.

[0056] At each node in the input layer (L=1), the input values ​​x1, x2, x3, x4, and x5 are output as is. On the other hand, at each node in the hidden layer (L=2), the output values ​​x1, x2, x3, x4, and x5 of each node in the input layer are input, and at each node in the hidden layer (L=2), the total input value u is calculated using the corresponding weight w and bias b. 2k =Σx·w+b(k=1~6) is calculated.

[0057] Then, this total input value u2k is transformed by the activation function f, and z 2k From each node, the output value z 2k (=f(u 2k On the other hand, each node in the hidden layer (L=3) receives the output value z 21 , z 22 , z 23 , z 24 , z 25 and z 26 is input, and at each node in the hidden layer (L=3), the total input value u is calculated using the corresponding weight w and bias b. 3k =Σz·w+b(k=1~6) is calculated. The total input value u 3k is similarly transformed by the activation function, and the output value z 31 , z 32 , z 33 , z 34 , z 35 and z 36 As an activation function, for example, a sigmoid function σ(x) = 1 / (1 + exp(-x)) or a rectified linear function (ReLU) S(u) = max(0,u) is used.

[0058] On the other hand, the nodes in the output layer (L = 4) are assigned the output values ​​z 31 , z 32 , z 33 , z 34 , z 35 and z 36 is input, and at the output layer nodes, the total input value u is calculated using the corresponding weights w and bias b. 4k =Σz·w+b is calculated, or the total input value is calculated using only the corresponding weights w. 4k In this example, the identity function is used in the output layer nodes, and the total input value u calculated at the output layer node is 4k is output as is as the output value y. Note that the number of nodes in the output layer may be two or more.

[0059] The learning data set is the input values ​​x1, x2, x3, x4, x5 and the training data y for the input values ​​x1, x2, x3, x4, x5. t For a certain input value, the training data y t is being calculated, and the output value from the output layer (L=4) for this input value is y. If the squared error is used as the error function, the squared error E is E=(1 / 2)·(yy t ) 2 It can be calculated as follows.

[0060] During training, the input values ​​contained in the training dataset are input to the neural network, and the resulting output value y is compared with the training data y t The squared error E is calculated from the above. Then, in order to minimize the sum of the squared errors E obtained from multiple learning datasets, the weights w and biases b of each node are calculated by performing operations such as backpropagation and stochastic gradient descent, and a trained model is created.

[0061] If the learning target is the degree of motion sickness of the passenger, information such as the passenger's electrocardiogram, heart rate, body temperature, etc., and the passenger's facial expression, body movement, etc. are acquired as input values ​​x1, x2, x3, x4, and x5, and a motion sickness index according to the degree of motion sickness felt by the passenger is acquired as teacher data, and is made into one data set. The electrocardiogram is input as RRI (RR Interval), which is a fluctuating time series data, and its HF component and LF component may be input. The temperature calculated by the face temperature calculation unit 162g is input as the body temperature. The result of classifying the facial image obtained by photographing the passenger's face by known template matching or the like, and the degree of eye opening obtained from the facial image are input as the facial expression of the passenger. The head acceleration representing the movement of the passenger's head is input as the body movement. The head acceleration is obtained by photographing the passenger's head and analyzing the movement of the image representing the head. As described above, the biological information used for learning may include facial expressions, body movements, and the like of the occupant in addition to the electrocardiogram, heart rate, body temperature, and the like detected by the biological information sensor 140c.

[0062] By performing machine learning using the learning dataset created in the above manner, a trained model is created that takes the occupant's biometric information (electrocardiogram, body temperature, facial expression, body movement) as input and outputs a motion sickness index.

[0063] The input values ​​x1, x2, x3, x4, and x5 may be the amount of change per predetermined time. When the amount of change per predetermined time of the temperature calculated by the face temperature calculation unit 162g as the body temperature is input, the error calibrated by the calibrator is cancelled, so that the motion sickness index can be calculated with high accuracy without using the calibrator.

[0064] Although a neural network is illustrated in FIG. 12, a recurrent neural network (RNN) may be used instead of the neural network, and further, a long short term memory (LSTM) network, which is a type of RNN, may be used.

[0065] The vehicle control unit 162i of the processor 162 controls the vehicle control device 120 based on the degree of motion sickness of the occupant estimated by the motion sickness estimation unit 162h, i.e., the motion sickness index. The vehicle control unit 162i controls the engine 120a, the motor 120b, or the transmission 120c so that the absolute value of the acceleration or jerk of the vehicle becomes smaller as the motion sickness index of the occupant increases, i.e., as the occupant feels motion sickness, the vehicle control unit 162i also controls the steering device 120d so as to suppress abrupt steering as the motion sickness index of the occupant increases. Furthermore, when the vehicle control device 120 includes a suspension capable of controlling the damping force, the vehicle control unit 162i controls the suspension so that the damping force becomes larger as the motion sickness index of the occupant increases. These controls make the ride of the vehicle milder, thereby mitigating the motion sickness of the occupant.

[0066] Fig. 13 is a flowchart showing the processing performed by the processor 162 of the ECU 160. The processing of Fig. 13 is performed by the processor 162 at each predetermined control cycle. First, the seating determination unit 162b of the processor 162 determines whether or not an occupant, whose body temperature is to be measured, is seated in the seat based on the output value of the seating sensor 140e (step S10). If an occupant is seated, the process proceeds to step S12, and if an occupant is not seated, the process waits in step S10.

[0067] In step S12, a request to acquire temperature distribution data is transmitted to the infrared sensor 140d. Next, the data acquisition unit 162a of the processor 162 acquires the temperature distribution data detected by the infrared sensor 140d (step S14).

[0068] Next, the face area determination unit 162c determines whether or not a face area is present based on the temperature distribution data detected by the infrared sensor 140d (step S16). If a face area is present, the division unit 162d of the processor 162 divides the face area into multiple blocks (step S18). If a face area is not present, the acquisition of temperature distribution data is stopped (step S30), and the process in this control cycle is terminated.

[0069] After step S18, the temperature determination unit 162e of the processor 162 determines whether or not there is a block whose temperature exceeds the predetermined range based on the temperature distribution data (step S20). If there is a block whose temperature exceeds the predetermined range, the selection unit 162f of the processor 162 excludes the block whose temperature exceeds the predetermined range and selects a block whose temperature is within the predetermined range (step S22).

[0070] On the other hand, if there is no block whose temperature exceeds the predetermined range, the selection unit 162f selects all blocks in the face region (step S23).

[0071] Next, the face temperature calculation unit 162g of the processor 162 calculates the face temperature based on the temperature of the block selected by the selection unit 162f (step S24).

[0072] Next, the motion sickness estimation unit 162h of the processor 162 uses the trained model created in advance to estimate the degree of motion sickness of the occupant based on the biological information detected by the biological information sensor 140c (step S26). The value calculated in step S24 is used as the body temperature of the occupant included in the biological information.

[0073] Next, the vehicle control unit 162i of the processor 162 controls the vehicle control device 120 based on the degree of motion sickness of the occupant estimated by the motion sickness estimation unit 162h (step S28).

[0074] In the above-described embodiment, an example of estimating the motion sickness of an occupant based on the calculated face temperature has been shown, but the present disclosure is not limited to this example. For example, the vehicle interior environment (temperature, etc.) may be optimally controlled based on the face temperature, or the health condition of the occupant may be estimated based on the face temperature.

[0075] As described above, according to this embodiment, the facial area of ​​the occupant is divided into a plurality of blocks, and the facial temperature is calculated based on the temperature of the blocks that are within a predetermined range. Therefore, the inclusion of errors in the facial temperature due to the influence of vehicle vibration or radiant heat is suppressed, and the facial temperature is detected with high accuracy. [Explanation of symbols]

[0076] 100 Vehicle Control System 120 Vehicle control equipment 120a Engine 120b motor 120c derailleur 120d steering gear 140 Sensors 140a Speed ​​Sensor 140b Acceleration sensor 140c Biometric Sensor 140d Infrared Sensor 140e Seat Sensor 160 Electronic Control Unit (ECU) 162 processors 162a Data Acquisition Section 162b Seating judgment unit 162c Face area determination unit 162d Division 162e Temperature judgment section 162f Selection section 162g Face temperature calculation unit 162h Estimation part 162i Vehicle control unit 164 Memory 166 Communication Interface 200 Front seat 202,214,302 Headrest 204 Infrared Camera 210 Back seat 212 Crew 216 Calibrator 300 faces

Claims

1. an acquisition unit that acquires temperature distribution data in the vicinity of a vehicle occupant detected by an infrared sensor; a face area determination unit that determines a face area of ​​an occupant based on the temperature distribution data; a dividing unit for dividing the face area into a plurality of blocks; a temperature determination unit that determines whether or not a temperature of each of the divided blocks is within a predetermined range based on the temperature distribution data; a face temperature calculation unit that calculates a face temperature of an occupant based on temperatures of the blocks whose temperatures are within the predetermined range; Equipped with A facial temperature calculation device in which the temperature determination unit uses the average temperature of all the blocks in the facial area as a reference, and if the difference between the temperature of any block and the average temperature is less than a threshold value, determines that the temperature of the block is within the specified range.

2. The face temperature calculation device according to claim 1 , wherein the temperature determination unit obtains an average value or a median value of the temperature for each of the divided blocks, and determines whether the temperature for each of the blocks is within the predetermined range based on the average value or the median value.

3. The facial temperature calculation device according to claim 1 , wherein the facial temperature calculation unit calculates the facial temperature by averaging temperatures of the blocks within the predetermined range.

Images