Information processing device, information processing method and non-transitory storage medium

Abstract

A control device, comprising: a living body presence / absence judgment unit, configured to judge whether a predetermined living body remains in a cabin of a vehicle; a cabin camera, configured to acquire image data containing a face of the predetermined living body remaining in the cabin; a facial expression analysis information acquisition unit, configured to acquire information about the facial expression of the predetermined living body based on the image data; a judgment unit, configured to judge whether the life safety of the predetermined living body is at or below a predetermined benchmark based on the information about the facial expression; and a control unit, configured to activate a safety function for ensuring the life safety of the predetermined living body when the judgment unit judges that the life safety of the predetermined living body is at or below a predetermined benchmark.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and a non-transitory storage medium. Background Art

[0002] Hitherto, there has been known a technology for detecting living bodies such as humans and pets remaining in a cabin of a vehicle and ensuring the life safety of the living bodies (for example, see Japanese Unexamined Patent Application Publication No. 2020-141489 (JP2020-141489A).

[0003] In JP2020-141489A, a decrease in the life safety of a living body is determined based on the temperature in the cabin of the vehicle or the time that has elapsed after the vehicle is locked, and a warning alarm is output. Therefore, the life safety of the living body is ensured. Summary of the invention

[0004] However, there are individual differences between living bodies. For this reason, even when the temperature in the cabin of the vehicle or the time elapsed after the vehicle is locked, etc. falls within the reference range, the life safety of a specific living body remaining in the cabin of the vehicle may not be ensured.

[0005] According to the above aspect, it is possible to ensure the life safety of the living body remaining in the vehicle in consideration of individual differences of the living body.

[0006] One aspect of the present disclosure relates to an information processing device. The information processing device includes: a first judgment unit configured to judge whether there is a predetermined living body remaining in a cabin of a vehicle; a facial expression information acquisition unit configured to acquire information about the facial expression of the face of the predetermined living body based on image data containing the face of the predetermined living body remaining in the cabin of the vehicle; a second judgment unit configured to judge whether the life safety of the predetermined living body is lower than or equal to a predetermined benchmark based on the information about the facial expression of the predetermined living body; and a control unit configured to activate a safety function for ensuring the life safety of the predetermined living body when the second judgment unit judges that the life safety of the predetermined living body is lower than or equal to the predetermined benchmark.

[0007] According to the above aspect, the information processing device can use information about the facial expression of the face of the predetermined living body remaining in the cabin of the vehicle. Therefore, the information processing device can grasp the state unique to the predetermined living body, etc. based on the facial expression of the face of the predetermined living body, and judge the degree of decline in the life safety of the predetermined living body. Thus, the information processing device can ensure the life safety of the living body remaining in the vehicle in consideration of the individual differences of the predetermined living body.

[0008] In the above aspect, the information processing device may also include an estimation unit, which is configured to estimate the degree of decline in the life safety of the predetermined living being based on the information about the facial expression of the predetermined living being acquired by the facial expression information acquisition unit, and the second judgment unit may be configured to judge whether the life safety of the predetermined living being is lower than or equal to the predetermined benchmark based on the estimation result of the estimation unit.

[0009] According to the above aspect, the information processing device is able to estimate the degree of decline in the life safety of the predetermined living body in consideration of individual differences of the predetermined living body by using information on the facial expression of the face of the predetermined living body remaining in the cabin of the vehicle. Therefore, when the information processing device determines whether the life safety of the predetermined living body is lower than or equal to a predetermined reference by using the estimation result, the information processing device is able to ensure the life safety of the living body remaining in the vehicle in consideration of the individual differences of the predetermined living body.

[0010] In the above aspect, the information about the facial expression of the predetermined living being may include information about the emotion of the predetermined living being; and the estimation unit may be configured to estimate the degree of decrease in the life safety of the predetermined living being so that, when the information about the emotion of the predetermined living being represents a negative emotion, the degree of decrease in the life safety of the predetermined living being is greater than when the information about the emotion of the predetermined living being represents a positive emotion or a neutral emotion.

[0011] According to the above aspect, the information processing apparatus can estimate the degree of decline in the life safety of the predetermined living being remaining in the vehicle specifically by capturing the emotion of the predetermined living being in consideration of individual differences of the predetermined living being.

[0012] In the above aspect, the information about the facial expression of the predetermined living being may include information about the facial state of the predetermined living being; and the estimation unit may be configured to estimate the degree of decrease in the life safety of the predetermined living being so that, when the information about the facial state of the predetermined living being represents a negative facial state, the degree of decrease in the life safety of the predetermined living being is greater than when the information about the facial state of the predetermined living being represents a positive facial state or a neutral facial state of the predetermined living being.

[0013] According to the above aspect, the information processing apparatus can estimate the degree of decline in the life safety of a predetermined living being remaining in a vehicle in consideration of individual differences of the predetermined living being specifically by capturing the facial state of the predetermined living being.

[0014] In the above aspect, the estimation unit can be configured to estimate the degree of decline in the life safety of the predetermined living being so that the decline in the life safety of the predetermined living being accelerates as the information about the emotion of the predetermined living being changes from a state representing the positive emotion or the neutral emotion to a state representing the negative emotion, or as the information about the facial state of the predetermined living being changes from a state representing the positive facial state or the neutral facial state to a state representing the negative facial state.

[0015] According to the above aspect, the information processing device can estimate the degree of decline in the life safety of the predetermined living being by capturing a situation in which the decline in the life safety of the predetermined living being is accelerated with a negative change in the emotion or facial state of the predetermined living being. Therefore, the information processing device can estimate the degree of decline in the life safety of the predetermined living being by specifically capturing the negative change in the emotion or facial state of the predetermined living being while taking into account the individual differences of the predetermined living being.

[0016] In the above aspect, the estimation unit may be configured to estimate the degree of decrease in the life safety of the predetermined living being so that the life safety of the predetermined living being decreases as the time that the predetermined living being has stayed in the cabin of the vehicle elapses; and the estimation unit may be configured to estimate the degree of decrease in the life safety of the predetermined living being so that, when the information about the emotion of the predetermined living being represents the negative emotion, the rate of decrease in the life safety of the predetermined living being relative to the elapse of time is higher than the rate of decrease in the life safety of the predetermined living being relative to the elapse of time when the information about the emotion of the predetermined living being represents the positive emotion or the neutral emotion, and when the information about the facial state of the predetermined living being represents the negative facial state, the rate of decrease in the life safety of the predetermined living being relative to the elapse of time is higher than the rate of decrease in the life safety of the predetermined living being relative to the elapse of time when the information about the facial state of the predetermined living being represents the positive facial state or the neutral facial state.

[0017] According to the above aspect, the information processing device can variably change the rate of decrease of the life safety of the predetermined living being depending on the difference of the emotion or facial state of the predetermined living being, assuming that the life safety of the predetermined living being decreases with the lapse of time that the predetermined living being has stayed in the cabin of the vehicle. Therefore, the information processing device can specifically make the decrease of the life safety of the predetermined living being accelerate with the negative change of the emotion or facial state of the predetermined living being to estimate the degree of decrease of the life safety of the predetermined living being.

[0018] In the above aspect, the information about the facial expression of the predetermined living being may include information about the emotion of the predetermined living being; and the estimation unit may be configured to estimate the degree of decrease in the life safety of the predetermined living being so that, when the information about the emotion of the predetermined living being represents any one type of negative emotion among a plurality of predefined types of negative emotions, the degree of decrease in the life safety of the predetermined living being is different depending on the type of negative emotion represented among the plurality of types of negative emotions.

[0019] According to the above aspect, even when the physical condition or the like of the predetermined living body expressed by the emotion may differ depending on the type of the negative emotion, the information processing apparatus can further appropriately estimate the degree of decline in the life safety of the predetermined living body.

[0020] In the above aspect, the information about the facial expression of the predetermined living being may include information about the age of the predetermined living being; and the estimation unit may be configured to estimate the degree of decline in the life safety of the predetermined living being so that the degree of decline in the life safety of the predetermined living being is different depending on the age of the predetermined living being.

[0021] According to the above aspect, even when characteristics such as physical health may differ depending on the age of the predetermined living body, the information processing apparatus can further appropriately estimate the degree of decline in the life safety of the predetermined living body.

[0022] In the above aspects, the safety function for ensuring the life safety of the predetermined living body may include at least one of the following: a function of issuing an alarm about the predetermined living body being left behind to the surroundings of the vehicle through a sound output device installed on the vehicle, a function of providing a notification about the predetermined living body being left behind to a terminal device used by a user of the vehicle, a function of providing a notification about the predetermined living body being left behind to a predetermined facility set outside the vehicle, and a function of activating the air conditioning device of the vehicle.

[0023] According to the above aspect, the information processing apparatus can specifically ensure the life safety of a predetermined living body.

[0024] Another aspect of the present disclosure may be implemented as an information processing method or a non-transitory storage medium.

[0025] According to the above aspect, it is possible to ensure the life safety of the living body remaining in the vehicle in consideration of individual differences of the living body. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described below with reference to the accompanying drawings, wherein like numerals represent like elements, and wherein:

[0027] Figure 1 A schematic diagram showing a first example of a living body legacy monitoring system;

[0028] Figure 2 A diagram showing an example of a hardware configuration of a vehicle;

[0029] Figure 3 is a functional block diagram showing an example of a functional configuration of a vehicle;

[0030] Figure 4 is a table illustrating one example of a method of selecting an evaluation index;

[0031] Figure 5 is a table illustrating one example of a method of selecting an evaluation index;

[0032] Figure 6 It is a graph showing a specific example of the time-series change of the evaluation index.

[0033] Figure 7 A flowchart schematically showing an example of a living body left-over monitoring process;

[0034] Figure 8 A diagram showing a second example of a living body legacy monitoring system;

[0035] Fig. 9 is a functional block diagram showing another example of the functional configuration of a vehicle;

[0036] Fig.10 A diagram showing a third example of a living body legacy monitoring system;

[0037] Fig.11 A diagram showing an example of a hardware configuration of a server; and

[0038] Fig.12 A functional block diagram showing an example of a functional configuration of a server. DETAILED DESCRIPTION

[0039] Hereinafter, embodiments will be described with reference to the accompanying drawings.

[0040] First example of a live body legacy monitoring system

[0041] Will refer to Figures 1 to 7 A first example of the living body remaining monitoring system according to the present embodiment is described.

[0042] Overview of the Living Body Remaining Monitoring System

[0043] Figure 1 1 is a schematic diagram showing a first example of the living body remaining monitoring system 1 .

[0044] The living body left-behind monitoring system 1 monitors the degree of decline in the life safety of a predetermined living body to be monitored (hereinafter referred to as a "monitored living body") remaining in the cabin of the vehicle 100. When the life safety of the monitored living body remaining in the cabin of the vehicle 100 is lower than or equal to a predetermined reference, the living body left-behind monitoring system 1 activates a function for ensuring the life safety of the monitored living body (hereinafter referred to as a "safety function"). The details of the safety function will be described later.

[0045] Examples of monitored living bodies include humans. Examples of monitored living bodies also include pets.

[0046] like Figure 1 As shown in , the living body remaining monitoring system 1 includes a vehicle 100 , a server 200 , and a user terminal 300 .

[0047] The number of vehicles 100 included in the living body remaining monitoring system 1 may be one or may be plural. In other words, the living body remaining monitoring system 1 can monitor the degree of decline in the life safety of the monitored living body remaining in each vehicle 100.

[0048] The vehicle 100 has a function of judging whether there is a predetermined monitored living body left in the cabin, and, when there is a monitored living body, monitoring the degree of decline in the life safety of the monitored living body. When the life safety of the monitored living body left in the cabin is lower than or equal to a predetermined benchmark, the vehicle 100 activates the safety function.

[0049] The vehicle 100 is equipped with a communication device 50 (described later) and is connected to an external device (e.g., a server 200, a user terminal 300, or a communication system of a police station PS, etc.) through a predetermined communication line to be able to communicate with the external device. With this configuration, the vehicle 100 can send a signal including information about a monitored living body to the server 200 or the user terminal 300, or can provide a notification about a left-behind monitored living body to the police station PS as will be described later.

[0050] Examples of the predetermined communication line include a mobile communication network with a base station as a terminal, a satellite communication network using a communication satellite, and a wide area network (WAN) such as an Internet network. Examples of the predetermined communication line may include a local area network (LAN). Examples of the predetermined communication line may include a near field communication line based on a wireless communication standard, such as Wi-Fi and Bluetooth (registered trademark).

[0051] The server 200 is installed in, for example, a monitoring center CT that monitors the state of the vehicle 100, etc. The server 200 is connected to the vehicle 100 through a predetermined communication line to be able to communicate with the vehicle 100.

[0052] The server 200 functions as an interface between an operator OP residing in, for example, the monitoring center CT and the vehicle 100 or an occupant of the vehicle 100. With this configuration, the operator OP can talk to the occupant of the vehicle 100 by phone, or can check information included in a signal transmitted from the vehicle 100 by voice or screen.

[0053] The user terminal 300 is a terminal device used (owned) by a user of the vehicle 100. Examples of the user of the vehicle 100 include the owner of the vehicle 100 and the family of the owner of the vehicle 100. Examples of the user terminal 300 include mobile (portable) terminal devices such as a smartphone, a tablet terminal, and a laptop computer terminal, etc. Examples of the user terminal 300 include fixed terminal devices such as a desktop computer terminal, etc.

[0054] The user terminal 300 is connected to the vehicle 100 through a predetermined communication line to be able to communicate with the vehicle 100. With this configuration, the user terminal 300 can receive a signal transmitted from the vehicle 100 and cause the user (user of the vehicle 100) to grasp the information contained in the signal through a display, a speaker, etc.

[0055] The user terminal 300 may be connected to the server 200 through a predetermined communication line to enable communication with the server 200, and may be connected to the vehicle 100 via the server 200 to enable communication with the vehicle 100. Hereinafter, the same applies to the second embodiment ( Figure 8 ) situation.

[0056] Vehicle Configuration

[0057] Figure 2 1 is a diagram showing one example of a hardware configuration of a vehicle 100 related to the living body remaining monitoring system 1 according to the present example. Figure 3 1 is a diagram showing one example of a functional configuration of a vehicle 100 related to the living body remaining monitoring system 1 according to the present example.

[0058] like Figure 2 As shown in FIG. 1 , a vehicle 100 includes a cabin camera 10 , a room temperature sensor 20 , a humidity sensor 30 , a vehicle state data output device 40 , a communication device 50 , a sound output device 60 , an air conditioning device 70 , and a control apparatus 80 .

[0059] The cabin camera is installed in the cabin of the vehicle 100 and captures the scene in the cabin of the vehicle 100. Specifically, the cabin camera 10 may be installed so that the faces of all occupants on seats in the cabin are included in the image capturing range. The output (image data) of the cabin camera 10 is taken into the control device 80 via a one-to-one communication line, an in-vehicle network, or the like. Examples of in-vehicle networks include a controller area network (CAN), a local interconnect network (LIN), FlexRay, and an in-vehicle Ethernet.

[0060] The room temperature sensor 20 measures the temperature in the cabin of the vehicle 100. The room temperature sensor 20 is installed in, for example, an instrument panel. The output (measurement data) of the room temperature sensor 20 is taken into the control device 80 via a one-to-one communication line, an in-vehicle network, or the like.

[0061] The humidity sensor 30 measures the humidity in the cabin of the vehicle 100. The humidity sensor 30 is installed in, for example, an instrument panel. The output (measurement data) of the humidity sensor 30 is taken into the control device 80 via a one-to-one communication line, an in-vehicle network, or the like.

[0062] The vehicle state data output device 40 outputs data about the state of the vehicle 100. The vehicle state data output device 40 is, for example, a sensor that outputs measurement data about the state of the vehicle 100. The vehicle state data output device 40 is, for example, an electronic control unit (ECU) that outputs control data about the state of the vehicle 100. The output of the vehicle state data output device 40 is taken into the control device 80 via a one-to-one communication line or an in-vehicle network.

[0063] The communication device 50 communicates with an external device through a predetermined communication line. The communication device 50 is, for example, a mobile communication module connected to a mobile communication network with a base station as a terminal and capable of communicating with an external device. The communication device 50 may specifically be a data communication module (Data Communication Module, DCM). The communication device 50 is connected to the control device 80 through a one-to-one communication line or an in-vehicle network so as to be able to communicate with the control device 80. The communication device 50 may, for example, send a signal to an external device under the control of the control device 80. The signal received by the communication device 50 from the external device may be taken into the control device 80.

[0064] The sound output device 60 is connected to the control device 80 through a one-to-one communication line or an in-vehicle network to be able to communicate with the control device 80. The sound output device 60 outputs a predetermined sound under the control of the control device 80. Examples of the sound output device 60 include a buzzer, an alarm, and a speaker.

[0065] The air conditioning device 70 adjusts the temperature and humidity in the cabin of the vehicle 100, etc. The air conditioning device 70 is installed in the dashboard, for example. The air conditioning device 70 is connected to the control device 80 through a one-to-one communication line or an in-vehicle network to be able to communicate with the control device 80. The air conditioning device 70 can operate under the control of the control device 80.

[0066] The control device 80 (one example of an information processing device) performs control on the living body left-behind monitoring system 1. The control device 80 is, for example, an ECU. The control device 80 may be a dedicated control device (ECU) dedicated to control on the living body left-behind monitoring system 1, or may be a general control device (ECU) shared in processing of control on another function related to the vehicle 100.

[0067] The functions of the control device 80 may be implemented by any hardware, or any combination of hardware and software. Figure 2 As shown in FIG. 1 , the control device 80 is mainly constituted by a computer including a central processing unit (CPU) 81 , a memory device 82 , an auxiliary storage device 83 , and an interface device 84 connected to each other via a bus B1 .

[0068] The CPU 81 runs the program loaded on the memory device 82 and controls various functions of the apparatus 80 according to the instructions of the program.

[0069] When, for example, a program is started, the memory device 82 loads the program from the auxiliary storage device 83 so that the CPU 81 can read the program. The memory device 82 is, for example, a random access memory (RAM).

[0070] The auxiliary storage device 83 is a nonvolatile storage device. The auxiliary storage device 83 stores installed programs and necessary files and data, etc. The auxiliary storage device 83 is, for example, a read-only memory (ROM).

[0071] The interface device 84 may be used, for example, as an interface for connection with an on-board network or one-to-one connection with various sensors and actuators, etc. The interface device 84 may include a plurality of different types of interface devices to support objects to be connected.

[0072] The program implementing various functions of the control device 80 is provided, for example, by a dedicated tool connected to a predetermined external connection connector linked to the vehicle network of the vehicle 100 via a detachable cable. The external connection connector is, for example, a data link coupler (Data Link Coupler, DLC). In response to a predetermined operation of the dedicated tool, the program is installed from the dedicated tool via the cable, the connector and the vehicle network to the auxiliary storage device 83 of the control device 80. The program can be downloaded from another computer (e.g., the server 200) through a predetermined communication line and installed in the auxiliary storage device 83.

[0073] like Figure 3 As shown in FIG. 8 , the control device 80 includes as functional units: a living body presence / absence judgment unit 801, a cabin environment information acquisition unit 802, a facial expression analysis information acquisition unit 803, an elapsed time acquisition unit 804, an evaluation information acquisition unit 805, a judgment unit 806, and a control unit 807. The functional units of the control device 80 are implemented, for example, by loading various programs installed in the auxiliary storage device 83 onto the memory device 82 and running the programs on the CPU 81.

[0074] When the vehicle 100 is in a parking state, the living body presence / absence judgment unit 801 (an example of a first judgment unit) judges whether a monitored living body exists in the cabin of the vehicle 100 based on the image data obtained from the cabin camera 10. In other words, the living body presence / absence judgment unit 801 judges whether the monitored living body remains in the cabin of the vehicle 100 in the parking state. The parking state of the vehicle 100 corresponds to a state in which the power of the accessories of the vehicle 100 is turned off, that is, an ACC-off state. Specifically, the living body presence / absence judgment unit 801 attempts to recognize the monitored living body in the cabin of the vehicle 100 from the image data obtained by the cabin camera 10 through a known image processing technology and a known image recognition technology. When the living body presence / absence judgment unit 801 recognizes the monitored living body from the image data obtained by the cabin camera 10, the living body presence / absence judgment unit 801 judges that the monitored living body exists in the cabin of the vehicle 100. When the living body presence / absence judgment unit 801 does not recognize the monitored living body, the living body presence / absence judgment unit 801 judges that the monitored living body does not exist in the cabin of the vehicle 100. More specifically, the living body presence / absence judgment unit 801 attempts to recognize the monitored living body in the cabin of the vehicle 100 at a predetermined control interval based on the latest image data acquired by the cabin camera 10. The living body presence / absence judgment unit 801 judges whether the monitored living body exists in the cabin of the vehicle 100 in consideration of not only the current attempt result but also the past attempt results. Therefore, even when the monitored living body is not recognized in the current attempt for some reason, but the monitored living body has been recognized in the past, the control device 80 can judge that the monitored living body exists in the cabin of the vehicle 100.

[0075] The living body presence / absence judgment unit 801 can judge whether a monitored living body exists in the cabin of the vehicle 100 based on different data other than the image data. For example, a millimeter wave radar may be provided in the cabin of the vehicle 100, and the living body presence / absence judgment unit 801 can judge whether a monitored living body exists in the cabin of the vehicle 100 based on the presence or absence of output of measurement data indicating minute movements of the skin surface due to the heartbeat or pulse of the monitored living body from the millimeter wave radar. Hereinafter, the same applies to the living body presence / absence judgment unit 2001 (described later).

[0076] When the vehicle 100 is in a parked state, the cabin environment information acquisition unit 802 acquires information about the cabin environment of the vehicle 100 (hereinafter referred to as "cabin environment information") based on the output of the cabin camera 10, the room temperature sensor 20, the humidity sensor 30, etc. Specifically, the cabin environment information is information indicating the physical and psychological load on the monitored living body in the cabin of the vehicle 100. The cabin environment information is, for example, a thermal index (wet bulb globe temperature (WBGT)) indicating the physical and psychological load caused by the heat on the monitored living body in the cabin of the vehicle 100. The cabin environment information acquisition unit 802 can, for example, perform image analysis processing based on image data obtained by the room temperature sensor 20, and estimate the influence of radiant heat on the monitored living body from the degree of sunlight entering the cabin of the vehicle 100 or other factors. Through this configuration, the cabin environment information acquisition unit 802 is able to calculate the thermal index based on the room temperature, humidity, and radiant heat in the cabin of the vehicle 100.

[0077] A thermal sensor capable of measuring radiant heat of components in the cabin of the vehicle 100 may be installed in the cabin of the vehicle 100. In this case, the cabin environment information acquisition unit 802 may acquire the influence of radiant heat on the monitored living body in the cabin of the vehicle 100 based on the output of the thermal sensor. The cabin environment information acquisition unit 802 may acquire information indicating the physical and mental load caused by the coldness (e.g., the room temperature of the vehicle 100, etc.) on the monitored living body in the cabin of the vehicle 100.

[0078] When the living body presence / absence judgment unit 801 judges that a monitored living body exists in the cabin of the vehicle 100, the facial expression analysis information acquisition unit 803 (an example of a facial expression information acquisition unit) performs a known facial expression analysis on the face of the monitored living body contained in the image data acquired by the cabin camera 10 based on the image data. The facial expression analysis information acquisition unit 803 acquires information about the result of the analysis of the facial expression of the face of the monitored living body contained in the image data acquired by the cabin camera 10 (hereinafter referred to as "facial expression analysis information"). The facial expression analysis information includes information about the emotion of the monitored living body (hereinafter referred to as "emotion information"), information about the facial state of the monitored living body (hereinafter referred to as "facial state information"), and information about the age of the monitored living body (hereinafter referred to as "age information").

[0079] The emotion information includes the type of emotion of the monitored living body estimated from the facial expression of the face of the monitored living body. Examples of the type of emotion include positive emotion, neutral emotion, and negative emotion. Positive emotions may include emotions such as "pleasant", "funny", and "happy". Negative emotions may include emotions such as "anger", "sadness", "surprise", and "fear".

[0080] The facial state information includes the type of facial state of the monitored living body in the facial expression appearing on the face of the monitored living body. Examples of the type of facial state include a positive facial state, a neutral facial state, and a negative facial state. Examples of positive facial states include a smiling state. Examples of negative facial states include a crying state. Examples of negative facial states include a closed-eyes state. This is because when the monitored living body keeps its eyes closed, it is estimated that fatigue caused by physical and mental loads may be relatively large.

[0081] The frequency (period) at which the facial expression analysis information acquisition unit 803 performs facial expression analysis may be the same as the frequency (period) at which the cabin camera 10 acquires image data, or may be set to be less than the frequency (period) at which the cabin camera 10 acquires image data. The frequency (period) at which the facial expression analysis information acquisition unit 803 performs facial expression analysis may be variable. For example, based on cabin environment information, the frequency of facial expression analysis may be set to be higher as the cabin environment of the vehicle 100 is in a state of higher physical and psychological load on the monitored living body, and may be set to be lower as the cabin environment of the vehicle 100 is in a state of lower load. Through this configuration, the control device 80 is able to reduce the processing load, and is able to implement facial expression analysis in real time when the cabin environment of the vehicle 100 is in a state of relatively high physical and psychological load, and to capture changes in facial expressions of the monitored living body in real time.

[0082] The elapsed time acquisition unit 804 acquires the elapsed time (hereinafter referred to as "leftover elapsed time") that the monitored living body has remained in the cabin of the vehicle 100 based on the output of the vehicle state data output device 40. For example, the elapsed time acquisition unit 804 acquires the leftover elapsed time by using a timer function of a real-time clock (RTC) built into the control device 80, etc. The leftover elapsed time may be, for example, the elapsed time starting from the time point when the vehicle 100 stops (i.e., ACC-off). The leftover elapsed time may be the elapsed time starting from the time point when the presence of the monitored living body is first determined after the vehicle 100 stops. Alternatively, the leftover elapsed time may be, for example, the elapsed time starting from a starting point that is the time point when the door of the vehicle 100 was last closed before the time point when the presence of the monitored living body is first determined after the vehicle 100 stops.

[0083] When the living body presence / absence judgment unit 801 judges that a living body exists in the cabin of the vehicle 100, the evaluation information acquisition unit 805 (an example of an estimation unit) estimates the degree of decline in the life safety of the monitored living body in the vehicle 100. Specifically, in the case where the monitored living body remains in the cabin of the vehicle 100, the evaluation information acquisition unit 805 acquires information indicating the estimated degree of decline in the life safety of the monitored living body in the vehicle 100 (hereinafter referred to as "evaluation information") at predetermined control intervals based on the cabin environment information, the facial expression analysis information, and the remaining elapsed time.

[0084] The evaluation information acquisition unit 805 may calculate, for example, an index value (hereinafter referred to as “evaluation index”) EI indicating the degree of decline in the life safety of the monitored living body in the vehicle 100 as the evaluation information.

[0085] The evaluation index EI has, for example, a value greater than or equal to zero, and is defined based on such a correlation that as the value increases, the degree of decrease in the life safety of the monitored living body remaining in the cabin of the vehicle 100 is estimated to be higher. Specifically, the evaluation index EI can be calculated by the following expressions (1) and (2) using a cabin environment index W selected based on cabin environment information, a facial expression analysis index F selected based on facial expression analysis information, a residual elapsed time t, and an adjustment index β.

[0086] EI=αt+β (1)

[0087] α=W+F (2)

[0088] Through this configuration, assuming that the degree of decline in the life safety of the monitored living body increases with the increase of the residual elapsed time t, the control device 80 is able to calculate the evaluation index EI so as to be able to adjust the progression rate α of the degree of decline in the life safety of the monitored living body by using the cabin environment index W and the facial expression analysis index F.

[0089] For the progress rate α, another index other than the cabin environment index W or the facial expression analysis index F may be considered. For example, for the progress rate α, an index related to the activity of the monitored living body may be considered. Specifically, for the progress rate α, the presence or absence of the activity of the monitored living body related to the decline in life safety, the number of the activity, or the like may be considered, and the progress rate α may be set so that the progress rate α increases as the number of the activity increases.

[0090] The cabin environment index W has a value greater than or equal to zero and is set to increase as the physical and mental load on the monitored living body due to the environment in the cabin of the vehicle 100 increases. For example, when the heat index (WBGT) is adopted as the cabin environment information, the cabin environment index W is set to increase as the heat index increases.

[0091] For example, Figure 4 is a table showing an example of a method for selecting the evaluation index EI. Specifically, Figure 4 is a table showing an example of a method for selecting the cabin environment index W.

[0092] As Figure 4 shown, predetermined values W1, W2, W3 greater than zero are respectively assigned as the heat index to three ranges, namely, the range below 25 °C, the range higher than or equal to 25 °C and lower than 28 °C, and the range higher than 28 °C. The predetermined values W1, W2, W3 respectively assigned to these ranges are defined such that the cabin environment index W increases (becomes higher) as the heat index increases (W1 < W2 < W3). With this configuration, the evaluation information acquisition unit 805 can select the cabin environment index W as the cabin environment information by determining which range the heat index applies to.

[0093] The range of the heat index higher than or equal to 28 °C can be divided into the range higher than or equal to 28 °C and lower than 31 °C and the range higher than or equal to 31 °C, and the value of the cabin environment index W can be set to be greater in the former range than in the latter range. In the case of the range where the heat index is higher than or equal to 31 °C, the influence can be incorporated into the adjustment index β, and the adjustment index β can be set to be greater than the adjustment index in the case where the heat index is lower than 31 °C.

[0094] The facial expression analysis index F is an index having a value greater than or equal to zero and is used to evaluate the physical and mental load on the monitored living body due to the cabin environment of the vehicle 100 based on the facial expression analysis information. The facial expression analysis index F is defined to increase as the facial expression analysis information indicates a greater physical and mental load on the monitored living body. For example, the facial expression analysis index F can be calculated according to the following expression (3) by using the emotion index FE selected based on the emotion information, the facial state index FS selected based on the facial state information, and the age index FA selected based on the age information.

[0095] F = FE + FS + FA (3)

[0096] The emotion index FE is an index having a value greater than or equal to zero, and is used to evaluate the physical and psychological load of the monitored living being remaining in the cabin of the vehicle 100 from the perspective of the emotion of the monitored living being. The emotion index FE is defined to increase as the emotion information indicates a greater physical and psychological load on the monitored living being. The emotion index FE is defined, for example, so that the emotion index FE in the case where the emotion information indicates a negative emotion of the monitored living being is greater than the emotion index FE in the case where the emotion information indicates a positive emotion or a neutral emotion of the monitored living being. When, for example, the emotion information indicates a negative emotion of the monitored living being, the emotion index FE may be different depending on the type of negative emotion. This is because the type of negative emotion may be different depending on the degree of reduction in the life safety of the monitored living being.

[0097] For example, Figure 5 This is a table showing an example of a method for selecting the evaluation index EI. Specifically, Figure 5 : is a table illustrating one example of a method of selecting the facial expression analysis index F.

[0098] like Figure 5 As shown in, for example, when the emotion information indicates a positive emotion, the emotion index FE is set to the minimum predetermined value FE0 (>0), and when the emotion information indicates a negative emotion, the emotion index FE is set to any one of predetermined values ​​FE1, FE2, and FE3 greater than the predetermined value FE0 according to the type of negative emotion. Specifically, when the emotion information indicates the emotion of "surprise", the emotion index FE is set to the predetermined value FE1, when the emotion information indicates the emotion of "sadness", the emotion index FE is set to the predetermined value FE2 (>FE1), and when the emotion information indicates the emotion of "anger", the emotion index FE is set to the predetermined value FE3 (>FE2).

[0099] The magnitude relationship between the values ​​of the emotion index FE defined according to the type of negative emotion is one example, and another magnitude relationship may also be defined. The category of the type of negative emotion used to change the emotion index FE may include another type of emotion, such as "fear", in addition to "surprise", "sadness" and "anger".

[0100] The facial state index FS is an index having a value greater than or equal to zero, and is used to evaluate the physical and psychological load of the monitored living being remaining in the cabin of the vehicle 100 from the facial state of the monitored living being. The facial state index FS is defined to increase as the facial state information indicates a greater physical and psychological load of the monitored living being. The facial state index FS is defined, for example, so that the facial state index FS in the case where the facial state information indicates a negative facial state of the monitored living being is greater than the facial state index FS in the case where the facial state information indicates a positive facial state or a neutral facial state of the monitored living being. When, for example, the facial state information indicates a negative facial state of the monitored living being, the facial state index FS may be different depending on the type of the negative facial state. This is because the type of the negative facial state may be different depending on the degree of reduction in the life safety of the monitored living being.

[0101] like Figure 5 As shown in, for example, when the facial state information indicates a positive emotion, the facial state index FS is set to a predetermined value FS0 (>0), and, when the facial state information indicates a negative emotion, the facial state index FS is set to any one of predetermined values ​​FS1, FS2 that is greater than the predetermined value FS0 according to the type. Specifically, when the facial state information indicates a "crying" facial state, the facial state index FS is set to a predetermined value FS1, and when the facial state information indicates a closed-eye state, the facial state index FS is set to a predetermined value FS2 (>FS1). This is because the state in which the monitored living body keeps its eyes closed can be estimated as, for example, a situation in which the degree of decline in the life safety of the monitored living body is relatively low and it is difficult for the monitored living body to respond using the facial state.

[0102] The age index FA is an index having a value greater than or equal to zero, and is used to evaluate the physical and psychological load of the monitored living body remaining in the cabin of the vehicle 100 from the age of the monitored living body. This is because the tolerance to the load may vary depending on the age of the monitored living body. The age index FA is defined to increase as the age of the monitored living body indicates a lower tolerance to physical and psychological load. When, for example, the age of the monitored living body corresponds to a newborn or an infant, the age index FA can be set to a relatively large value. A newborn, for example, corresponds to a monitored living body of 0 years old, and an infant, for example, corresponds to a monitored living body of 1 to 5 years old. When, for example, the age of the monitored living body corresponds to an elderly person, the age index FA can be set to a relatively large value. An elderly person, for example, corresponds to a monitored living body over 65 years old.

[0103] like Figure 5As shown in, the age index FA is set to any one of the predetermined values ​​FA0, FA1, FA2, FA3, FA4, depending on which age category is indicated by the age information, such as "junior high school students and above (excluding the elderly)", "primary school students", "elderly people", "infants and young children", and "newborns". The "junior high school students and above (excluding the elderly)" category corresponds to a monitored living body that is over 12 years old and under 65 years old. The "primary school students" category corresponds to a monitored living body that is over 6 years old and under 12 years old. Specifically, when the age information indicates an age corresponding to "junior high school students and above (excluding the elderly)", the age index FA is set to the minimum predetermined value FA0, and when the age information indicates an age corresponding to "primary school students", the age index FA is set to the second minimum predetermined value FA1 (>FA0). On the other hand, when the age information indicates an age corresponding to "elderly people", the age index FA is set to a relatively large predetermined value FA2 (>FA1). When the age information indicates an age corresponding to "infants and young children", the age index FA is set to a relatively large predetermined value FA3 (>FA2). When the age information indicates an age corresponding to a "newborn", the age index FA is set to a relatively large predetermined value FA4 (> FA3).

[0104] The magnitude relationship between the age indexes FA corresponding to the categories of “elderly”, “infant”, and “newborn”, respectively, is one example, and another magnitude relationship may also be defined.

[0105] The adjustment index β is used, for example, to adjust the evaluation index EI so that when the progression rate α changes, the evaluation index EI changes at the progression rate α with reference to the value immediately before the evaluation index EI changes (see Figure 6 (described later)). As described above, the adjustment index β is set so that the adjustment index β in the case where the thermal index is higher than or equal to 31° C. is greater than the adjustment index β in the case where the thermal index is lower than 31° C. Therefore, the influence of the case where the thermal index is higher than or equal to 31° C. on the degradation of the life safety of the monitored living body can be considered.

[0106] Instead of or in addition to the progression rate α, at least one of cabin environment information and facial expression analysis information may be incorporated into the adjustment index β.

[0107] Return to reference Figure 3 The judgment unit 806 (an example of a second judgment unit) judges whether the life safety of the monitored living body is lower than or equal to a predetermined benchmark based on the evaluation information sequentially acquired by the evaluation information acquisition unit 805 when the monitored living body remains in the cabin of the vehicle 100.

[0108] Specifically, when the evaluation index EI calculated by the evaluation information acquisition unit 805 is greater than or equal to a predetermined threshold value EI_th, the judgment unit 806 may judge that the life safety of the monitored living body is lower than or equal to a predetermined reference.

[0109] For example, Figure 6 1 is a diagram showing a specific example of the time-series change of the evaluation index EI in a case where a monitored living body (eg, an infant) remains in the cabin of the vehicle 100 .

[0110] like Figure 6 As shown in , in the period TP1 where the residual elapsed time t changes from 0 to the predetermined value t1, the cabin environment information (thermal index), the emotion information, and the facial state information indicate a state below 25°C, a positive emotion, and a neutral facial state, respectively. Therefore, as the residual elapsed time t increases, the evaluation index EI increases with a relatively small slope (progression rate α).

[0111] Thereafter, when the remaining elapsed time t reaches the predetermined value t1, the cabin environment information (thermal index) and the mood information are converted (deteriorated) to a state higher than or equal to 25°C and lower than 28°C and a negative mood, respectively. Therefore, in the time period TP2 in which the remaining elapsed time t changes from the predetermined value t1 to the predetermined value t2, the increase rate (progression rate α) of the evaluation index EI for the increase of the remaining elapsed time t is higher than the increase rate (progression rate α) of the evaluation index EI for the increase of the remaining elapsed time t in the time period TP1, and the increase of the evaluation index EI is accelerated.

[0112] When the remaining elapsed time t reaches the predetermined value t2, the facial state of the monitored living body changes (deteriorates) from the neutral state to the crying state. Therefore, in the period TP3 where the remaining elapsed time t is later than the predetermined value t2, the increase rate (progression rate α) of the evaluation index EI for the increase of the remaining elapsed time t is higher than the increase rate (progression rate α) of the evaluation index EI for the increase of the remaining elapsed time t in the period TP2, and the increase of the evaluation index EI is further accelerated.

[0113] When the remaining elapsed time t reaches a predetermined value t3, the evaluation index EI reaches a threshold value EI_th, and the judgment unit 806 judges that the life safety of the monitored living body is lower than or equal to a predetermined reference.

[0114] Return to reference Figure 3 , when the judgment unit 806 judges that the life safety of the monitored living body remaining in the cabin of the vehicle 100 is lower than or equal to a predetermined reference, the control unit 807 activates the safety function.

[0115] The safety function may include, for example, a function of issuing an alarm about a monitored living body remaining in the cabin of the vehicle 100 to the surroundings of the vehicle 100 (hereinafter referred to as a "vehicle surrounding alarm function"). Specifically, the control unit 807 causes the sound output device 60 to output auditory information (sound information) about the monitored living body remaining in the cabin of the vehicle 100 to the surroundings of the vehicle 100. Examples of the alarm about the left-behind monitored living body may include an alarm indicating a state where the monitored living body is left in the cabin of the vehicle 100 and an alarm indicating that the life safety of the monitored living body is decreasing due to the monitored living body being left in the cabin of the vehicle 100. With this configuration, the control device 80 is able to issue an alarm to people around the vehicle 100 about the presence of a monitored living body remaining in the cabin of the vehicle 100, or the decrease in the life safety of the monitored living body, etc. Therefore, the control device 80 is able to prompt people around the vehicle 100 to provide a notification to a predetermined agency (e.g., a police station PS) or the like.

[0116] The safety function may include, for example, a function of providing a notification about a monitored living body left in the cabin of the vehicle 100 to the user of the vehicle 100 through the user terminal 300 (hereinafter referred to as a "user alert function"). Specifically, the control unit 807 may implement the user alert function by communicating with the user terminal 300 and sending an alert signal (push notification) about the monitored living body left in the cabin of the vehicle 100 to the user terminal 300 using the communication device 50. Through this configuration, the control device 80 is able to prompt the user of the vehicle 100 who has left the vehicle 100 to return to the vehicle 100 by causing the user to recognize the monitored living body left behind or causing the user to grasp the decline in the life safety of the monitored living body left behind.

[0117] The safety function may include, for example, a function of issuing an alarm (providing notification) to an operator OP of a monitoring center CT (an example of a predetermined facility) or the like regarding a monitored living body remaining in the cabin of the vehicle 100 (hereinafter referred to as an "operator notification function"). Specifically, the control unit 807 may implement the operator alarm function by communicating with the server 200 using the communication device 50 and sending an alarm signal regarding a monitored living body remaining in the cabin of the vehicle 100 to the server 200. With this configuration, the control device 80 is able to notify the operator OP of the monitoring center CT or the like through the server 200 of the fact that a monitored living body is remaining in the cabin of the vehicle 100, or the fact that the life safety of the remaining monitored living body is in danger, etc. Therefore, the control device 80 is able to prompt the operator 100 of the monitoring center CT or the like to contact the user of the vehicle 100, provide notification to the police station PS, or arrange an ambulance, etc.

[0118] The safety function may include, for example, a function of issuing an alarm (providing notification) to a police station PS (an example of a predetermined facility) about a monitored living body remaining in the cabin of the vehicle 100 (hereinafter referred to as a "police station notification function"). Specifically, the control unit 807 may implement the police station notification function by communicating with the police station PS (e.g., automatic voice call, etc.) and providing notification about a monitored living body remaining in the cabin of the vehicle 100, using the communication device 50. With this configuration, the control device 80 is able to request dispatch to the police station PS or request arrangement of an ambulance through the police station PS.

[0119] The safety function may include, for example, a function of operating the air conditioning device 70 (hereinafter referred to as an "air conditioning device operation function"). With this configuration, the control device 80, for example, can activate the cooling function of the air conditioning device 70 in a case where the cabin environment of the vehicle 100 deteriorates in a direction where the heat index increases. The control device 80, for example, can activate the heating function of the air conditioner in a case where the cabin environment of the vehicle 100 deteriorates in a direction where the room temperature greatly decreases, such as in a case where the room temperature of the vehicle 100 is below a freezing temperature.

[0120] Thus, in this example, the control unit 80 can determine (estimate) the degree of reduction in the life safety of the monitored living body remaining in the cabin of the vehicle 100 by using the facial expression analysis information, taking into account the emotions, facial state, or age, etc., which are unique to the monitored living body. Therefore, the control device 80 can ensure the life safety of the monitored living body remaining in the cabin of the vehicle 100 in consideration of the individual differences of the monitored living body.

[0121] Control device processing flow

[0122] Figure 7 1 is a flowchart schematically showing one example of the living body remaining monitoring process executed by the control device 80. For example, when the vehicle 100 is in a parked state, the flowchart is repeatedly executed at predetermined control intervals.

[0123] like Figure 7 As shown in FIG. 1 , in step S102, the living body presence / absence judgment unit 801 judges whether there is a monitored living body in the cabin of the vehicle 100. When there is a monitored living body in the cabin of the vehicle 100, the living body presence / absence judgment unit 801 proceeds to step S104; on the other hand, when there is no monitored living body in the cabin of the vehicle 100, the living body presence / absence judgment unit 801 ends the processing of the current flowchart.

[0124] In step S104 , the cabin environment information acquiring unit 802 acquires cabin environment information.

[0125] When the control device 80 completes the process of step S104 , the control device 80 proceeds to step S106 .

[0126] In step S106, the facial expression analysis information acquisition unit 803 acquires the latest facial expression analysis information. The latest facial expression analysis information is the facial expression analysis information output during the processing period of the flowchart or the facial expression analysis information acquired for the last time before the processing period of the flowchart. This is because the frequency of facial expression analysis may be less than the processing frequency of the flowchart, or the frequency of facial expression analysis may be variable as described above.

[0127] When the control device 80 completes the process of step S106 , the control device proceeds to step S108 .

[0128] In step S108 , the elapsed time acquisition unit 804 acquires the remaining elapsed time.

[0129] When the control device 80 completes the process of step S108 , the control device 80 proceeds to step S110 .

[0130] In step S110 , the evaluation information acquisition unit 805 acquires (calculates) the evaluation index EI based on the acquired cabin environment information, facial expression analysis information, and remaining elapsed time.

[0131] When the control device 80 completes the process of step S110 , the control device 80 proceeds to step S112 .

[0132] In step S112, the judgment unit 806 judges whether the evaluation index EI is greater than or equal to the threshold EI_th. When the evaluation index EI is greater than or equal to the threshold EI_th, the judgment unit 806 judges that the life safety of the monitored living body is lower than or equal to the predetermined benchmark, and proceeds to step S114. When the evaluation index EI is not greater than or equal to the threshold EI_th, the judgment unit 806 ends the processing of the current flowchart.

[0133] In step S114 , the control unit 807 activates the safety function. Thus, the control device 80 can ensure the life safety of the monitored living body remaining in the cabin of the vehicle 100 .

[0134] In this way, the control device 80 can monitor the degree of decline in the life safety of the monitored living body in the cabin of the vehicle 100 by sequentially calculating the evaluation index EI and monitoring whether the evaluation index EI is greater than or equal to the threshold value EI_th at a predetermined control interval. When the evaluation index EI is greater than or equal to the threshold value EI_th, the control device 80 can ensure the life safety of the monitored living body by activating the safety function.

[0135] Second Example of Living Body Remaining Monitoring System

[0136] Will refer to Figure 8 and Fig. 9 A second example of the living body remaining monitoring system 1 according to the present embodiment is described. Hereinafter, the same reference numerals denote components identical to or corresponding to those of the first example, and portions different from the first example will be mainly described.

[0137] Overview of the Living Body Remaining Monitoring System

[0138] Figure 8 FIG. 1 is a schematic diagram showing a second example of the living body remaining monitoring system 1 .

[0139] like Figure 8 As shown in , the living body remaining monitoring system 1 includes a vehicle 100 , a server 200 , a user terminal 300 , and a facial expression analysis server 400 .

[0140] The facial expression analysis server 400 is connected to the vehicle 100 through a predetermined communication line so as to be able to communicate with the vehicle 100. The facial expression analysis server 400 performs facial expression analysis based on the image data from the cabin camera 10 transmitted (uploaded) from the vehicle 100, and returns a signal including information on the facial expression analysis result to the vehicle 100. Through this configuration, the vehicle 100 (control device 80) is able to reduce, for example, a processing load for monitoring the degree of decline in the life safety of a living body remaining in the cabin. The vehicle 100 is able to suppress a decline in the remaining capacity of the battery of the vehicle 100 by reducing the processing load of the control device 80.

[0141] Vehicle Configuration

[0142] Fig. 9 1 is a functional block diagram showing another example of the functional configuration of the vehicle 100 related to the living body left-behind monitoring system 1 .

[0143] The hardware configuration of the vehicle 100 related to the living body remaining monitoring system 1 is the same as that of the first example, and Figure 2 , and therefore the hardware configuration of the vehicle 100 is not illustrated.

[0144] As in the case of the first example, the vehicle 100 includes a cabin camera 10 , a room temperature sensor 20 , a humidity sensor 30 , a vehicle state data output device 40 , a communication device 50 , a sound output device 60 , an air conditioning device 70 , and a control apparatus 80 .

[0145] like Fig. 9As shown in , as in the case of the first example, the control device 80 includes as functional units: a living body presence / absence judgment unit 801, a cabin environment information acquisition unit 802, a facial expression analysis information acquisition unit 803, an elapsed time acquisition unit 804, an evaluation information acquisition unit 805, a judgment unit 806 and a control unit 807.

[0146] The facial expression analysis information acquisition unit 803 sends a signal for requesting facial expression analysis (hereinafter referred to as a “facial expression analysis request signal”) to the facial expression analysis server 400 via the communication device 50 and sends (uploads) image data acquired by the cabin camera 10 to the facial expression analysis server 400 .

[0147] The facial expression analysis server 400 performs facial expression analysis on the uploaded image data in response to the facial expression analysis request signal received from the vehicle 100 , and returns information on the facial expression analysis result (facial expression analysis information) to the vehicle 100 .

[0148] Through this configuration, the facial expression analysis information acquisition unit 803 can acquire the facial expression analysis information returned from the facial expression analysis server 400 .

[0149] The frequency (period) of sending the facial expression analysis request signal to the facial expression analysis server 400 through the communication device 50 may be the same as the frequency (period) of acquiring image data by the cabin camera 10, or may be set to be less than the frequency (period) of acquiring image data by the cabin camera 10. The frequency (period) of sending the facial expression analysis request signal to the facial expression analysis server 400 may be variable as in the case of the frequency of the facial expression analysis of the first example. Through this configuration, the control device 80 is able to reduce the communication load or communication fee, and is able to implement facial expression analysis in real time when the cabin environment of the vehicle 100 is under a relatively high physical and psychological load, and to capture changes in the facial expressions of the monitored living body in real time.

[0150] Thus, in this example, the control device 80 can make a request for facial expression analysis of image data to the facial expression analysis server 400, and obtain facial expression analysis information from the facial expression analysis server 400. With this configuration, the control device 80 can reduce the processing load in monitoring the monitored living body remaining in the cabin of the vehicle 100.

[0151] The control device 80 can not only perform facial expression analysis of image data, but also make a request to an external device to perform image recognition processing for identifying a monitored living body from image data. In this case, the living body presence / absence judgment unit 801 judges whether a monitored living body exists in the cabin of the vehicle 100 based on the result of the image recognition processing returned from the external device. With this configuration, the control device 80 can further reduce the processing load in monitoring the monitored living body remaining in the cabin of the vehicle 100.

[0152] The third example of the living body remaining monitoring system

[0153] Will refer to Figures 10 to 12 A third example of the living body remaining monitoring system according to the present embodiment is described. Hereinafter, the same reference numerals denote components identical to or corresponding to those of the first or second example, and portions different from those of the first or second example will be mainly described.

[0154] Overview of the Living Body Remaining Monitoring System

[0155] Fig.10 1 is a schematic diagram showing a third example of the living body remaining monitoring system 1 .

[0156] like Fig.10 As shown in , the living body remaining monitoring system 1 includes a vehicle 100 , a server 200 , and a user terminal 300 .

[0157] When the vehicle 100 is in a parked state, the vehicle 100 transmits (uploads) various data (hereinafter referred to as "monitoring data") for determining whether a monitored living body is left in the cabin and monitoring the degree of decrease in the safety of the monitored living body to the server 200. The monitoring data includes image data obtained by the cabin camera 10, output data (measurement data) of the room temperature sensor 20, output data (measurement data) of the humidity sensor 30, and output data of the vehicle state data output device 40.

[0158] The server 200 (an example of an information processing device) has a function of judging whether a predetermined monitored living body is left in the cabin of the vehicle 100 based on the monitoring data transmitted from the vehicle 100, and, when a monitored living body is present, monitoring the degree of decline in the life safety of the monitored living body. When the life safety of the monitored living body left in the cabin is lower than or equal to a predetermined benchmark, the server 200 activates the safety function.

[0159] The user terminal 300 is connected to the server 200 through a predetermined communication line to be able to communicate with the server 200. With this configuration, the user terminal 300 can receive a signal transmitted from the server 200 and cause the user (user of the vehicle 100) to grasp the information contained in the signal through a display, a speaker, etc.

[0160] Server Configuration

[0161] Fig.11 1 is a diagram showing an example of the hardware configuration of the server 200 related to the living body remaining monitoring system 1. Fig.12 1 is a functional block diagram showing one example of the functional configuration of the server 200 related to the living body remaining monitoring system 1.

[0162] The hardware configuration of the vehicle 100 according to this example is similar to that of the first example and other examples, and Figure 2 In this example, the functions of the living body presence / absence judgment unit 801, the cabin environment information acquisition unit 802, the facial expression analysis information acquisition unit 803, the elapsed time acquisition unit 804, the evaluation information acquisition unit 805, the judgment unit 806 and the control unit 807 of the control device 80 are omitted and assigned to the server 200, as will be described later.

[0163] The functions of the server 200 may be implemented by any hardware, any combination of hardware and software, etc. For example, Fig.11 As shown in FIG. 1 , the server 200 includes a drive device 201 , an auxiliary storage device 202 , a memory device 203 , a CPU 204 , an interface device 205 , a display device 206 , and an input device 207 , which are connected to each other via a bus B2 .

[0164] The programs for implementing the various functions of the server 200 are provided by, for example, a portable recording medium 201A such as a Compact Disc Read Only Memory (CD-ROM), a Digital Versatile Disc Read Only Memory (DVD-ROM), and a Universal Serial Bus (USB) flash drive, etc. When the recording medium 201A having the program recorded thereon is loaded into the drive device 201, the program is installed from the recording medium 201A to the auxiliary storage device 202 via the drive device 201. The program may be downloaded from another computer via a communication network and installed in the auxiliary storage device 202.

[0165] The auxiliary storage device 202 stores various installed programs and stores necessary files, data, and the like.

[0166] When the memory device 203 receives an instruction to start a program, the memory device 203 reads the program from the auxiliary storage device 202 and stores the program.

[0167] The CPU 204 runs various programs stored in the memory device 203 to implement various functions of the server 200 according to these programs.

[0168] The interface device 205 serves as an interface for connecting with an external communication line. With this configuration, the server 200 can communicate with an external facility such as the vehicle 100, the user terminal 300, or a police station PS, etc., through a predetermined communication line.

[0169] The display device 206 displays a graphical user interface (GUI) according to a program executed by the CPU 204 , for example.

[0170] The input device 207 is used to allow an operator or administrator of the server 200 to input various operation instructions on the server 200 .

[0171] like Fig.12 As shown in FIG. 2 , the server 200 includes as functional units: a living body presence / absence judgment unit 2001, a cabin environment information acquisition unit 2002, a facial expression analysis information acquisition unit 2003, an elapsed time acquisition unit 2004, an evaluation information acquisition unit 2005, a judgment unit 2006, and a control unit 2007. The functional units of the server 200 are implemented, for example, by loading various programs installed in the auxiliary storage device 202 onto the memory device 203 and running the programs on the CPU 204.

[0172] When the vehicle 100 is in a parked state, the living body presence / absence judgment unit 2001 (an example of the first judgment unit) judges whether there is a monitored living body in the cabin of the vehicle 100 based on the image data obtained by the cabin camera 10 and received from the vehicle 100. The specific judgment method may be similar to that in the case of the first example (living body presence / absence judgment unit 801).

[0173] When the vehicle 100 is in a parked state, the cabin environment information acquisition unit 2002 acquires information about the cabin environment of the vehicle 100 based on output data of the cabin camera 10, the room temperature sensor 20, the humidity sensor 30, etc. received from the vehicle 100. The specific acquisition method of the cabin environment information acquired by the cabin environment information acquisition unit 2002 and the specific details of the cabin environment information to be acquired may be similar to those in the first example and other examples (cabin environment information acquisition unit 802).

[0174] When the living body presence / absence judgment unit 2001 judges that a monitored living body exists in the cabin of the vehicle 100, the facial expression analysis information acquisition unit 2003 (an example of a facial expression information acquisition unit) performs a known facial expression analysis on the face of the monitored living body contained in the image data obtained by the cabin camera 10 based on the image data. The facial expression analysis information acquisition unit 2003 acquires facial expression analysis information based on the result of the facial expression analysis. The specific details of the facial expression analysis to be acquired by the facial expression analysis information acquisition unit 2003 may be similar to those in the case of the first example and other examples (the facial expression analysis information acquisition unit 803).

[0175] The facial expression analysis information acquisition unit 2003 may make a request for image analysis to an external device such as the facial expression analysis server 400 in the second example, for example.

[0176] The elapsed time acquisition unit 2004 acquires the remaining elapsed time based on the output data of the vehicle state data output device 40 received from the vehicle 100. The specific acquisition method of the elapsed time acquisition unit 2004 to acquire the remaining elapsed time may be similar to that in the first example and other examples (the elapsed time acquisition unit 804).

[0177] When the living body presence / absence judgment unit 2001 judges that a living body exists in the cabin of the vehicle 100, the evaluation information acquisition unit 2005 (an example of an estimation unit) estimates the degree of decline in the life safety of the monitored living body in the vehicle 100. Specifically, in the case where the monitored living body remains in the cabin of the vehicle 100, the evaluation information acquisition unit 2005 acquires the evaluation index EI regarding the degree of decline in the life safety of the monitored living body in the vehicle 100 at a predetermined control interval based on the cabin environment information, the facial expression analysis information, and the remaining elapsed time. The specific acquisition method of the evaluation index EI acquired by the evaluation information acquisition unit 2005 and the specific details of the evaluation index EI to be acquired may be similar to those in the first example and other examples (evaluation information acquisition unit 805).

[0178] The judgment unit 2006 (an example of the second judgment unit) judges whether the life safety of the monitored living body is lower than or equal to a predetermined reference based on the evaluation information sequentially acquired by the evaluation information acquisition unit 2005 under the condition that the monitored living body remains in the cabin of the vehicle 100. The specific judgment method may be similar to that in the first example and other examples (judgment unit 806).

[0179] When the judgment unit 2006 judges that the life safety of the monitored living body remaining in the cabin of the vehicle 100 is lower than or equal to a predetermined reference, the control unit 2007 activates the safety function.

[0180] The control unit 2007 can activate the vehicle surrounding alarm function or the air conditioning device operation function by sending a control command to the vehicle 100, for example.

[0181] The control unit 2007 may activate the user alert function by, for example, sending an alert signal to the user terminal 300 .

[0182] The control unit 2007 may implement the operator notification function by, for example, causing the display device 206 to be visually recognizable by the operator OP, or the like, to display alarm information regarding the monitored living body remaining in the cabin of the vehicle 100 .

[0183] The control unit 2007 can activate the police station notification function by communicating with the police station PS via a predetermined communication line (automatic voice call, etc.) to provide a notification about the monitored living body remaining in the cabin of the vehicle 100, for example.

[0184] The server 200 performs a living body remaining monitoring process similar to that in the case of the first example and other examples ( Figure 7 ). In this case, execute Figure 7 The main body of the processing is replaced by a living body presence / absence judgment unit 2001, a cabin environment information acquisition unit 2002, a facial expression analysis information acquisition unit 2003, an elapsed time acquisition unit 2004, an evaluation information acquisition unit 2005, a judgment unit 2006 and a control unit 2007.

[0185] Thus, in this example, the server 200 monitors the degree of decline in the life safety of the monitored living body remaining in the cabin from the outside of the vehicle 100 by using the monitoring data uploaded from the vehicle 100. When the life safety of the monitored living body remaining in the cabin of the vehicle 100 is lower than or equal to a predetermined reference, the server 200 activates a safety function to ensure the life safety of the monitored living body. Through this configuration, the living body remaining monitoring system 1 can reduce the processing load on the vehicle 100 (edge) side, and can reduce the power consumption caused by the increase in power consumption when the vehicle 100 is in a parked state.

[0186] run

[0187] Next, the operation of the living body remaining monitoring system 1 (control device 200 or server 200) according to the present embodiment will be described.

[0188] In the present embodiment, the control device 80 includes a living body presence / absence judgment unit 801, a facial expression analysis information acquisition unit 803, a judgment unit 806, and a control unit 807. Specifically, the living body presence / absence judgment unit 801 judges whether the monitored living body remains in the cabin of the vehicle 100. The facial expression analysis information acquisition unit 803 acquires information about the facial expression of the monitored living body based on image data obtained by the cabin camera 10 including the face of the monitored living body remaining in the cabin of the vehicle 100. The judgment unit 806 judges whether the life safety of the monitored living body is lower than or equal to a predetermined benchmark based on the information about the facial expression of the monitored living body. When the judgment unit 806 judges that the life safety of the monitored living body is lower than or equal to the predetermined benchmark, the control unit 807 activates a safety function for ensuring the life safety of the monitored living body.

[0189] Similarly, the server 200 includes a living body presence / absence judgment unit 2001, a facial expression analysis information acquisition unit 2003, a judgment unit 2006, and a control unit 2007. Specifically, the living body presence / absence judgment unit 2001 judges whether the monitored living body remains in the cabin of the vehicle 100. The facial expression analysis information acquisition unit 2003 acquires information about the facial expression of the monitored living body based on image data obtained by the cabin camera 10 including the face of the monitored living body remaining in the cabin of the vehicle 100. The judgment unit 2006 judges whether the life safety of the monitored living body is lower than or equal to a predetermined benchmark based on the information about the facial expression of the monitored living body. When the judgment unit 2006 judges that the life safety of the monitored living body is lower than or equal to the predetermined benchmark, the control unit 2007 activates a safety function for ensuring the life safety of the monitored living body.

[0190] Through this configuration, the control device 80 or the server 200 (hereinafter referred to as "the control device 80, etc.") can use information about the facial expression of the face of the monitored living body remaining in the cabin of the vehicle 100. Therefore, the control device 80, etc. can grasp the state unique to the monitored living body, etc. based on the facial expression of the face of the monitored living body, and judge the degree of decline in the life safety of the monitored living body. Thus, the control device 80, etc. can ensure the life safety of the living body remaining in the vehicle 100 in consideration of the individual differences of the monitored living body.

[0191] In this embodiment, the control device 80 may include an evaluation information acquisition unit 805. Specifically, the evaluation information acquisition unit 805 may estimate the degree of decline in the life safety of the monitored living body based on the information about the facial expression of the monitored living body acquired by the facial expression analysis information acquisition unit 803. The judgment unit 806 judges whether the life safety of the monitored living body is lower than or equal to a predetermined benchmark based on the estimation result of the evaluation information acquisition unit 805.

[0192] Similarly, the server 200 may include an evaluation information acquisition unit 2005. Specifically, the evaluation information acquisition unit 2005 may estimate the degree of decline in the life safety of the monitored living body based on the information about the facial expression of the monitored living body acquired by the facial expression analysis information acquisition unit 2003. The judgment unit 2006 judges whether the life safety of the monitored living body is lower than or equal to a predetermined benchmark based on the estimation result of the evaluation information acquisition unit 2005.

[0193] Through this configuration, the control device 80 etc. can estimate the degree of decline in the life safety of the monitored living body remaining in the cabin of the vehicle 100 by taking into account the individual differences of the monitored living body by using the information on the facial expressions of the monitored living body. Therefore, when the control device 80 etc. determines whether the life safety of the monitored living body is lower than or equal to a predetermined benchmark by using the estimation result, the control device 80 etc. can ensure the life safety of the monitored living body remaining in the vehicle 100 by specifically taking into account the individual differences of the monitored living body.

[0194] The judgment unit 806 or the judgment unit 2006 can directly use the cabin environment information, facial expression analysis information, and the remaining elapsed time instead of the evaluation information (evaluation index EI) to judge whether the life safety of the monitored living body is lower than or equal to the predetermined benchmark. In this case, the evaluation information acquisition unit 805 or the evaluation information acquisition unit 2005 can be omitted. For example, the judgment unit 806 or the judgment unit 2006 can judge whether the life safety of the monitored living body is lower than or equal to the predetermined benchmark by judging whether the conditions defined for the cabin environment information, the facial expression analysis information, the elapsed time, etc. are met.

[0195] In this embodiment, the information about the facial expression of the monitored living body may include information about the emotion of the monitored living body. The evaluation information acquisition unit 805 may estimate the degree of decline in the life safety of the monitored living body so that when the information about the emotion of the monitored living body represents a negative emotion, the degree of decline in the life safety of the monitored living body is greater than when the information about the emotion of the monitored living body represents a positive emotion or a neutral emotion.

[0196] Similarly, the evaluation information acquisition unit 2005 can estimate the degree of decrease in the life safety of the monitored living being so that, when the information about the emotions of the monitored living being represents negative emotions, the degree of decrease in the life safety of the monitored living being is greater than the degree of decrease in the life safety of the monitored living being when the information about the emotions of the monitored living being represents positive emotions or neutral emotions.

[0197] By this configuration, the control device 80 or the like can estimate the degree of decline in the life safety of the monitored living body remaining in the vehicle 100 in consideration of individual differences of the monitored living body specifically by capturing the emotions of the monitored living body.

[0198] In this embodiment, the information about the facial expression of the monitored living body may include information about the facial state of the monitored living body. The evaluation information acquisition unit 805 may estimate the degree of decline in the life safety of the monitored living body so that when the information about the facial state of the monitored living body represents a negative facial state, the degree of decline in the life safety of the monitored living body is greater than when the information about the facial state of the monitored living body represents a positive facial state or a neutral facial state.

[0199] Similarly, the evaluation information acquisition unit 2005 can estimate the degree of decrease in the life safety of the monitored living body so that, when the information about the facial state of the monitored living body represents a negative facial state, the degree of decrease in the life safety of the monitored living body is greater than the degree of decrease in the life safety of the monitored living body when the information about the facial state of the monitored living body represents a positive facial state or a neutral facial state.

[0200] By this configuration, the control device 80 or the like can estimate the degree of decline in the life safety of the monitored living body remaining in the vehicle 100 in consideration of individual differences of the monitored living body specifically by capturing the facial state of the monitored living body.

[0201] In this embodiment, the evaluation information acquisition unit 805 can estimate the degree of decline in the life safety of the monitored living being so that the decline in the life safety of the monitored living being is accelerated as the information about the emotion of the monitored living being changes from a state representing a positive emotion or a neutral emotion to a state representing a negative emotion, or as the information about the facial state of the monitored living being changes from a state representing a positive facial state or a neutral facial state to a state representing a negative facial state.

[0202] Similarly, the evaluation information acquisition unit 2005 can estimate the degree of decline in the life safety of the monitored living being so that the decline in the life safety of the monitored living being accelerates as the information about the emotion of the monitored living being changes from a state representing a positive emotion or a neutral emotion to a state representing a negative emotion, or as the information about the facial state of the monitored living being changes from a state representing a positive facial state or a neutral facial state to a state representing a negative facial state.

[0203] Through this configuration, the control device 80 etc. can estimate the degree of decline in the life safety of the monitored living body by capturing a situation in which the decline in the life safety of the monitored living body is accelerated with a negative change in the emotion or facial state of the monitored living body. Therefore, the control device 80 etc. can estimate the degree of decline in the life safety of the monitored living body by specifically capturing the negative change in the emotion or facial state of the monitored living body while taking into account the individual differences of the monitored living body.

[0204] In the present embodiment, the evaluation information acquisition unit 805 estimates the degree of decline in the life safety of the monitored living being such that the life safety of the monitored living being declines as the time that the monitored living being has stayed in the cabin of the vehicle 100 elapses. The evaluation information acquisition unit 805 estimates the degree of decline in the safety of the monitored living being such that the rate of decline in the life safety of the monitored living being with respect to the elapse of time is higher when the information about the emotion of the monitored living being represents a negative emotion than the rate of decline in the life safety of the monitored living being with respect to the elapse of time when the information about the emotion of the monitored living being represents a positive emotion or a neutral emotion, and the rate of decline in the life safety of the monitored living being with respect to the elapse of time is higher when the information about the facial state of the monitored living being represents a negative facial state than the rate of decline in the life safety of the monitored living being with respect to the elapse of time when the information about the facial state of the monitored living being represents a positive facial state or a neutral facial state.

[0205] Similarly, the evaluation information acquisition unit 2005 estimates the degree of decline in the life safety of the monitored living being such that the life safety of the monitored living being declines as the time that the monitored living being has stayed in the cabin of the vehicle 100 elapses. The evaluation information acquisition unit 805 estimates the degree of decline in the safety of the monitored living being such that the rate of decline in the life safety of the monitored living being with respect to the elapse of time is higher when the information about the emotion of the monitored living being represents a negative emotion than the rate of decline in the life safety of the monitored living being with respect to the elapse of time when the information about the emotion of the monitored living being represents a positive emotion or a neutral emotion, and the rate of decline in the life safety of the monitored living being with respect to the elapse of time is higher when the information about the facial state of the monitored living being represents a negative facial state than the rate of decline in the life safety of the monitored living being with respect to the elapse of time when the information about the facial state of the monitored living being represents a positive facial state or a neutral facial state.

[0206] By this configuration, the control device 80 etc. can variably change the rate of decline of the life safety of the monitored living being depending on the difference in the emotion or facial state of the monitored living being, assuming that the life safety of the monitored living being declines with the lapse of time that the monitored living being has stayed in the cabin of the vehicle 100. Therefore, the control device 80 etc. can specifically make the decline of the life safety of the monitored living being accelerate with the negative change of the emotion or facial state of the monitored living being to estimate the degree of decline of the life safety of the monitored living being.

[0207] In this embodiment, the information about the facial expression of the monitored living body may include information about the emotion of the monitored living body. The evaluation information acquisition unit 805 may estimate the degree of decrease in the life safety of the monitored living body so that, when the information about the emotion of the monitored living body represents any one of a plurality of predefined types of negative emotions, the degree of decrease in the life safety of the monitored living body is different depending on the one of the plurality of types of negative emotions represented.

[0208] Similarly, the evaluation information acquisition unit 2005 can estimate the degree of decline in the life safety of the monitored living being so that, when the information about the emotion of the monitored living being represents any one type of negative emotion among a plurality of predefined types of negative emotions, the degree of decline in the life safety of the monitored living being is different depending on the one type of negative emotion represented among the plurality of types of negative emotions.

[0209] By this configuration, even when the physical condition of the monitored living body expressed by the emotion or the like may differ depending on the type of the negative emotion, the control device 80 or the like can further appropriately estimate the degree of decline in the life safety of the monitored living body.

[0210] In this embodiment, the information about the facial expression of the monitored living body may include information about the age of the monitored living body. The evaluation information acquisition unit 805 may estimate the degree of decline in the life safety of the monitored living body so that the degree of decline in the life safety of the monitored living body is different depending on the age of the monitored living body.

[0211] Similarly, the evaluation information acquisition unit 2005 may estimate the degree of decline in the life safety of the monitored living body such that the degree of decline in the life safety of the monitored living body differs depending on the age of the monitored living body.

[0212] By this configuration, when characteristics such as physical health may differ depending on the age of the monitored living body, the control device 80 and the like can further appropriately estimate the degree of decline in the life safety of the monitored living body.

[0213] In this embodiment, the safety functions for ensuring the life safety of the monitored living body may include at least one of the following: a function of issuing an alarm about a left-behind monitored living body to the surroundings of the vehicle 100 through a sound output device 60 installed on the vehicle 100, a function of providing a notification about a left-behind monitored living body to a user terminal 300 used by a user of the vehicle 100, a function of providing a notification about a left-behind monitored living body to a predetermined facility set outside the vehicle 100, and a function of activating the air conditioning device of the vehicle 100.

[0214] With this configuration, the control device 80 and the like can specifically ensure the life safety of the monitored living body.

[0215] The embodiments have been described in detail; however, the present disclosure is not limited to the specific embodiments. Various modifications or improvements can be applied within the scope of the subject matter described in the appended claims.

Claims

1. Information processing equipment, It is characterized in that include: a first determination unit configured to determine whether a predetermined living body is left in the cabin of the vehicle; a facial expression information acquisition unit configured to acquire information about a facial expression of the face of the predetermined living being based on image data containing the face of the predetermined living being remaining in the cabin of the vehicle; wherein the information about the facial expression of the predetermined living being includes information about an emotion of the predetermined living being and information about a facial state of the predetermined living being; an estimating unit configured to estimate a degree of decrease in life safety of the predetermined living body based on the information on the facial expression of the predetermined living body acquired by the facial expression information acquiring unit, so that: When the information about the emotion of the predetermined living being represents a negative emotion, the degree of decrease in the life safety of the predetermined living being is greater than the degree of decrease in the life safety of the predetermined living being when the information about the emotion of the predetermined living being represents a positive emotion or a neutral emotion; and When the information about the facial state of the predetermined living being represents a negative facial state, the degree of decrease in the life safety of the predetermined living being is greater than when the information about the facial state of the predetermined living being represents a positive facial state or a neutral facial state of the predetermined living being; and The estimation unit is configured to estimate the degree of decrease in the life safety of the predetermined living body by using the following expression: EI = αt + β; and α=W+F; Wherein, EI represents an index value indicating the degree of decline of the life safety of the predetermined living body, α represents the progression rate, t represents the residual elapsed time, β represents the adjustment index, W represents the cabin environment index, and F represents the facial expression analysis index; a second judgment unit configured to judge whether the life safety of the predetermined living body is lower than or equal to a predetermined reference according to the estimation result of the estimation unit; and A control unit is configured to activate a safety function for ensuring the life safety of the predetermined living body when the second judgment unit judges that the life safety of the predetermined living body is lower than or equal to the predetermined reference.

2. The information processing device according to claim 1, It is characterized in that The estimation unit is configured to estimate the degree of decline in the life safety of the predetermined living being so that the decline in the life safety of the predetermined living being is accelerated as the information about the emotion of the predetermined living being changes from a state representing the positive emotion or the neutral emotion to a state representing the negative emotion, or as the information about the facial state of the predetermined living being changes from a state representing the positive facial state or the neutral facial state to a state representing the negative facial state.

3. The information processing device according to claim 2, Features: the estimating unit is configured to estimate a degree of decrease in the life safety of the predetermined living body such that the life safety of the predetermined living body decreases as a time elapses during which the predetermined living body has remained in the cabin of the vehicle; and The estimation unit is configured to estimate the degree of decline in the life safety of the predetermined living being so that, when the information about the emotion of the predetermined living being represents the negative emotion, the rate of decline in the life safety of the predetermined living being relative to the passage of time is higher than the rate of decline in the life safety of the predetermined living being relative to the passage of time when the information about the emotion of the predetermined living being represents the positive emotion or the neutral emotion, and when the information about the facial state of the predetermined living being represents the negative facial state, the rate of decline in the life safety of the predetermined living being relative to the passage of time is higher than the rate of decline in the life safety of the predetermined living being relative to the passage of time when the information about the facial state of the predetermined living being represents the positive facial state or the neutral facial state.

4. The information processing device according to any one of claims 1 to 3, Features: The estimation unit is configured to estimate a degree of decrease in the life safety of the predetermined living being so that, when the information about the emotion of the predetermined living being represents any one type of negative emotion among a plurality of predefined types of negative emotions, the degree of decrease in the life safety of the predetermined living being is different depending on the one type of negative emotion represented among the plurality of types of negative emotions.

5. The information processing device according to any one of claims 1 to 3, Features: The information about the facial expression of the predetermined living being includes information about the age of the predetermined living being; and The estimation unit is configured to estimate a degree of decline in the life safety of the predetermined living body such that the degree of decline in the life safety of the predetermined living body differs depending on the age of the predetermined living body.

6. The information processing device according to any one of claims 1 to 3, It is characterized in that The safety function for ensuring the life safety of the predetermined living body includes at least one of the following: A function of issuing an alarm about the predetermined living body left behind to the surroundings of the vehicle through a sound output device installed in the vehicle, a function of providing a notification about the left-behind predetermined living body to a terminal device used by a user of the vehicle, a function of providing a notification about the predetermined living body left behind to a predetermined facility provided outside the vehicle, and Activate a function of the vehicle's air conditioning system.

7. An information processing method performed by an information processing device, It is characterized in that The information processing method comprises: Determining whether a predetermined living body is left in the cabin of the vehicle; Based on image data containing the face of the predetermined living being remaining in the cabin of the vehicle, acquiring information about a facial expression of the face of the predetermined living being; wherein the information about the facial expression of the predetermined living being includes information about an emotion of the predetermined living being and information about a facial state of the predetermined living being; Based on the information about the facial expression of the predetermined living being, the degree of reduction in the life safety of the predetermined living being is estimated so that: When the information about the emotion of the predetermined living being represents a negative emotion, the degree of decrease in the life safety of the predetermined living being is greater than the degree of decrease in the life safety of the predetermined living being when the information about the emotion of the predetermined living being represents a positive emotion or a neutral emotion; and When the information about the facial state of the predetermined living being represents a negative facial state, the degree of decrease in the life safety of the predetermined living being is greater than when the information about the facial state of the predetermined living being represents a positive facial state or a neutral facial state of the predetermined living being; and The degree of decline in the life safety of the predetermined living body is estimated by the following expression: EI = αt + β; and α=W+F; Wherein, EI represents an index value indicating the degree of decline of the life safety of the predetermined living body, α represents the progression rate, t represents the residual elapsed time, β represents the adjustment index, W represents the cabin environment index, and F represents the facial expression analysis index; determining whether the life safety of the predetermined living being is lower than or equal to a predetermined reference according to the estimated degree of decline of the life safety of the predetermined living being; and When it is judged that the life safety of the predetermined living body is lower than or equal to the predetermined reference, a safety function for ensuring the life safety of the predetermined living body is activated.

8. A non-transitory storage medium storing instructions that are executable by one or more processors and cause the one or more processors to perform functions, It is characterized in that The features include: Determining whether a predetermined living body is left in the cabin of the vehicle; Based on image data containing the face of the predetermined living being remaining in the cabin of the vehicle, acquiring information about a facial expression of the face of the predetermined living being; wherein the information about the facial expression of the predetermined living being includes information about an emotion of the predetermined living being and information about a facial state of the predetermined living being; Based on the information about the facial expression of the predetermined living being, the degree of reduction in the life safety of the predetermined living being is estimated so that: When the information about the emotion of the predetermined living being represents a negative emotion, the degree of decrease in the life safety of the predetermined living being is greater than the degree of decrease in the life safety of the predetermined living being when the information about the emotion of the predetermined living being represents a positive emotion or a neutral emotion; and When the information about the facial state of the predetermined living being represents a negative facial state, the degree of decrease in the life safety of the predetermined living being is greater than when the information about the facial state of the predetermined living being represents a positive facial state or a neutral facial state of the predetermined living being; and The degree of decline in the life safety of the predetermined living body is estimated by the following expression: EI = αt + β; and α=W+F; Wherein, EI represents an index value indicating the degree of decline of the life safety of the predetermined living body, α represents the progression rate, t represents the residual elapsed time, β represents the adjustment index, W represents the cabin environment index, and F represents the facial expression analysis index; determining whether the life safety of the predetermined living being is lower than or equal to a predetermined reference according to the estimated degree of decline of the life safety of the predetermined living being; and When it is judged that the life safety of the predetermined living body is lower than or equal to the predetermined reference, a safety function for ensuring the life safety of the predetermined living body is activated.

Images