Information processing device and information processing method

Abstract

An information processing device (100, 200, 300, 400) comprises: an occupant information acquisition unit (110) that acquires occupant information including image information regarding an occupant of a vehicle; a protective action detection unit (120) that detects, on the basis of the image information acquired by the occupant information acquisition unit (110), a protective action taken by the occupant to protect oneself from a specific moving object moving in the interior of the vehicle; an occupant condition detection unit (130) that detects, on the basis of the occupant information acquired by the occupant information acquisition unit (110), a change in the occupant condition from a first condition to a second condition; and a determination unit (140) that determines whether or not the change in the occupant from the first condition to the second condition has been detected by the occupant condition detection unit (130) in association with the detection of the protective action by the protective action detection unit (120).

Description

Information processing device and information processing method

[0001] The present disclosure relates to an information processing device and an information processing method.

[0002] Conventionally, a foreign object ejection device for ejecting insects that have invaded a vehicle cabin to the outside of the vehicle cabin has been disclosed (see, for example, Patent Document 1). This foreign object ejection device is configured to detect hand or arm movements of a driver in the vehicle cabin to scare away insects from an image of the interior of the vehicle obtained from a camera disposed in the vehicle cabin, and to open or close a vehicle window when the driver's hand or arm movements to scare away insects are detected.

[0003] JP 2024-000831 A

[0004] However, in general, defensive actions taken by occupants to protect themselves from moving objects such as insects moving within the vehicle cabin can be difficult to distinguish from other actions, and there is a need to improve the reliability of detection when detecting such defensive actions.

[0005] The present disclosure was made in response to the recognition of the above-mentioned problems, and aims to provide an information processing device and an information processing method that can improve the reliability of defensive actions taken by occupants to protect themselves from specific moving objects moving within the vehicle cabin compared to conventional methods.

[0006] The information processing device of the present disclosure is characterized by comprising an occupant information acquisition unit that acquires occupant information including image information of the occupants of the vehicle; a defensive action detection unit that detects defensive actions taken by the occupant to protect themselves from a specific moving object moving within the vehicle cabin based on the image information acquired by the occupant information acquisition unit; an occupant state detection unit that detects a change in the occupant's state from a first state to a second state based on the occupant information acquired by the occupant information acquisition unit; and a determination unit that determines whether or not a change in the occupant's state from the first state to the second state has been detected by the occupant state detection unit following the detection of a defensive action by the defensive action detection unit.

[0007] The information processing device according to the present disclosure can improve the reliability of the defensive action of an occupant to protect themselves from a specific moving object moving inside the passenger compartment of a vehicle compared to conventional methods.

[0008] 1 is a block diagram showing a schematic configuration of an information processing system according to a first embodiment. FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device according to the first embodiment. FIG. 3 is a block diagram showing an example of a hardware configuration of an information processing device according to the first embodiment. FIG. 4 is a flowchart showing an example of processing performed by the information processing device according to the first embodiment. FIG. 5 is a block diagram showing a schematic configuration of an information processing system according to a second embodiment. FIG. 6 is a flowchart showing an example of processing performed by the information processing device according to the second embodiment. FIG. 7 is a block diagram showing a schematic configuration of an information processing system according to a third embodiment. FIG. 8 is a flowchart showing an example of processing performed by the information processing device according to the third embodiment. FIG. 9 is a block diagram showing a schematic configuration of an information processing system according to a fourth embodiment. FIG. 10 is a flowchart showing an example of processing performed by the information processing device according to the fourth embodiment. FIG. 11 is a block diagram showing a schematic configuration of an information processing system according to a fifth embodiment. FIG. 12 is a flowchart showing an example of processing performed by the information processing device according to the fifth embodiment. FIG. 13 is a block diagram showing a schematic configuration of an information processing system according to a sixth embodiment. FIG. 14 is a flowchart showing an example of processing performed by the information processing device according to the sixth embodiment. FIG. 15 is a block diagram showing a schematic configuration of an information processing system according to a seventh embodiment. FIG. 16 is a flowchart showing an example of processing performed by the information processing device according to the seventh embodiment. FIG. 17 is a block diagram showing a schematic configuration of an information processing system according to an eighth embodiment. FIG. 18 is a flowchart showing an example of processing for machine learning performed by the information processing device according to the eighth embodiment.

[0009] Hereinafter, embodiments according to the present disclosure will be described in detail with reference to the drawings. Embodiment 1. First, a schematic configuration of an information processing system according to embodiment 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing a schematic configuration of the information processing system according to embodiment 1. The information processing system according to embodiment 1 is a system for detecting a defensive action of an occupant to protect themselves from a specific moving object in the cabin of a specific vehicle. As shown in FIG. 1, the information processing system according to embodiment 1 includes an occupant information acquisition device 10, an external device 20, and an information processing device 100, which are electrically connected wirelessly or by wire so as to be able to communicate with each other. In embodiment 1, an "occupant" refers to either a driver who drives a specific vehicle in the cabin of the vehicle, or an occupant other than the driver who is in the cabin of the vehicle.

[0010] The occupant information acquisition device 10 acquires occupant information including image information of occupants of a vehicle (not shown). Specifically, the occupant information acquisition device 10 acquires occupant information, which is information about occupants while the vehicle is traveling, including image information of the occupants while the vehicle is traveling. For example, the occupant information acquisition device 10 is configured as part of a DMS (Driver Monitoring System) or an OMS (Occupant Monitoring System) that acquires information about occupants in the vehicle cabin, and has an imaging device (not shown) that captures an image of a specific area including the occupants in the vehicle cabin to generate image information indicating the occupants' visual information. For example, the imaging device has an imaging element (image sensor) and a lens (not shown), and generates image information of the occupants by converting light captured through the lens into an electrical signal using the imaging element. The imaging device has an imaging element such as a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. For example, the imaging device is disposed in a vehicle cabin and captures an image of the interior of the vehicle cabin at a preset frame rate to generate image information of a range including the upper body of an occupant sitting in a seat in the vehicle cabin.

[0011] The imaging device may be any device that generates image information that indicates the visual information of the occupant. For example, the imaging device may be a visible light camera that generates image information based on visible light, an infrared camera that generates image information based on infrared light, a thermography camera that generates image information based on infrared light, a millimeter-wave imaging radar that generates image information by detecting reflected waves of output millimeter waves, or a LiDER (Light Detection and Ranging) sensor that generates image information.

[0012] For example, the occupant information acquisition device 10 includes a biosensor that acquires bioinformation generated based on the bioactivity of the occupant. For example, the biosensor is configured as a contact or non-contact sensor disposed on a steering wheel (not shown) that acquires bioinformation related to the driver gripping the steering wheel, such as information indicating the driver's pulse rate, blood pressure, blood flow, etc. For example, the biosensor is configured as a wearable device such as a smartwatch worn by the occupant, and acquires information indicating the occupant's pulse rate, blood pressure, blood flow, etc. using the contact or non-contact sensor. For example, the occupant information acquisition device 10 includes a microphone that acquires audio information indicating the voice of the occupant. The imaging device outputs the generated occupant information to the information processing device 100.

[0013] The external device 20 performs a specific process based on the information input from the information processing device 100. For example, the external device 20 is configured as a storage device arranged inside a vehicle and storing the information input from the information processing device 100. Alternatively, for example, the external device 20 is configured as an alarm device arranged inside or outside the vehicle and notifying the information input from the information processing device 100 by video or audio. Alternatively, for example, the external device 20 is configured as a vehicle control device arranged inside or outside the vehicle and controlling devices provided in the vehicle based on the information input from the information processing device 100. For example, the external device 20 as a vehicle control device controls an air conditioning device, a window opening / closing device, and a driving assistance device that controls the speed and direction of the vehicle based on the information input from the information processing device 100.

[0014] The information processing device 100 includes an occupant information acquisition unit 110 , a defensive action detection unit 120 , an occupant state detection unit 130 , and a determination unit 140 .

[0015] The occupant information acquisition unit 110 acquires occupant information including image information of the vehicle occupant based on information from the occupant information acquisition device 10. Specifically, the occupant information acquisition unit 110 acquires occupant information, which is information about the occupant while the vehicle is traveling, including image information of the occupant while the vehicle is traveling, based on information from the occupant information acquisition device 10. For example, the occupant information acquisition unit 110 acquires image information of the occupant at a preset frame rate. Furthermore, for example, the occupant information acquisition unit 110 acquires occupant information including occupant biometric information and occupant voice information in addition to the occupant image information as occupant information.

[0016] The defensive action detection unit 120 detects a defensive action of an occupant to protect themselves from a specific moving object moving within the vehicle cabin, based on image information included in the occupant information acquired by the occupant information acquisition unit 110. In other words, the defensive action detection unit 120 detects a predetermined specific action that is estimated to be a defensive action of an occupant to protect themselves from a specific moving object moving within the vehicle cabin, based on image information included in the occupant information acquisition unit 110. For example, the defensive action detection unit 120 detects, by image analysis based on the image information included in the occupant information acquired by the occupant information acquisition unit 110, a predetermined specific action of at least one of the occupant's hands, arms, head, and face as a defensive action of an occupant to protect themselves from a specific moving object moving within the vehicle cabin. For example, specific moving objects moving within the vehicle cabin include insects, animals, other occupants, and floating objects with odors, such as smoke, dust, mist-like liquid, and gas.

[0017] Furthermore, for example, the defensive movement detection unit 120 extracts feature amounts indicating the positions of specific body parts of the occupant from the image information included in the occupant information acquired by the occupant information acquisition unit 110, and detects the defensive movement of the occupant based on changes in the extracted feature amounts. Furthermore, for example, the defensive movement detection unit 120 detects the defensive movement of the occupant using a trained model that is stored in the information processing device 100 or the external device 20 and that infers the presence or absence of a defensive movement of the occupant related to the occupant information based on input of the occupant information acquired by the occupant information acquisition unit 110. Note that since the defensive movement detection unit 120 can detect the defensive movement of the occupant based on the occupant information acquired by the occupant information acquisition unit 110, the occupant information acquired by the occupant information acquisition unit 110 can be said to include information indicating the defensive movement of the occupant. Details of the defensive movements detected by the defensive movement detection unit 120 will be described later.

[0018] The occupant state detection unit 130 detects a change in the state of the occupant from a first state to a second state based on the occupant information acquired by the occupant information acquisition unit 110. For example, the occupant state detection unit 130 detects a change in the occupant's line of sight, facial orientation, facial expression, posture, mouth shape, and voice state as a change in the state of the occupant's state from the first state to the second state based on the occupant information acquired by the occupant information acquisition unit 110. Furthermore, for example, the occupant state detection unit 130 detects a change in the state of the occupant from the first state to the second state using a trained model that is stored in the information processing device 100 or the external device 20 and that infers whether or not the state of the occupant related to the occupant information has changed from the first state to the second state based on the occupant information acquired by the occupant information acquisition unit 110.

[0019] Note that the occupant state detection unit 130 can detect a change in state from the first state to the second state based on the occupant information acquired by the occupant information acquisition unit 110. Therefore, the occupant information acquired by the occupant information acquisition unit 110 can be said to include information indicating a change in the occupant's state from the first state to the second state. For example, if the occupant is a driver, the first state is a normal state indicating a high level of concentration on driving, and the second state is an abnormal state indicating a lower level of concentration on driving than the first state. Details of the change in the occupant's state from the first state to the second state detected by the occupant state detection unit 130 will be described later.

[0020] The determination unit 140 determines whether the occupant state detection unit 130 has detected a change of the occupant from the first state to the second state in response to the detection of a defensive action by the defensive action detection unit 120. For example, the determination unit 140 determines whether the occupant state detection unit 130 has detected a change of the occupant from the first state to the second state in response to the detection of a defensive action by the defensive action detection unit 120 by determining whether the occupant state detection unit 130 has detected a change of the occupant from the first state to the second state in response to the detection of a defensive action by the defensive action detection unit 120. Specifically, the determination unit 140 determines whether the occupant state detection unit 130 has detected a change of the occupant from the first state to the second state in response to the detection of a defensive action by the defensive action detection unit 120 in a 30-second period starting from the time the defensive action was detected by the defensive action detection unit 120. The determination unit 140 outputs the determination result to the external device 20.

[0021] Generally, the defensive actions of an occupant to protect themselves from a specific moving object moving within the vehicle cabin as described above are often relatively simple actions. Even if an occupant's action matching a preset action presumed to be an occupant's defensive action is detected, it is possible that the occupant's action is not actually a defensive action. The information processing device 100 according to the first embodiment focuses on the fact that when an occupant performs a defensive action, a characteristic change in the occupant's state is often observed in the occupant's line of sight, facial orientation, etc., and, upon detecting an occupant's defensive action, determines whether there has been a characteristic change in the occupant's state associated with the detected defensive action, thereby improving the reliability of the defensive action detection result compared to conventional methods. In particular, when the occupant is a driver of a vehicle, a characteristic change in the state indicating a decrease in the occupant's concentration on driving often occurs when the occupant performs a defensive action. The information processing device 100 detects a change in the driver's state indicating a decrease in the driver's concentration using the occupant state detection unit 130, thereby improving the reliability of the defensive action detection result compared to conventional methods.

[0022] Specific examples of the occupant's defensive actions and the occupant's change from the first state to the second state will be described below. For example, the defensive action detection unit 120 detects, as the occupant's defensive actions, an action of the occupant's hand moving near the occupant's head and an action of the occupant's hand hitting an object around the occupant. The occupant state detection unit 130 detects, as the changes from the first state to the second state, a change in the occupant's gaze direction from a state where it is not directed toward the occupant's moving hand to a state where it is directed toward the occupant's moving hand, a change in the occupant's facial direction from a state where it is not directed toward the occupant's moving hand to a state where it is directed toward the occupant's moving hand, a change in the amount of change per unit time of the occupant's gaze direction from a state where it is small to a state where it is large, a change in the amount of change per unit time of the occupant's facial direction from a state where it is small to a state where it is large, a change in the occupant's gaze direction from a state where it is within a specific range to a state where it is outside the specific range, a change in the occupant's discomfort level from a state where it is low to a state where it is high, and a change in the occupant's voice from a state where it is quiet to a state where it is loud. For example, the action of an occupant's hand moving near the occupant's head corresponds to the action of protecting oneself from a specific moving object, such as an insect or small animal, that has invaded the vehicle cabin by brushing away the specific moving object, and the action of an occupant's hand hitting an object around the occupant corresponds to the action of protecting oneself from a specific moving object, such as an insect or small animal, that has invaded the vehicle cabin by brushing away the specific moving object.

[0023] Furthermore, for example, a change from a state in which the amount of change per time in the occupant's gaze direction or the occupant's facial orientation is small to a state in which the amount of change in angle per second in the occupant's gaze direction or the occupant's facial orientation is detected based on image information included in the occupant information acquired by the occupant information acquisition unit 110 from a state in which the amount of change per second in the occupant's gaze direction or the occupant's facial orientation is small to a state in which the amount of change per second in ... Furthermore, for example, a change in the gaze direction of an occupant from a state in which it is within a specific range to a state in which it is outside the specific range is detected based on image information contained in the occupant information acquired by the occupant information acquisition unit 110, and is a change in the gaze direction of the driver as an occupant from a state in which it is within the range in which the vehicle's windshield is formed to a state in which it is outside the range in which the specific windshield is formed, and similarly corresponds to a change when the gaze direction is moved to chase a specific moving object such as an insect or small animal that has entered the vehicle cabin.

[0024] Furthermore, the change from a state indicating that the occupant's discomfort level is low to a state indicating that it is high is a change in the occupant's facial expression from a state indicating that the occupant's discomfort level is low to a state indicating that it is high, which is detected based on image information included in the occupant information acquired by the occupant information acquisition unit 110, and corresponds to a change in the occupant's discomfort level increasing due to a specific moving object such as an insect or small animal that has entered the vehicle cabin. Furthermore, for example, the change from a state indicating that the occupant's discomfort level is low to a state indicating that it is high is a change in the pitch, volume, and content of the occupant's voice from a state indicating that the occupant's discomfort level is low to a state indicating that it is high, which is detected based on audio information included in the occupant information acquired by the occupant information acquisition unit 110.

[0025] For example, such a change in the content of the voice may be a change from not making an exclamation, a shout, a scream, or the like, which indicates the occupant's high level of discomfort, to making the same, or a change from not making a sound to making a sound made when scaring away a specific moving object such as an insect or a small animal. Furthermore, a change from a quiet to a loud voice by the occupant is a change from a voice made by the occupant that is quieter than a preset threshold to a voice made louder, which is detected based on image information included in the occupant information acquired by the occupant information acquisition unit 110. The occupant state detection unit 130 may be configured to detect these changes by estimating changes in the pitch, volume, and content of the occupant's voice based on visual changes, such as changes in the shape and opening of the occupant's mouth, which are detected based on image information included in the occupant information acquired by the occupant information acquisition unit 110.

[0026] For example, if such a defensive action by the occupant and a change in the occupant's state from the first to the second state accompanying the defensive action are both detected, it is inferred that the occupant's action detected by the defensive action detection unit 120 is a defensive action to protect the occupant from a specific moving object such as an insect or small animal that has entered the vehicle cabin, and there is a high probability that it is actually a defensive action by the occupant.

[0027] For example, the defensive action detection unit 120 detects, as the occupant's defensive actions, actions of the occupant's hands chasing a specific moving object, actions of the occupant's hands moving from the occupant's upper body toward a specific moving object, actions of the occupant's arms moving between the occupant and a specific moving object, and actions of the occupant's hands moving to cover the occupant's head. The occupant state detection unit 130 detects, as the occupant's changes from a first state to a second state, actions of the occupant's upper body changing from not tilted to tilted toward a specific moving object, a change in the occupant's eyebrow wrinkles from small to large, a change in the degree of eyelid opening from small to large, and a change in the occupant's voice from small to large. For example, the actions of the occupant's hands chasing a specific moving object and the actions of the occupant's hands moving from the occupant's upper body toward a specific moving object correspond to actions to protect oneself from a specific moving object, such as a suspicious person or animal, that has entered the vehicle cabin by threatening the specific moving object. Furthermore, for example, the action of an occupant's arm moving between the occupant and a specific moving object, or the action of the occupant's hand moving to cover the occupant's head, corresponds to an action of protecting oneself from a specific moving object such as a suspicious person or animal that has entered the vehicle cabin.

[0028] For example, a change in the occupant's upper body from a non-inclined state to a state in which it is inclined toward a specific moving object corresponds to a change from not pushing a specific moving object, such as a suspicious person or animal that has entered the vehicle cabin, to pushing the specific moving object away from the occupant, as detected based on image information included in the occupant information acquired by the occupant information acquisition unit 110. For example, a change in the occupant's eyebrow wrinkles from a small state to a large state and a change in the degree of eyelid opening from a small state to a large state correspond to a change from the occupant not threatening a specific moving object, such as a suspicious person or animal that has entered the vehicle cabin, to a threatening state, as detected based on image information included in the occupant information acquired by the occupant information acquisition unit 110. For example, a change in the occupant's voice from a low state to a high state corresponds to a change from the occupant not threatening a specific moving object, such as a suspicious person or animal that has entered the vehicle cabin, to a threatening state, as detected based on audio information included in the occupant information acquired by the occupant information acquisition unit 110.

[0029] For example, if such a defensive action by the occupant and a change in the occupant's state from the first to the second state accompanying the defensive action are both detected, it is inferred that the occupant's action detected by the defensive action detection unit 120 is a defensive action to protect the occupant from a specific moving object such as a suspicious person or animal that has entered the vehicle cabin, and there is a high probability that it is actually a defensive action by the occupant.

[0030] For example, the defensive action detection unit 120 detects, as the occupant's defensive action, an action of the occupant's hand moving back and forth near the occupant's head and in front of the occupant's face, and an action of the occupant's hand pinching the occupant's nose. The occupant state detection unit 130 detects, as the occupant's change from a state indicating a low level of discomfort to a state indicating a high level of discomfort, as the occupant's change from a first state to a second state. For example, the action of the occupant's hand moving back and forth near the occupant's head and in front of the occupant's face corresponds to a protective action against a specific moving object, such as a floating object that has entered the vehicle cabin or that has been generated within the vehicle cabin and is accompanied by an odor. For example, the action of the occupant pinching the occupant's nose corresponds to a protective action against a specific moving object, such as a floating object that has entered the vehicle cabin or that is accompanied by an odor, by pinching the occupant's nose.

[0031] Furthermore, for example, a change from a state indicating that the occupant's discomfort level is low to a state indicating that it is high is a change in the occupant's facial expression from a state indicating that the occupant's discomfort level is low to a state indicating that it is high, which is detected based on image information contained in the occupant information acquired by the occupant information acquisition unit 110, and corresponds to a change in which the occupant's discomfort level increases due to a specific moving object such as an odorous floating object.

[0032] In this way, when the information processing device 100 detects a defensive movement by an occupant, it detects characteristic changes in the occupant's condition that accompany the detected defensive movement, thereby improving the reliability of the defensive movement detection results compared to conventional methods.

[0033] Next, the hardware configuration of the information processing device 100 will be described with reference to Figures 2 and 3. Figure 2 is a diagram showing an example of the hardware configuration of the information processing device 100, and Figure 3 is a diagram showing an example of the hardware configuration of the information processing device 100 that is different from that shown in Figure 2. For example, as shown in Figure 2, the information processing device 100 is configured as a computer having a processor 100a, a memory 100b, and an I / O port 100c, and is configured so that the processor 100a reads and executes a program stored in the memory 100b. The memory 100b is configured, for example, by RAM (Random Access Memory), ROM (Read Only Memory), flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), etc.

[0034] 3, the information processing device 100 is configured as a computer that includes a processing circuit 100d, which is dedicated hardware, and an I / O port 100c, and executes a program. The processing circuit 100d is configured, for example, by a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. Each function of the information processing device 100 is realized by the processor 100a or the processing circuit 100d, which is dedicated hardware, executing a program. The information processing device 100 may also include hardware other than those described above, such as a hardware timer.

[0035] Next, details of the processing performed by the information processing device 100 will be described with reference to Figures 1 and 4. Figure 4 is a flowchart showing an example of the processing performed by the information processing device 100 according to the first embodiment. As shown in Figure 4, when the information processing device 100 starts the processing, it first acquires occupant information (step ST01). In this processing, the information processing device 100 acquires occupant information including image information of the vehicle occupant using the occupant information acquisition unit 110 based on information from the occupant information acquisition device 10.

[0036] After performing the process of step ST01, the information processing device 100 determines whether a defensive action has been detected (step ST02). In this process, the information processing device 100 detects, by the defensive action detection unit 120, a defensive action of an occupant to protect themselves from a specific moving object moving in the passenger compartment of the vehicle, based on the image information included in the occupant information acquired by the occupant information acquisition unit 110.

[0037] If a defensive action is detected in the process of step ST02 (YES in step ST02), the information processing device 100 determines whether a change in the occupant's state from the first state to the second state has been detected within a specific period (step ST03). In this process, the information processing device 100 detects a change in the occupant's state from the first state to the second state using the occupant state detection unit 130 based on the occupant information acquired by the occupant information acquisition unit 110.

[0038] If a change in the occupant's state from the first state to the second state within the specific period is detected in the processing of step ST03 (YES in step ST03), the information processing device 100 outputs a determination result (step ST04). In this processing, the information processing device 100 determines, by the determination unit 140, whether or not a change in the occupant's state from the first state to the second state associated with the detection of the defensive action by the occupant has been detected, based on the fact that a change in the occupant's state from the first state to the second state has been detected in the processing of step ST03 during the specific period including the time when the defensive action was detected in the processing of step ST02.

[0039] If no defensive action is detected in the process of step ST02 (NO in step ST02), if no change in the occupant's state from the first to the second state within the specified period is detected in the process of step ST03 (NO in step ST03), or if the process of step ST04 is performed, the information processing device 100 determines whether or not the vehicle has stopped driving (step ST09). In this process, the information processing device 100 acquires information indicating, for example, the state of the power switch, the state of the ignition switch, the position of the shift lever, etc. of the vehicle from a vehicle control device (not shown), and determines whether or not the driver has stopped driving the vehicle based on the acquired information.

[0040] If the vehicle has not finished traveling in the processing of step ST09 (NO in step ST09), the information processing device 100 returns the processing to step ST01, and if the vehicle has finished traveling in the processing of step ST09 (YES in step ST09), the information processing device 100 ends the processing.

[0041] As described above, the information processing device 100 according to the first embodiment includes an occupant information acquisition unit that acquires occupant information including image information of the occupants of the vehicle, a defensive action detection unit that detects defensive actions taken by the occupant to protect themselves from a specific moving object moving within the vehicle cabin based on the image information acquired by the occupant information acquisition unit, an occupant state detection unit that detects a change in the occupant state from a first state to a second state based on the occupant information acquired by the occupant information acquisition unit, and a determination unit that determines whether or not a change in the occupant state from the first state to the second state has been detected by the occupant state detection unit following the detection of a defensive action by the defensive action detection unit.

[0042] For example, the information processing device 100 detects a specific movement of at least one of the occupant's hands, arms, head, and face as a defensive movement, and based on the image information acquired by the occupant information acquisition unit 110, detects that the occupant's line of sight, facial direction, facial expression, posture, mouth shape, and voice state have changed as a change from the first state to the second state.

[0043] Furthermore, for example, the information processing device 100 detects that the state of the occupant has changed from a first state to a second state based on the image information acquired by the occupant information acquisition unit 110 .

[0044] When the information processing device 100 detects a specific action that is presumed to be a defensive action by an occupant, the information processing device 100 determines whether a characteristic change in the occupant's state associated with the detected action has been detected, thereby improving the reliability of detecting a defensive action by an occupant to protect themselves from a specific moving object moving within the vehicle cabin compared to the conventional method. Furthermore, by improving the reliability of detecting a defensive action by an occupant, it becomes possible to indirectly detect a specific moving object within the vehicle cabin based on the detection result of the defensive action with higher accuracy than conventional methods, without directly detecting the specific moving object within the vehicle cabin.

[0045] In the first embodiment, the information processing device 100 is configured to detect, as a change from the first state to the second state of the occupant, changes in the occupant's line of sight, facial orientation, facial expression, posture, mouth shape, and voice state, based on the image information acquired by the occupant information acquisition unit 110. However, the present invention is not limited to this. The information processing device may be configured to detect, as a change from the first state to the second state of the occupant due to the occupant's defensive action, based on the occupant information acquired by the occupant information acquisition unit 110. For example, the information processing device may be configured to detect, as a change from the first state to the second state of the occupant due to the occupant's defensive action, changes in the occupant's pulse rate, blood pressure, and blood flow, based on the occupant's biometric information included in the occupant information acquired by the occupant information acquisition unit 110. Alternatively, the information processing device may be configured to detect, as a change from the first state to the second state of the occupant due to the occupant's defensive action, changes in the occupant's pulse rate, blood pressure, and blood flow, based on visual changes in the occupant's skin, based on the image information of the occupant included in the occupant information acquired by the occupant information acquisition unit 110.

[0046] Second Embodiment Next, an information processing system according to a second embodiment will be described with reference to Figures 5 and 6. The information processing system according to the second embodiment differs from the information processing system according to the first embodiment in that the configuration of the information processing device is partially different, but the other configurations are the same. The same configurations as those in the first embodiment are given the same names and symbols as those in the first embodiment, and the description thereof will be omitted.

[0047] Fig. 5 is a block diagram showing a schematic configuration of an information processing system according to embodiment 2. As shown in Fig. 5, the information processing system according to embodiment 2 includes an occupant information acquisition device 10, an external device 20, and an information processing device 200, which are electrically connected wirelessly or by wire so as to be able to communicate with each other.

[0048] The information processing device 200 includes an occupant information acquisition unit 110 , a defensive action detection unit 120 , an occupant state detection unit 130 , a determination unit 140 , and a classification unit 250 .

[0049] The classification unit 250 classifies a specific moving object present in the vehicle cabin based on the occupant information acquired by the occupant information acquisition unit 110. For example, in a case where the defensive action detection unit 120 is configured to be capable of detecting that an occupant has performed any one of a plurality of preset defensive actions, and the occupant state detection unit 130 is configured to be capable of detecting that the occupant's state has changed from a first state to any one of a plurality of preset second states, the classification unit 250 classifies the specific moving object based on a combination of any one of the defensive actions detected by the defensive action detection unit 120 and any one of the changes to the second states detected by the occupant state detection unit 130.

[0050] For example, when the defensive action detection unit 120 detects, as the occupant's defensive action, either one of an action of the occupant's hand moving near the occupant's head or an action of the occupant's hand hitting an object around the occupant, and the occupant state detection unit 130 detects, as the change from the first state to the second state, any one of a change in the occupant's gaze direction from a state where it is not directed toward the occupant's moving hand to a state where it is directed toward the occupant's moving hand, a change in the occupant's facial direction from a state where it is not directed toward the occupant's moving hand to a state where it is directed toward the occupant's moving hand, a change in the amount of change per unit time of the occupant's gaze direction from a state where it is small to a state where it is large, a change in the amount of change per unit time of the occupant's facial direction from a state where it is small to a state where it is large, a change in the occupant's gaze direction from a state where it is within a specific range to a state where it is outside the specific range, a change from a state indicating a low level of discomfort to a state indicating a high level of discomfort, or a change in the occupant's voice from a state where it is quiet to a state where it is loud, the classification unit 250 classifies a specific moving object estimated to be present in the vehicle cabin into a first class corresponding to an insect, a small animal, or the like.

[0051] Furthermore, for example, when the defensive movement detection unit 120 detects, as the occupant's defensive movement, any one of the following movements: the occupant's hand chasing a specific moving object; the occupant's hand moving from the occupant's upper body toward the specific moving object; the occupant's arm moving between the specific moving object and the occupant; and the occupant's hand moving to cover the occupant's head; and the occupant state detection unit 130 detects, as the occupant's change from the first state to the second state, any one of the following movements: the occupant's upper body changing from not tilted to tilted toward the specific moving object; the occupant's eyebrow wrinkles changing from small to large; the occupant's eyelid opening changing from small to large; and the occupant's voice changing from small to large; the classification unit 250 classifies the specific moving object estimated to be present in the vehicle cabin into a second class corresponding to a suspicious person, an animal, or the like.

[0052] Furthermore, for example, when the defensive movement detection unit 120 detects, as a defensive movement of the occupant, one of the movement of the occupant's hands moving back and forth near the occupant's head and in front of the occupant's face, and the occupant state detection unit 130 detects, as a change from the first state to the second state of the occupant, a change from a state indicating a low level of discomfort to a state indicating a high level of discomfort, the classification unit 250 classifies a specific moving object estimated to be present in the vehicle cabin into a third class corresponding to odorous floating objects, etc. The classification unit 250 outputs the classification result to the external device 20.

[0053] The hardware configuration of the information processing device 200 according to the second embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0054] Next, details of the processing performed by the information processing device 200 will be described with reference to Figures 5 and 6. Figure 6 is a flowchart showing an example of the processing performed by the information processing device 200 according to embodiment 2. Note that part of the processing performed by the information processing device 200 according to embodiment 2 is similar to the processing performed by the information processing device 100 according to embodiment 1, and therefore, the same processes as those in embodiment 1 are denoted by the same reference numerals as those in embodiment 1 and will not be described again. As shown in Figure 6, when the information processing device 200 starts processing, it first acquires occupant information (step ST01).

[0055] After performing the processing of step ST01, the information processing device 200 determines whether or not a defensive action has been detected (step ST02). If a defensive action has been detected in the processing of step ST02 (YES in step ST02), the information processing device 200 determines whether or not a change in the occupant's state from a first state to a second state within a specific period has been detected (step ST03). If a change in the occupant's state from the first state to the second state within a specific period has been detected in the processing of step ST03 (YES in step ST03), the information processing device 200 classifies the moving object based on a combination of the detected defensive action and the second state (step ST05). In this processing, the information processing device 200 classifies a specific moving object estimated to be present in the vehicle cabin based on a combination of the occupant's defensive action detected by the defensive action detection unit 120 and the change in the occupant's state detected by the occupant state detection unit 130.

[0056] After completing the process of step ST05, the information processing device 200 outputs the classification result (step ST06). In this process, the information processing device 200, for example, outputs information indicating the class of the classification result of the specific moving object to the external device 20, and notifies the occupant or the operator of the external device 20 of the classification result of the specific moving object. Also, for example, the information processing device 200 outputs information indicating the class of the classification result of the specific moving object to the external device 20, and stores the information indicating the classification result of the specific moving object in the external device 20. Also, for example, the information processing device 200 outputs information indicating the class of the classification result of the specific moving object to the external device 20, and causes the external device 20 to perform a specific process based on the classification result of the specific moving object.

[0057] If no defensive action is detected in the processing of step ST02 (NO in step ST02), if no change in the occupant's state from the first to the second state within the specific period is detected in the processing of step ST03 (NO in step ST03), or if the processing of step ST06 is performed, the information processing device 200 determines whether or not the vehicle has stopped traveling (step ST09). If the vehicle has not stopped traveling in the processing of step ST09 (NO in step ST09), the information processing device 200 returns the processing to step ST01, and if the vehicle has stopped traveling in the processing of step ST09 (YES in step ST09), the information processing device 200 ends the processing.

[0058] As described above, the information processing device 200 according to the second embodiment includes a classification unit 250 that classifies specific moving objects, the defensive action detection unit 120 is capable of detecting that an occupant has performed any one of a plurality of preset defensive actions, the occupant state detection unit 130 is capable of detecting that the occupant's state has changed from a first state to any one of a plurality of preset second states, and the classification unit 250 is configured to classify specific moving objects based on a combination of any one of the defensive actions detected by the defensive action detection unit 120 and the change to any one of the second states detected by the occupant state detection unit 130.

[0059] For example, the classification unit 250 is configured to classify a specific moving object into a first class when the defensive action detection unit 120 detects, as one of the defensive actions, either an action of the occupant's hand moving near the occupant's head or an action of the occupant's hand hitting a surrounding object, and the occupant state detection unit 130 detects, as a change from the first state to any of the second states, any one of the following: a change in the occupant's gaze direction from a state where it is not directed toward the occupant's moving hand to a state where it is directed toward the occupant's moving hand; a change in the occupant's facial direction from a state where it is not directed toward the occupant's moving hand to a state where it is directed toward the occupant's moving hand; a change in the amount of change per unit time of the occupant's gaze direction from a state where it is small to a state where it is large; a change in the occupant's gaze direction from a state where it is within a specific range to a state where it is outside the specific range; a change from a state indicating a low level of discomfort to a state indicating a high level of discomfort; and a change in the occupant's voice from a state where it is quiet to a state where it is loud.

[0060] Furthermore, for example, the classification unit 250 is configured to classify the specific moving object into the second class when the defensive action detection unit 120 detects any one of the following defensive actions: an action of the occupant's hand chasing a specific moving object, an action of the occupant's hand moving from the occupant's upper body toward the specific moving object, an action of the occupant's arm moving between the specific moving object and the occupant, and an action of the occupant's hand moving to cover the occupant's head; and the occupant state detection unit 130 detects any one of the following changes from the first state to any second state: a change from a state in which the occupant's upper body is not tilted to a state in which it is tilted toward the specific moving object, a change from a state in which the occupant's eyebrow wrinkles are small to a state in which they are large, a change from a state in which the occupant's eyelids are small to a state in which they are large, and a change from a state in which the occupant's voice is small to a state in which it is loud.

[0061] Furthermore, for example, the classification unit 250 is configured to classify a specific moving object into a third class when the defensive action detection unit 120 detects either an action of the occupant's hand moving back and forth near the occupant's head and in front of the occupant's face, or an action of the occupant's hand pinching the occupant's nose, as one of the defensive actions, and the occupant state detection unit 130 detects either a change from a state indicating that the occupant's discomfort is low to a state indicating that the occupant's discomfort is high as a change from the first state to any of the second states.

[0062] With this configuration, the information processing device 200 can, for example, control various devices of the vehicle based on the classification result of the specific moving object, and collect information indicating the classification result of the specific moving object.

[0063] Embodiment 3 Next, an information processing system according to embodiment 3 will be described with reference to Figures 7 and 8. The information processing system according to embodiment 3 differs from the information processing system according to embodiment 1 in that it includes a vehicle control device and in that the configuration of a portion of the information processing device for controlling the vehicle control device is different, but the other configurations are the same, and the same configurations as those in embodiment 1 are given the same names and symbols as those in embodiment 1, and description thereof will be omitted.

[0064] Fig. 7 is a block diagram showing a schematic configuration of an information processing system according to embodiment 3. As shown in Fig. 7, the information processing system according to embodiment 3 includes an occupant information acquisition device 10, a vehicle control device 36, and an information processing device 300, which are electrically connected wirelessly or by wire so as to be able to communicate with each other.

[0065] The vehicle control device 36 controls a driving assistance device that controls the speed and direction of travel of the vehicle. For example, the vehicle control device 36 switches the autonomous driving level of the vehicle based on information input from the information processing device 300. Furthermore, for example, the vehicle control device 36 switches one or more of adaptive cruise control (ACC), autonomous emergency braking (AEB), and lane keep assist system (LKAS) between an on state and an off state based on information input from the information processing device 300.

[0066] The information processing device 300 includes an occupant information acquisition unit 110 , a defensive action detection unit 120 , an occupant state detection unit 130 , a determination unit 140 , and a vehicle control unit 360 .

[0067] The vehicle control unit 360 controls the vehicle based on the determination result by the determination unit 140. In other words, the vehicle control unit 360 outputs a control signal for controlling the vehicle to the vehicle control device 36 based on the determination result by the determination unit 140. For example, if the presence of a specific moving object in the vehicle cabin is detected, it is conceivable that the driver's concentration on driving may decrease due to the specific moving object. In contrast, the vehicle control unit 360 according to the third embodiment outputs a control signal to the vehicle control device 36, in order to cause the driving assistance device to control the speed and direction of movement of the vehicle, so as to reduce the burden on the driver in driving, when the determination result by the determination unit 140 indicates that the occupant state detection unit 130 has detected a change in the occupant from the first state to the second state in response to the detection of a defensive action by the occupant by the defensive action detection unit 120. In other words, when the determination result by the determination unit 140 indicates that a specific moving object is present in the vehicle cabin, the vehicle control unit 360 in embodiment 3 outputs a control signal to the vehicle control device 36 for the driving assistance device to control the speed and direction of movement of the vehicle so as to reduce the burden on the driver when driving.

[0068] For example, when the determination result by the determination unit 140 indicates that the occupant state detection unit 130 has detected a change of the occupant from the first state to the second state in response to the detection of a defensive action by the defensive action detection unit 120, the vehicle control unit 360 switches the autonomous driving level of the vehicle to a level higher than that before the defensive action was detected, and outputs a control signal to the vehicle control device 36 for controlling at least one of the speed and the moving direction of the vehicle. Also, for example, when the determination result by the determination unit 140 indicates that the occupant state detection unit 130 has detected a change of the occupant from the first state to the second state in response to the detection of a defensive action by the defensive action detection unit 120, the vehicle control unit 360 causes the vehicle control device 36 to perform fully autonomous driving of the vehicle, and outputs a control signal to the vehicle control device 36 for controlling the speed and the moving direction of the vehicle so as to move the vehicle toward a specific location.

[0069] The hardware configuration of the information processing device 300 according to the third embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0070] Next, details of the processing performed by the information processing device 300 will be described with reference to Figures 7 and 8. Figure 8 is a flowchart showing an example of the processing performed by the information processing device 300 according to embodiment 3. Note that part of the processing performed by the information processing device 300 according to embodiment 3 is similar to the processing performed by the information processing device 100 according to embodiment 1, and therefore, the same processes as those in embodiment 1 are denoted by the same reference numerals as in embodiment 1 and will not be described again. As shown in Figure 8, when the information processing device 300 starts processing, it first acquires occupant information (step ST01).

[0071] After performing the processing of step ST01, the information processing device 300 determines whether a defensive action has been detected (step ST02). If a defensive action has been detected in the processing of step ST02 (YES in step ST02), the information processing device 300 determines whether a change in the occupant's state from the first state to the second state has been detected within a specific period of time (step ST03). If a change in the occupant's state from the first state to the second state has been detected within a specific period of time (YES in step ST03), the information processing device 300 outputs a control signal for the vehicle (step ST07). In this processing, the information processing device 300 determines that a change in the occupant's state from the first state to the second state due to the detection of the defensive action has been detected based on the fact that a change in the occupant's state from the first state to the second state has been detected within a specific period of time that includes the time when the defensive action was detected in the processing of step ST02, and outputs a control signal to the vehicle control device 36 to control the vehicle control device 36 so as to reduce the driver's burden on driving.

[0072] If no defensive action is detected in the processing of step ST02 (NO in step ST02), if no change in the occupant's state from the first to the second state within the specific period is detected in the processing of step ST03 (NO in step ST03), or if the processing of step ST07 is performed, the information processing device 300 determines whether or not the vehicle has stopped traveling (step ST09). If the vehicle has not stopped traveling in the processing of step ST09 (NO in step ST09), the information processing device 300 returns the processing to step ST01, and if the vehicle has stopped traveling in the processing of step ST09 (YES in step ST09), the information processing device 300 ends the processing.

[0073] As described above, the information processing device 300 according to the third embodiment includes a vehicle control unit 360 that controls the vehicle based on the determination result by the determination unit 140. For example, the vehicle control unit 360 controls at least one of the speed and direction of movement of the vehicle. Furthermore, for example, the vehicle control unit 360 controls the speed and direction of movement of the vehicle so as to move the vehicle toward a specific location. With this configuration, even when a defensive action or a change in the state of the occupant indicating the presence of a specific moving object in the vehicle cabin is detected, the information processing device 300 can reduce the burden on the driver by having the vehicle control device 36 control the vehicle.

[0074] 9 and 10, an information processing system according to a fourth embodiment will be described. The information processing system according to the fourth embodiment differs from the information processing system according to the second embodiment in that it includes a vehicle device and an external device that are made up of various devices that a vehicle has, and in that the information processing device for controlling the vehicle device has a partial configuration, but the other configurations are the same. The same configurations as those in the second embodiment are assigned the same names and symbols as those in the second embodiment, and descriptions thereof will be omitted.

[0075] Fig. 9 is a block diagram showing a schematic configuration of an information processing system according to embodiment 4. As shown in Fig. 9, the information processing system according to embodiment 4 includes an occupant information acquisition device 10, a vehicle device 30, an external device 40, and an information processing device 400, which are electrically connected wirelessly or by wire so as to be able to communicate with each other.

[0076] The vehicle device 30 is disposed in the vehicle and includes a communication device 31 , an air conditioning device 32 , a window opening / closing device 33 , a light emitting device 34 , an alarm sound device 35 , and a vehicle control device 36 .

[0077] The communication device 31 communicates between the vehicle and the external device 40 based on information input from the information processing device 400. For example, the communication device 31 communicates with the external device 40, which is an external computer used by an operator who monitors the vehicle from the outside, based on information input from the information processing device 400. For example, the communication device 31 communicates one or a plurality of pieces of information selected from image information, audio information, text information, and other information between the vehicle and the external device 40.

[0078] The air conditioner 32 is a device for performing air conditioning of the vehicle based on information input from the information processing device 400. The air conditioner 32 performs ventilation inside the vehicle cabin based on information input from the information processing device 400.

[0079] The window opening / closing device 33 opens and closes the vehicle windows based on information input from the information processing device 400 .

[0080] The light-emitting device 34 changes the light emission mode of a light source disposed inside or outside the vehicle cabin based on information input from the information processing device 400. For example, the light-emitting device 34 turns on or blinks a light-emitting source disposed outside the vehicle cabin to notify the outside that the condition inside the vehicle cabin is not normal, based on information input from the information processing device 400.

[0081] The sound alarm device 35 issues a sound to the interior or exterior of the vehicle cabin based on information input from the information processing device 400. For example, the sound alarm device 35 issues a sound to the interior or exterior of the vehicle cabin based on information input from the information processing device 400 to indicate that the condition inside the vehicle cabin is not normal.

[0082] The information processing device 400 includes an occupant information acquisition unit 110 , a defensive action detection unit 120 , an occupant state detection unit 130 , a determination unit 140 , a classification unit 450 , and a vehicle control unit 460 .

[0083] The classification unit 450 classifies specific moving objects present in the vehicle cabin based on the occupant information acquired by the occupant information acquisition unit 110. The function of the classification unit 450 to classify specific moving objects is similar to that of the classification unit 250 according to the second embodiment, and therefore a description thereof will be omitted.

[0084] The vehicle control unit 460 controls the vehicle based on the classification result by the classifier 450. For example, the vehicle control unit 460 outputs a control signal to the vehicle device 30 based on the classification result by the classifier 450, so that the vehicle device 30 performs control in accordance with the classification result. For example, when the classification result by the classifier 450 indicates that a specific moving object estimated to be present in the vehicle cabin has been classified into a first class corresponding to an insect, a small animal, or the like, the vehicle control unit 460 outputs a control signal to the vehicle device 30 to open the window by the window opening / closing device 33 and to perform driving assistance by the vehicle control device 36.

[0085] Furthermore, for example, when the classification result by the classification unit 450 indicates that a specific moving object presumed to be present in the vehicle cabin has been classified into the second class corresponding to a suspicious person, animal, or the like, the vehicle control unit 460 causes the communication device 31 to transmit information informing the operator using the external device 40, an administrative organization related to public safety such as the police, a security guard, or the like of the intrusion of a suspicious person, animal, or the like into the vehicle cabin, causes the light-emitting device 34 to flash the light source to notify the outside that the condition inside the vehicle cabin is not normal, causes the alarm device 35 to start sounding an alarm both inside and outside the vehicle cabin to indicate that the condition inside the vehicle cabin is not normal, and outputs a control signal to the vehicle device 30 via the vehicle control device 36 to provide driving assistance.

[0086] Furthermore, for example, when the classification result by the classifier 450 indicates that a specific moving object estimated to be present in the vehicle cabin has been classified into the third class corresponding to odorous floating matter, etc., the vehicle control unit 460 outputs a control signal to the vehicle device 30 to cause the air conditioning unit 32 to ventilate the vehicle cabin or to cause the window opening / closing unit 33 to open the window, and to cause the vehicle control device 36 to provide driving assistance. Furthermore, for example, when the classification result by the classifier 450 indicates that a specific moving object estimated to be present in the vehicle cabin has been classified into the third class corresponding to odorous floating matter, etc., if the odor is smoke, etc., the vehicle control unit 460 causes the communication unit 31 to transmit information informing a dealer or repair shop using the external device 40 that odorous floating matter, etc. has been generated, since repair of one of the vehicle's devices may be required.

[0087] The hardware configuration of the information processing device 400 according to the fourth embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0088] Next, details of the processing performed by the information processing device 400 will be described with reference to Figures 9 and 10. Figure 10 is a flowchart showing an example of the processing performed by the information processing device 400 according to embodiment 4. Note that part of the processing performed by the information processing device 400 according to embodiment 4 is similar to the processing performed by the information processing device 100 according to embodiment 2, and therefore, the same processes as those in embodiment 1 are denoted by the same reference numerals as in embodiment 1 and will not be described again. As shown in Figure 10, when the information processing device 400 starts processing, it first acquires occupant information (step ST01).

[0089] After performing the processing of step ST01, the information processing device 400 determines whether or not a defensive action has been detected (step ST02). If a defensive action has been detected in the processing of step ST02 (YES in step ST02), the information processing device 400 determines whether or not a change in the occupant's state from a first state to a second state within a specific period has been detected (step ST03). If a change in the occupant's state from the first state to the second state within a specific period has been detected in the processing of step ST03 (YES in step ST03), the information processing device 400 classifies the moving object based on the combination of the detected defensive action and the second state (step ST05).

[0090] After completing the process of step ST05, the information processing device 400 outputs a control signal based on the classification result (step ST08). In this process, the information processing device 400 outputs a control signal to the vehicle device 30 to cause the vehicle device 30 to perform control according to the classification result of the specific moving object estimated to be present in the vehicle cabin.

[0091] If no defensive action is detected in the processing of step ST02 (NO in step ST02), if no change in the occupant's state from the first to the second state within the specific period is detected in the processing of step ST03 (NO in step ST03), or if the processing of step ST06 is performed, the information processing device 400 determines whether or not the vehicle has stopped traveling (step ST09). If the vehicle has not stopped traveling in the processing of step ST09 (NO in step ST09), the information processing device 400 returns the processing to step ST01, and if the vehicle has stopped traveling in the processing of step ST09 (YES in step ST09), the information processing device 400 ends the processing.

[0092] As described above, the information processing device 400 according to the fourth embodiment controls the vehicle communication device 31 to output a specific signal to the vehicle device 30 communicatively connected to the information processing device 400 based on the detection results by the defensive operation detection unit 120 and the occupant state detection unit 130. For example, the information processing device 400 controls the vehicle communication device 31 to transmit information to the external device 40 communicatively connected to the vehicle via the vehicle device 30, indicating that the state inside the vehicle cabin is not normal, based on the detection results by the defensive operation detection unit 120 and the occupant state detection unit 130. Specifically, the information processing device 400 controls the vehicle communication device 31 to transmit information to the external device 40 communicatively connected to the vehicle via the vehicle device 30, indicating that a suspicious person, animal, or the like has entered the vehicle cabin, based on the detection results by the defensive operation detection unit 120 and the occupant state detection unit 130. With this configuration, the information processing device 400 is able to control the vehicle according to the classification result of the specific moving object inside the vehicle cabin.

[0093] 11 and 12, an information processing system according to a fifth embodiment will be described. Similar to the information processing system according to the first embodiment, the information processing system according to the fifth embodiment is a system for acquiring occupant information including information indicating the defensive action of the occupant and information indicating a change from a first state to a second state, and storing information generated based on the occupant information in a storage device. The same components as those in the first embodiment will be assigned the same names and symbols as those in the first embodiment, and descriptions thereof will be omitted.

[0094] Fig. 11 is a block diagram showing a schematic configuration of an information processing system according to embodiment 5. As shown in Fig. 11, the information processing system according to embodiment 5 includes a vehicle V1, an information processing device 500, and a storage device 70, which are electrically connected wirelessly or by wire so as to be able to communicate with each other. For example, the information processing device 500 and the storage device 70 are configured as external devices connected to the vehicle V1 so as to be able to communicate with each other.

[0095] The vehicle V1 includes an occupant information acquisition device 10, an input device 11, and a communication device 12. The input device 11 receives an input operation from the occupant and outputs a signal corresponding to the input operation. For example, the input device 11 includes a liquid crystal display device that displays information and a touch panel that receives an input operation from the occupant. The input device 11 may include a mechanical switch that can be turned on / off, or may include an audio input device having a microphone that receives audio input. For example, the input device 11 is a device that is operated when a specific moving object is present in the cabin of the vehicle V1. When the occupant recognizes the presence of the specific moving object in the cabin of the vehicle V1, the input device 11 outputs a signal indicating the presence of the specific moving object in the cabin of the vehicle V1.

[0096] The communication device 12 communicates information between the vehicle V1 and the information processing device 500. For example, the communication device 12 transmits, to the information processing device 500, information acquired by the occupant information acquisition device 10 and a signal corresponding to an input operation by the occupant from the input device 11. For example, the communication device 12 transmits, to the information processing device 500, a signal corresponding to an input operation by the occupant from the input device 11 and information acquired by the occupant information acquisition device 10 during a specific period including the time when the input operation was performed. Specifically, the communication device 12 transmits, to the information processing device 500, a signal corresponding to an input operation by the occupant from the input device 11 and information acquired by the occupant information acquisition device 10 during a 30-second period starting from the time when the input operation was performed.

[0097] The communication device 12 may be configured to transmit, to the information processing device 500, information indicating the time at which a signal corresponding to an input operation by the occupant from the input device 11 was acquired, and all of the information acquired by the occupant information acquisition device 10. For example, if the signal from the input device 11 is a signal indicating the presence of a specific moving object in the passenger compartment of the vehicle V1, the information acquired by the occupant information acquisition device 10 during a specific period including the time at which the input operation related to the signal was performed can be said to include information indicating a defensive action taken by the occupant to protect themselves from the specific moving object, and information indicating a change in the occupant's state from a first state to a second state accompanying the defensive action.

[0098] The storage device 70 stores information input from the information processing device 500. For example, the storage device 70 stores information input from the information processing device 500, such as information used by the information processing device 500 in various processes, results of various processes performed by the information processing device 500, and information acquired by the information processing device 500 from the vehicle V1.

[0099] The information processing device 500 includes an occupant information acquiring unit 110 , a signal acquiring unit 570 , and a motion information generating unit 580 .

[0100] The signal acquisition unit 570 acquires a signal from the input device 11 via the communication device 12. For example, the signal acquisition unit 570 acquires a signal from the input device 11 indicating the presence of a specific moving object in the cabin of the vehicle V1.

[0101] The motion information generating unit 580 generates motion information including information indicating a defensive motion of the occupant to protect themselves from a specific moving object and information indicating a change in the occupant's state from a first state to a second state accompanying the defensive motion, based on the occupant information acquired by the occupant information acquiring unit 110 during a specific period including the time when the signal acquiring unit 570 acquired the signal. For example, the motion information generating unit 580 generates motion information including image information of the occupant acquired by the occupant information acquiring unit 110. Furthermore, for example, the motion information generating unit 580 generates motion information consisting of the image information of the occupant acquired by the occupant information acquiring unit 110. Furthermore, for example, the motion information generating unit 580 extracts feature amounts indicating the positions of specific body parts of the occupant from the occupant information acquired by the occupant information acquiring unit 110, and generates motion information indicating the motion of the occupant. Furthermore, for example, the motion information generating unit 580 generates motion information by extracting some information from the occupant information acquired by the occupant information acquiring unit 110.

[0102] The motion information generating unit 580 generates a plurality of pieces of motion information. For example, the motion information generating unit 580 generates the plurality of pieces of motion information based on occupant information related to a plurality of occupants aboard the vehicle V1. Furthermore, for example, the motion information generating unit 580 generates the plurality of pieces of motion information based on a plurality of pieces of occupant information acquired from a plurality of vehicles including the vehicle V1.

[0103] The hardware configuration of the information processing device 500 according to the fifth embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0104] Next, details of the processing performed by the information processing device 500 will be described with reference to Figures 11 and 12. Figure 12 is a flowchart showing an example of the processing performed by the information processing device 500 according to embodiment 5. Note that part of the processing performed by the information processing device 500 according to embodiment 5 is similar to the processing performed by the information processing device 100 according to embodiment 1, and therefore, the same processes as those in embodiment 1 are denoted by the same reference numerals as in embodiment 1 and description thereof will be omitted. As shown in Figure 12, when the information processing device 500 starts processing, it first acquires occupant information (step ST01).

[0105] After performing the process of step ST01, the information processing device 500 determines whether or not an input signal due to an occupant's operation has been acquired (step ST11). If an input signal due to an occupant's operation has been acquired in the process of step ST11 (YES in step ST11), the information processing device 500 generates operation information (step ST14). After performing the process of step ST14, the information processing device 500 stores the operation information (step ST15). In this process, the information processing device 500 outputs the operation information generated in the process of step ST14 to the storage device 70 and stores it in the storage device 70.

[0106] If an input signal due to an occupant's operation is not acquired in the processing of step ST11 (NO in step ST11), and if the processing of step ST15 is performed, the information processing device 500 determines whether or not to end the acquisition of information (step ST16). For example, in this processing, the information processing device 500 acquires information indicating whether or not the vehicle V1 has finished traveling from the vehicle V1, and determines whether or not to end the acquisition of information based on whether or not the vehicle V1 has finished traveling. Also, for example, in this processing, the information processing device 500 determines whether or not to end the acquisition of information based on whether or not the number of pieces of operation information acquired from the vehicle V1 has reached a preset acquisition target number.

[0107] If the information acquisition is not completed in the processing of step ST16 (NO in step ST16), the information processing device 500 returns the processing to step ST01, and if the information acquisition is completed in the processing of step ST16 (YES in step ST16), the information processing device 500 terminates the processing.

[0108] As described above, the information processing device 500 according to the fifth embodiment includes an occupant information acquisition unit 110 that acquires occupant information including image information of an occupant of the vehicle V1, a signal acquisition unit 570 that acquires a signal indicating the presence of a specific moving object in the cabin of the vehicle V1, and an action information generation unit 580 that generates action information including information indicating a defensive action taken by the occupant to protect themselves from the specific moving object and information indicating a change in the state of the occupant from a first state to a second state accompanying the defensive action, based on the occupant information acquired by the occupant information acquisition unit 110 during a specific period including the time at which the signal was acquired by the signal acquisition unit 570. For example, the signal acquisition unit 570 is configured to acquire a signal indicating the presence of a specific moving object in the cabin from an input device 11 that is arranged in the cabin and accepts input operations by the vehicle occupant.

[0109] With this configuration, the information processing device 500 can collect action information, which includes information indicating a defensive action taken by an occupant to protect themselves from a specific moving object and information indicating a change in the occupant's state from a first state to a second state accompanying the defensive action, based on the occupant information acquired by the occupant information acquisition device 10 of the vehicle V1. Furthermore, by performing image analysis or machine learning based on the multiple pieces of action information collected in this manner, it becomes possible to improve the reliability of detecting defensive actions taken by an occupant to protect themselves from a specific moving object moving within the vehicle cabin compared to conventional methods.

[0110] Sixth Embodiment Next, an information processing system according to a sixth embodiment will be described with reference to Fig. 13 and Fig. 14. The information processing system according to the sixth embodiment differs from the information processing system according to the fifth embodiment in the configuration related to the function of inputting a signal from an input device to an information processing device, but the other configurations are the same, and the same names and symbols as those in the fifth embodiment will be used and the description thereof will be omitted.

[0111] Fig. 13 is a block diagram showing a schematic configuration of an information processing system according to embodiment 6. As shown in Fig. 13, the information processing system according to embodiment 6 includes a vehicle V2, an information processing device 600, a display device 50, an input device 60, and a storage device 70, which are electrically connected wirelessly or by wire so as to be able to communicate with each other. For example, the information processing device 600, the display device 50, the input device 60, and the storage device 70 are configured as external devices connected to the vehicle V2 so as to be able to communicate with each other.

[0112] The vehicle V2 is equipped with an occupant information acquisition device 10 and a communication device 12. The communication device 12 communicates information between the vehicle V2 and the information processing device 600. For example, the communication device 12 transmits information acquired by the occupant information acquisition device 10 to the information processing device 600.

[0113] The display device 50 displays information input from the information processing device 600. For example, the display device 50 is configured with a liquid crystal display device used by an operator who monitors the vehicle from the outside, and displays to the operator information input from the information processing device 600. For example, the display device 50 displays information indicating the status of the interior of the vehicle V2, specifically, information indicating whether a specific moving object is present in the interior of the vehicle V2, as information input from the information processing device 600.

[0114] The input device 60 accepts an input operation by an operator and outputs a signal corresponding to the input operation. For example, the input device 60 accepts an input operation by an operator monitoring the vehicle from the outside and outputs a signal corresponding to the input operation. For example, the input device 60 is configured with a keyboard, a mouse, a touch panel, a mechanical switch, a voice input device, etc. that accept input operations by the operator, and accepts an input operation when the operator monitoring the vehicle from the outside determines that a specific moving object is present in the cabin of the vehicle V2 based on the information displayed on the display device 50, and outputs a signal indicating the presence of the specific moving object in the cabin of the vehicle V2 to the information processing device 600.

[0115] The information processing device 600 includes an occupant information acquiring unit 610 , a signal acquiring unit 670 , and a motion information generating unit 580 .

[0116] The occupant information acquisition unit 610 acquires occupant information including image information of the vehicle occupants based on information from the occupant information acquisition device 10. The function of the occupant information acquisition unit 610 to acquire occupant information is similar to that of the occupant information acquisition unit 110, and therefore a description thereof will be omitted. The occupant information acquisition unit 610 outputs the acquired occupant information to the display device 50.

[0117] The signal acquisition unit 670 acquires a signal from the input device 60. For example, the signal acquisition unit 670 acquires a signal from the input device 60 indicating that a specific moving object is present in the cabin of the vehicle V2.

[0118] The motion information generating unit 580 according to the sixth embodiment generates motion information including information indicating a defensive action taken by the occupant to protect themselves from a specific moving object and information indicating a change in the state of the occupant from a first state to a second state accompanying the defensive action, based on the occupant information acquired by the occupant information acquiring unit 610 during a specific period including the time at which the signal was acquired by the signal acquiring unit 670. The detailed function of the motion information generating unit 580 according to the sixth embodiment is the same as that of the motion information generating unit 580 according to the fifth embodiment, and therefore will not be described here.

[0119] The hardware configuration of the information processing device 600 according to the sixth embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0120] Next, details of the processing performed by the information processing device 600 will be described with reference to Figures 13 and 14. Figure 14 is a flowchart showing an example of the processing performed by the information processing device 600 according to embodiment 6. Note that part of the processing performed by the information processing device 600 according to embodiment 6 is similar to the processing performed by the information processing device 500 according to embodiment 5, and therefore, the same processes as those in embodiment 5 are denoted by the same reference numerals as those in embodiment 5 and description thereof will be omitted. As shown in Figure 14, when the information processing device 600 starts processing, it first acquires occupant information (step ST01).

[0121] After completing the process of step ST01, the information processing device 600 displays the occupant information on the display device 50 (step ST12). In this process, the information processing device 600 outputs the occupant information acquired by the occupant information acquisition unit 610 to the display device 50, and displays the occupant information to the operator who operates the display device 50.

[0122] After the processing of step ST12, the information processing device 600 determines whether or not an input signal due to an operator's operation has been acquired (step ST13). If an input signal due to an operator's operation has been acquired in the processing of step ST13 (YES in step ST13), the information processing device 600 generates motion information (step ST14). After the processing of step ST14, the information processing device 600 stores the motion information (step ST15). In this processing, the information processing device 600 outputs the motion information generated in the processing of step ST14 to the storage device 70 and stores it in the storage device 70.

[0123] If an input signal by an operator's operation is not acquired in the process of step ST13 (NO in step ST13), and if the process of step ST15 is performed, the information processing device 600 determines whether or not to end the acquisition of information (step ST16). If the acquisition of information is not completed in the process of step ST16 (NO in step ST16), the information processing device 600 returns the process to step ST01, and if the acquisition of information is completed in the process of step ST16 (YES in step ST16), the information processing device 600 ends the process.

[0124] As described above, the information processing device 600 according to the sixth embodiment includes an occupant information acquisition unit 610 that acquires occupant information including image information of an occupant of the vehicle V2, a signal acquisition unit 670 that acquires a signal indicating the presence of a specific moving object in the cabin of the vehicle V2, and an action information generation unit 580 that generates action information including information indicating a defensive action taken by the occupant to protect themselves from the specific moving object and information indicating a change in the state of the occupant from a first state to a second state accompanying the defensive action, based on the occupant information acquired by the occupant information acquisition unit 610 during a specific period including the time at which the signal was acquired by the signal acquisition unit 670. For example, the signal acquisition unit 670 is configured to acquire a signal indicating the presence of a specific moving object in the cabin from a device external to the vehicle V2 that is communicatively connected to the vehicle V2.

[0125] With this configuration, the information processing device 600 can collect, based on the occupant information acquired by the occupant information acquisition device 10 of the vehicle V2, behavior information including information indicating the occupant's defensive behavior to protect themselves from a specific moving object and information indicating a change in the occupant's state from a first state to a second state accompanying the defensive behavior. Furthermore, by performing image analysis or machine learning based on the collected behavior information, it is possible to improve the reliability of detecting the occupant's defensive behavior to protect themselves from a specific moving object moving within the vehicle cabin compared to conventional methods. Furthermore, in the sixth embodiment, the operator displays the occupant information transmitted from the vehicle V2 on a display device in real time and determines the presence or absence of a specific moving object within the vehicle cabin through the operator's operation. However, the operator's operation does not need to be real-time. The occupant information may be temporarily stored in a storage device and displayed offline on a display device for the operator to make a judgment. With this configuration, the operator can make a judgment at any time, allowing the judgment to be performed at any time. Furthermore, the judgment process can be made more efficient, for example, by allowing multiple operators to share the work. Furthermore, in combination with the seventh embodiment described below, the determination unit of the seventh embodiment may make a preliminary determination of the presence or absence of a specific moving object in the vehicle cabin, and present the occupant information at the time when the presence of the moving object is determined to be present to the operator, so that the operator can make a final determination of the presence or absence of the specific moving object. Since the operator does not need to check all of the occupant information, the amount of work required by the operator is significantly reduced. In this case, a determination threshold or determination logic that makes it easier to determine the presence of a moving object than the logic used by the determination unit of the seventh embodiment to determine the presence or absence of a specific moving object in the vehicle cabin may be used. Conversely, the final determination may be made using the determination logic of the seventh embodiment based on the result of the operator's determination.

[0126] Seventh Embodiment Next, an information processing system according to a seventh embodiment will be described with reference to Fig. 15 and Fig. 16. The information processing system according to the seventh embodiment differs from the information processing system according to the sixth embodiment in some configurations related to the generation of operation information by the information processing device, but other configurations are the same, and the same names and symbols as those in the sixth embodiment will be used and descriptions thereof will be omitted.

[0127] Fig. 15 is a block diagram showing a schematic configuration of an information processing system according to embodiment 7. As shown in Fig. 15, the information processing system according to embodiment 7 includes a vehicle V2, an information processing device 700, and a storage device 70, which are electrically connected wirelessly or by wire so as to be able to communicate with each other. For example, the information processing device 700 and the storage device 70 are configured as external devices connected to the vehicle V2 so as to be able to communicate with each other.

[0128] The information processing device 700 includes an occupant information acquisition unit 110, a defensive movement detection unit 120, an occupant state detection unit 130, a determination unit 140, and a movement information generation unit 780. Note that the functions of the occupant information acquisition unit 110, the defensive movement detection unit 120, the occupant state detection unit 130, and the determination unit 140 are similar to those of the information processing device 100 according to the first embodiment, and therefore description thereof will be omitted.

[0129] The movement information generating unit 780 generates movement information including information indicating a defensive movement made by the occupant to protect themselves from a specific moving object and information indicating a change in the occupant's state from a first state to a second state due to the defensive movement, based on the occupant information acquired by the occupant information acquiring unit 110 during a specific period including the time when the defensive movement of the occupant was detected by the defensive movement detecting unit 120. The defensive movement detecting unit 120 acquires a signal indicating the presence of a specific moving object within the vehicle cabin by detecting the defensive movement of the occupant from a specific moving object moving within the vehicle cabin, based on the image information acquired by the occupant information acquiring unit 110.

[0130] For example, the motion information generating unit 780 extracts information from the occupant information acquired by the occupant information acquiring unit 110 about a period in which there is a high probability that the defensive motion of the occupant detected by the defensive motion detecting unit 120 is included, and generates motion information based on the extracted information.

[0131] The hardware configuration of the information processing device 700 according to the seventh embodiment is similar to the hardware configuration of the information processing device 100 according to the first embodiment, and therefore a description thereof will be omitted.

[0132] Next, details of the processing performed by the information processing device 700 will be described with reference to Figures 15 and 16. Figure 16 is a flowchart showing an example of processing performed by the information processing device 700 according to embodiment 6. Note that part of the processing performed by the information processing device 700 according to embodiment 6 is similar to the processing performed by the information processing device 500 according to embodiment 5, and therefore, the same processes as those in embodiment 5 are denoted by the same reference numerals as those in embodiment 5 and description thereof will be omitted. As shown in Figure 16, when the information processing device 700 starts processing, it first acquires occupant information (step ST01).

[0133] After performing the processing of step ST01, the information processing device 700 determines whether or not a defensive action has been detected (step ST02). If a defensive action has been detected in the processing of step ST02 (YES in step ST02), the information processing device 700 determines whether or not a change in the occupant's state from the first to the second state within the specific period has been detected (step ST03). If a change in the occupant's state from the first to the second state within the specific period has been detected in the processing of step ST03 (YES in step ST03), the information processing device 700 generates action information (step ST14). After performing the processing of step ST14, the information processing device 700 stores the action information (step ST15).

[0134] If no defensive action is detected in the processing of step ST02 (NO in step ST02), if no change of the occupant from the first state to the second state within the specific period is detected in the processing of step ST03 (NO in step ST03), or if the processing of step ST15 is performed, the information processing device 700 determines whether or not to end the acquisition of information (step ST16). If the acquisition of information is not ended in the processing of step ST16 (NO in step ST16), the information processing device 700 returns the processing to step ST01, and if the acquisition of information is ended in the processing of step ST16 (YES in step ST16), the information processing device 700 ends the processing.

[0135] As described above, the information processing device 700 according to the seventh embodiment includes an occupant information acquisition unit 110 that acquires occupant information including image information of the occupants of the vehicle V2, a defensive action detection unit 120 that acquires a signal indicating the presence of a specific moving object in the passenger compartment of the vehicle V2, and an action information generation unit 780 that generates action information including information indicating the defensive action of the occupant to protect themselves from the specific moving object and information indicating a change in the state of the occupant from a first state to a second state accompanying the defensive action, based on the occupant information acquired by the occupant information acquisition unit 110 during a specific period including the time when the signal was acquired by the defensive action detection unit 120.

[0136] With this configuration, the information processing device 700 can collect action information, which includes information indicating a defensive action taken by an occupant to protect themselves from a specific moving object and information indicating a change in the occupant's state from a first state to a second state accompanying the defensive action, based on the occupant information acquired by the occupant information acquisition device 10 of the vehicle V2. Furthermore, by performing image analysis or machine learning based on the multiple pieces of action information collected in this manner, it is possible to improve the reliability of detecting defensive actions taken by an occupant to protect themselves from a specific moving object moving within the vehicle cabin compared to conventional methods.

[0137] Eighth Embodiment Next, an information processing system according to an eighth embodiment will be described with reference to Fig. 17 and Fig. 18. The information processing system according to the eighth embodiment differs from the information processing system according to the fifth embodiment in that the information processing device includes a learning unit for performing machine learning, but other configurations are the same. The same names and symbols as those in the fifth embodiment will be used and descriptions thereof will be omitted.

[0138] 17 is a block diagram showing a schematic configuration of an information processing system according to embodiment 8. As shown in FIG. 17 , the information processing system according to embodiment 8 includes a vehicle V1, an information processing device 800, and a storage device 70, which are electrically connected wirelessly or by wire so as to be able to communicate with each other. For example, the information processing device 800 and the storage device 70 are configured as external devices connected to the vehicle V1 so as to be able to communicate with each other.

[0139] The information processing device 500 includes an occupant information acquiring unit 110 , a signal acquiring unit 570 , a motion information generating unit 580 , and a learning unit 890 .

[0140] For example, the learning unit 890 learns the plurality of pieces of motion information generated by the motion information generating unit 580, and generates a trained model that infers the presence or absence of a specific moving object in the vehicle cabin based on input of occupant information acquired by the occupant information acquiring unit 110. For example, the learning unit 890 learns the plurality of pieces of motion information generated by the motion information generating unit 580 using a CNN (Convolutional Neural Network).

[0141] Furthermore, for example, the learning unit 890 learns the plurality of pieces of motion information generated by the motion information generating unit 580, and generates a trained model that infers whether or not the occupant related to the occupant information has performed a defensive action and whether or not the state of the occupant has changed from a first state to a second state, based on input of the occupant information acquired by the occupant information acquiring unit 110. For example, the learning unit 890 learns the plurality of pieces of motion information generated by the motion information generating unit 580 using a CNN.

[0142] Next, details of the processing performed by the information processing device 800 will be described with reference to Fig. 17 and Fig. 18. Fig. 18 is a flowchart showing an example of processing for performing machine learning performed by the information processing device 800 according to embodiment 8. Note that the processing for generating motion information performed by the information processing device 800 according to embodiment 8 is similar to the processing performed by the information processing device 500 according to embodiment 5, and therefore description thereof will be omitted. As shown in Fig. 18, when the information processing device 800 starts processing, it first acquires motion information (step ST21).

[0143] After performing the process of step ST21, the information processing device 800 generates a trained model (step ST22). In this process, the information processing device 800 learns the motion information acquired in step ST21 and generates the trained model.

[0144] After performing the processing of step ST22, the information processing device 800 determines whether to end learning (step ST23). For example, in this processing, the information processing device 800 determines whether to end generation of the trained model based on the fact that all of the motion information generated by the motion information generating unit 580 has been learned. Also, for example, in this processing, the information processing device 800 determines whether to end generation of the trained model based on the fact that the number of pieces of motion information that have been trained has reached a preset target number.

[0145] In the process of step ST23, if the learning is not to be ended (NO in step ST23), the information processing device 800 returns the process to step ST21. In the process of step ST23, if the learning is to be ended (YES in step ST23), the information processing device 800 stores the trained model (step ST24). In this process, the information processing device 800 outputs the generated trained model to the storage device 70 and stores it in the storage device 70. After performing the process of step ST24, the information processing device 800 ends the process.

[0146] As described above, the information processing device 800 according to the eighth embodiment includes, for example, a learning unit 890 that learns a plurality of pieces of motion information generated by the motion information generating unit 580 and generates a trained model that infers the presence or absence of a specific moving object in the vehicle cabin based on input of occupant information acquired by the occupant information acquiring unit 110. With this configuration, the information processing device 800 infers the presence or absence of a specific moving object in the vehicle cabin based on the defensive actions of the occupants and changes in the state accompanying the defensive actions of the occupants, thereby improving the reliability of the defensive actions of the occupants compared to conventional methods and the accuracy of inferring the presence or absence of a specific moving object in the vehicle cabin.

[0147] The information processing device 800 according to the eighth embodiment also includes a learning unit 890 that learns, for example, a plurality of pieces of motion information generated by the motion information generating unit 580 and generates a trained model that infers, based on input occupant information acquired by the occupant information acquiring unit 110, whether or not an occupant has performed a defensive action and whether or not the occupant's state has changed from a first state to a second state. With this configuration, the information processing device 800 can improve the reliability of the occupant's defensive action compared to conventional methods when inferring whether or not an occupant has performed a defensive action and whether or not the occupant's state has changed from a first state to a second state based on the occupant's defensive action and the state change associated with the occupant's defensive action. The trained model thus generated can be used, for example, by the defensive action detection unit 120 and the occupant state detection unit 130 according to the first embodiment to detect defensive actions and changes in the occupant's state.

[0148] In any of the above-described embodiments 5 to 8, the information processing device may include a classification unit that classifies the specific moving object, the signal acquisition unit acquires a signal corresponding to the type of the specific moving object present in the vehicle cabin as a signal indicating the presence of the specific moving object in the vehicle cabin, the classification unit classifies the specific moving object based on the signal acquired by the signal acquisition unit, and the motion information generation unit generates motion information that correlates information indicating a defensive action taken by an occupant to protect themselves from the specific moving object, information indicating a change in the occupant's state from a first state to a second state accompanying the defensive action, and information indicating a classification result by the classification unit. Details of such a classification unit are similar to those of the classification unit 250 in the second embodiment. Furthermore, since the information processing device according to the eighth embodiment includes such a classification unit, the learning unit may be configured to learn the multiple pieces of motion information generated by the motion information generation unit and generate a trained model that infers a classification result of the specific moving object in the vehicle cabin based on input of the occupant information acquired by the occupant information acquisition unit. For example, such a trained model outputs, for each class, a probability that the specific moving object in the vehicle cabin is classified into each of a plurality of preset classes.

[0149] In any of the above-described embodiments, the information processing device may include some or all of the other components of the information processing system, some of the components of the information processing device may be provided in another device communicatively connected to the information processing device, or the information processing device may be communicatively connected to other components of the information processing system via another device, computer, or communication network (not shown). Furthermore, the components of the information processing system may be communicatively connected to each other via another device, computer, or communication network (not shown). For example, some or all of the components of the information processing system may be provided in the vehicle, or some may be provided in an external server communicatively connected to the vehicle.

[0150] In addition, the present disclosure allows for free combination of the respective embodiments, modification of any of the components of the respective embodiments, or omission of any of the components of the respective embodiments.

[0151] An information processing device according to the present disclosure can be used to detect a specific moving object moving inside the cabin of a vehicle and to detect a defensive action taken by an occupant to protect themselves from the moving object.

[0152] 10 Occupant information acquisition device, 11 Input device, 12 Communication device, 20 External device, 30 Vehicle device, 31 Communication device, 32 Air conditioning device, 33 Window opening / closing device, 34 Light emitting device, 35 Alarm sound device, 36 Vehicle control device, 40 External device, 50 Display device, 60 Input device, 70 Storage device, 100 Information processing device, 100a Processor, 100b Memory, 100c I / O port, 100d Processing circuit, 110 Occupant information acquisition unit, 120 Defensive action detection unit (signal acquisition unit), 130 Occupant state detection unit, 140 Determination unit, 200 Information processing device, 250 Classification unit, 300 Information processing device, 360 Vehicle control unit, 400 Information processing device, 450 Classification unit, 460 Vehicle control unit, 500 Information processing device, 570 Signal acquisition unit, 580 Motion information generation unit, 600 information processing device, 610 occupant information acquisition unit, 670 signal acquisition unit, 700 information processing device, 780 motion information generation unit, 800 information processing device, 890 learning unit, V1 vehicle, V2 vehicle.

Claims

1. An information processing device comprising: an occupant information acquisition unit that acquires occupant information including image information of vehicle occupants; a defensive action detection unit that detects defensive actions taken by the occupant to protect themselves from a specific moving object moving within the passenger compartment of the vehicle based on the image information acquired by the occupant information acquisition unit; an occupant state detection unit that detects a change in the state of the occupant from a first state to a second state based on the occupant information acquired by the occupant information acquisition unit; and a judgment unit that judges whether or not a change in the occupant from the first state to the second state has been detected by the occupant state detection unit in response to the detection of the defensive action by the defensive action detection unit.

2. The information processing device described in claim 1, characterized in that the defensive movement detection unit detects, as the defensive movement, a specific movement of at least one of the occupant's hands, arms, head, and face, and the occupant state detection unit detects, as a change from the first state to the second state, a change in the occupant's line of sight, facial direction, facial expression, posture, mouth shape, and voice state, based on image information acquired by the occupant information acquisition unit.

3. The information processing device according to claim 2, characterized in that the occupant state detection unit detects that the state of the occupant has changed from the first state to the second state based on image information acquired by the occupant information acquisition unit.

4. An information processing device as described in claim 1, characterized in that it comprises a classification unit that classifies the specific moving object, wherein the defensive action detection unit is capable of detecting that the occupant has performed any one of a plurality of preset defensive actions including the defensive action, and the occupant state detection unit is capable of detecting that the occupant's state has changed from the first state to any one of a plurality of preset second states including the second state, and the classification unit classifies the specific moving object based on a combination of any one of the defensive actions detected by the defensive action detection unit and the change to any one of the second states detected by the occupant state detection unit.

5. An information processing device according to claim 1 or 4, further comprising a vehicle control unit that controls the vehicle based on the determination result by the determination unit.

6. The information processing device according to claim 5, wherein the vehicle control unit controls at least one of the speed and direction of movement of the vehicle.

7. The information processing device according to claim 5, wherein the vehicle control unit controls the speed and direction of movement of the vehicle so as to move the vehicle toward a specific position.

8. The information processing device according to claim 4, wherein the classification unit classifies the specific moving object into a first class when the defensive action detection unit detects, as one of the defensive actions, either one of an action of the occupant's hand moving near the occupant's head or an action of the occupant's hand hitting a surrounding object, and the occupant state detection unit detects, as the change from the first state to one of the second states, either a change in the occupant's gaze direction from a state where it is not directed toward the occupant's moving hand to a state where it is directed toward the occupant's moving hand, a change in the occupant's facial direction from a state where it is not directed toward the occupant's moving hand to a state where it is directed toward the occupant's moving hand, a change in the amount of change per unit time of the occupant's gaze direction from a state where it is small to a state where it is large, a change in the occupant's facial direction from a state where it is small to a state where it is large, a change in the occupant's gaze direction from a state where it is within a specific range to a state where it is outside the specific range, a change from a state indicating a low level of discomfort to a state indicating a high level of discomfort, or a change in the occupant's voice from a state where it is quiet to a state where it is loud.

9. The information processing device of claim 4, wherein the classification unit classifies the specific moving object into a second class when the defensive action detection unit detects one of the following defensive actions: an action of the occupant's hand chasing the specific moving object, an action of the occupant's hand moving from the occupant's upper body toward the specific moving object, an action of the occupant's arm moving between the specific moving object and the occupant, and an action of the occupant's hand moving to cover the occupant's head; and the occupant state detection unit detects one of the following changes from the first state to any of the second states: a change from a state in which the occupant's upper body is not tilted to a state in which it is tilted toward the specific moving object, a change from a state in which the occupant's eyebrow wrinkles are small to a state in which they are large, a change from a state in which the occupant's eyelids are small to a state in which they are large, and a change from a state in which the occupant's voice is small to a state in which it is loud.

10. The information processing device of claim 4, wherein the classification unit classifies the specific moving object into a third class when the defensive action detection unit detects either one of the defensive actions, namely, the occupant's hand moving back and forth near the occupant's head and in front of the occupant's face, or the occupant's hand pinching the occupant's nose, and when the occupant state detection unit detects either one of the changes from the first state to any of the second states, from a state indicating that the occupant's discomfort is low to a state indicating that the occupant's discomfort is high.

11. The information processing device according to claim 5, characterized in that the vehicle control unit controls the communication device of the vehicle so as to output a specific signal to an external device communicatively connected to the vehicle based on the detection results of the defensive action detection unit and the detection results of the occupant state detection unit.

12. An information processing method performed by a device having an occupant information acquisition unit, a defensive action detection unit, an occupant state detection unit, and a judgment unit, comprising: a step in which the occupant information acquisition unit acquires occupant information including image information of the occupant of the vehicle; a step in which the defensive action detection unit detects, based on the image information acquired by the occupant information acquisition unit, a defensive action taken by the occupant to protect himself or herself from a specific moving object moving within the passenger compartment of the vehicle; a step in which the occupant state detection unit detects, based on the occupant information acquired by the occupant information acquisition unit, a change in the state of the occupant from a first state to a second state; and a step in which the judgment unit judges whether or not the occupant state detection unit has detected a change in the occupant from the first state to the second state in response to the detection of the defensive action by the defensive action detection unit.

13. An information processing device comprising: an occupant information acquisition unit that acquires occupant information including image information of vehicle occupants; a signal acquisition unit that acquires a signal indicating the presence of a specific moving object in the passenger compartment of the vehicle; and an action information generation unit that generates action information including information indicating a defensive action taken by the occupant to protect themselves from the specific moving object and information indicating a change in the occupant's state from a first state to a second state accompanying the defensive action, based on the occupant information acquired by the occupant information acquisition unit during a specific period including the time when the signal was acquired by the signal acquisition unit.

14. An information processing device as described in claim 13, characterized in that it comprises a classification unit that classifies the specific moving object, wherein the signal acquisition unit acquires a signal according to the type of the specific moving object present in the vehicle cabin as a signal indicating the presence of the specific moving object in the vehicle cabin, the classification unit classifies the specific moving object based on the signal acquired by the signal acquisition unit, and the action information generation unit generates action information that correlates information indicating a defensive action taken by the occupant to protect themselves from the specific moving object, information indicating a change in the state of the occupant from a first state to a second state accompanying the defensive action, and information indicating the classification result by the classification unit.

15. The information processing device according to claim 13, characterized in that the signal acquisition unit acquires a signal indicating the presence of the specific moving object within the vehicle cabin from an input device arranged within the vehicle cabin and accepting input operations by an occupant of the vehicle, or a signal indicating the presence of the specific moving object within the vehicle cabin from a device external to the vehicle that is communicatively connected to the vehicle.

16. The information processing device according to claim 13, characterized in that the signal acquisition unit acquires a signal indicating the presence of a specific moving object within the vehicle cabin by detecting a defensive action taken by the occupant to protect themselves from a specific moving object moving within the vehicle cabin based on image information acquired by the occupant information acquisition unit.

17. An information processing apparatus according to claim 13, wherein said motion information generating section generates motion information made up of image information of said occupant.

18. The information processing device described in claim 13, characterized in that the movement information generation unit generates movement information including information indicating a specific movement of at least one of the occupant's hands, arms, head, and face as information indicating the occupant's defensive movement to protect himself or herself from the specific moving object, and information indicating a change in the occupant's state from a first state to a second state accompanying the defensive movement, including information indicating a change in the occupant's line of sight, facial direction, facial expression, posture, mouth shape, and vocal content.

19. An information processing device as described in claim 13, characterized in that it comprises a learning unit that learns multiple pieces of motion information including the motion information generated by the motion information generation unit, and generates a learned model that infers the presence or absence of the specific moving object in the vehicle cabin based on input of occupant information acquired by the occupant information acquisition unit.

20. An information processing device as described in claim 13, characterized in that it is provided with a learning unit that learns multiple pieces of motion information including the motion information generated by the motion information generation unit, and generates a trained model that infers whether or not the defensive motion has been performed and whether or not the occupant's state has changed from the first state to the second state based on input of occupant information acquired by the occupant information acquisition unit.

Images