Vehicle occupant monitoring device and traffic system

Abstract

The present invention relates to a vehicle occupant monitoring device and a traffic system, and aims to appropriately execute multiple determination functions in the vehicle occupant monitoring device according to the situation. The occupant monitoring device (40) of the automobile (1) has: a camera device that captures the occupants of the automobile (1); and an occupant monitoring ECU (58) that can execute multiple determination processes including at least action detection processing for the driver riding in the automobile based on the captured image. When the occupant monitoring ECU (58) predicts or detects a collision of the automobile (1), it stops the determination processes other than the action detection process and executes the action detection process.

Description

Technical Field

[0001] The invention relates to an occupant monitoring device for a vehicle and a traffic system. Background Art

[0002] Sometimes, an occupant monitoring device is provided in a vehicle.

[0003] The occupant monitoring device has a pre-programmed function of monitoring an occupant riding in a vehicle (Patent Document 1). The occupant monitoring device reflects information obtained by monitoring to the control of an occupant protection device.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 11-43009

[0007] Patent Document 2: Japanese Patent Application Publication No. 2016-038793 Summary of the invention

[0008] Problems to be solved by the invention

[0009] However, when the occupant monitoring device of a vehicle is put into practical use, it is also required to determine the posture of the driver, etc. as in Patent Document 2. The occupant monitoring device is required to have a variety of functions.

[0010] On the other hand, it is expected that in the future, the occupant monitoring device will not only be expected to have these determination functions for normal driving, but also to detect the actions of the occupants during a collision, etc. In particular, it is expected that not only the driver but also other occupants other than the driver will be expected to be able to detect the actions during a collision.

[0011] In order to execute these various detection functions and determination functions, the ECU as a control unit for the occupant monitoring device is required to have high processing capabilities.

[0012] Furthermore, if the ECU as a control unit for the occupant monitoring device merely has a high processing capability, there is a possibility that sufficient performance for occupant protection may not be obtained.

[0013] As described above, the occupant monitoring device of a vehicle is required to appropriately perform a plurality of determination functions according to the situation.

[0014] Means of solving problems

[0015] The vehicle occupant monitoring device of the present invention comprises: a camera device, which captures images of occupants riding in the vehicle; and a control unit, which is capable of executing a plurality of determination processes including a motion detection process for at least the driver riding in the vehicle based on the captured images, wherein the control unit stops determination processes other than the motion detection process and executes the motion detection process when a collision of the vehicle is predicted or detected.

[0016] Preferably, there may be a switching unit which switches the viewing angle of the camera device between a state of photographing the driver and a state of photographing a vehicle including the driver and other passengers, wherein when the control unit predicts or detects a collision of the vehicle, the control unit switches from the state of photographing the driver to the state of photographing a vehicle including the driver and other passengers through the switching unit, and performs action detection processing of the driver and other passengers.

[0017] Preferably, when a collision of the vehicle is predicted, the control unit can switch from the state of photographing the driver to the state of photographing both the driver and other passengers through the switching unit, execute action detection processing of the driver and other passengers, and stop determination processing other than the action detection processing.

[0018] Preferably, there may be a memory capable of recording images captured by the camera device, wherein, when the control unit detects a collision of the vehicle, it stops the determination process including the action detection process and records the images captured by the camera device in the memory.

[0019] Preferably, the control unit may start a determination process other than the action detection process when no collision is detected after the collision of the vehicle is predicted.

[0020] Preferably, as multiple determination processes, the control unit is capable of executing determination processes of multiple safety functions for occupant safety including the action detection process and determination processes of convenience functions for occupant convenience. When no collision is detected after a collision of the vehicle is predicted, the determination process of the safety function can be restarted with priority over the determination process of the convenience function.

[0021] Preferably, the control unit may include the action detection processing, driver's squint determination, drowsiness determination, sleepiness determination, and deceased person determination in the determination processing of the safety function; and include posture determination, emotion determination, and personal determination in the determination processing of the convenience function.

[0022] Preferably, the control unit may start a determination process other than the behavior detection process after a predetermined period has elapsed after the collision of the vehicle has been detected, and may record the determination result in a memory.

[0023] Preferably, the control unit is capable of executing an update function for updating the determination process, and when a collision of the vehicle is predicted or detected, the determination process and the update function other than the action detection process may be stopped, and the action detection process may be executed.

[0024] The traffic system of the present invention comprises: an occupant monitoring device for a vehicle as described in any one of the above items, which is arranged in the vehicle; and other communication devices, which communicate with the occupant monitoring device, wherein the other communication devices at least send the possibility of a collision to the occupant monitoring device, and the occupant monitoring device predicts or detects a collision of the vehicle based on the received possibility of a collision.

[0025] The vehicle occupant monitoring device of the present invention includes: a camera device that captures occupants riding in the vehicle; and a control unit that can execute multiple determination processes based on the images captured by the camera device, the multiple determination processes including a motion detection process that detects the actions of the occupants, wherein the control unit stops determination processes other than the motion detection process and executes the motion detection process when a collision of the vehicle is predicted or detected.

[0026] Preferably, the action detection process detects an action of at least a driver among the passengers.

[0027] Preferably, it also has: a switching unit, which is configured to switch the viewing angle captured by the camera device between a first viewing angle when capturing the driver and a second viewing angle that is wider than the first viewing angle when capturing a vehicle including the driver and other passengers, wherein when a collision of the vehicle is predicted or detected, the switching unit switches the shooting viewing angle from the first viewing angle to the second viewing angle, and the control unit detects the actions of the driver and the other passengers as the action detection processing.

[0028] Preferably, it also has: a memory capable of recording images captured by the camera device, wherein, when the control unit predicts a collision of the vehicle, it executes the action detection processing and stops the determination processing other than the action detection processing; when the collision of the vehicle is detected, it stops the determination processing including the action detection processing and records the images captured by the camera device in the memory.

[0029] Preferably, the control unit restarts the determination process other than the behavior detection process when canceling the prediction of the collision after predicting the collision of the vehicle and before actually detecting the collision of the vehicle.

[0030] Preferably, as the multiple judgment processes, the control unit is capable of executing judgment processes of multiple safety functions for the safety of the occupant including the motion detection process and judgment processes of convenience functions for the convenience of the occupant, and when the control unit restarts the judgment process other than the motion detection process, it gives priority to restarting the judgment process of the safety function over the judgment process of the convenience function.

[0031] Preferably, the control unit, in the determination process of the safety function, includes, in addition to the action detection process, at least one of the following: driver's strabismus determination, drowsiness determination, sleepiness determination and deceased person determination; and in the determination process of the convenience function, includes at least one of the following: posture determination, emotion determination and personal determination.

[0032] Preferably, it also includes: a memory, which is constructed to be able to record the determination results of the multiple determination processes performed by the control unit, wherein the control unit starts the determination process other than the action detection process when a prescribed period has passed after detecting a collision of the vehicle, and records the determination results in the memory.

[0033] Preferably, the control unit is capable of executing an update function for updating the determination process, and when a collision of the vehicle is predicted or detected, the determination process other than the action detection process and the update function are stopped, and the action detection process is executed.

[0034] The transportation system of the present invention includes: an occupant monitoring device for a vehicle according to any one of the above-mentioned embodiments, which is arranged in the vehicle; and other communication devices outside the vehicle, which are configured to communicate with the occupant monitoring device, wherein the communication device at least sends the possibility of a collision to the occupant monitoring device, and the occupant monitoring device predicts or detects a collision of the vehicle based on the possibility of the collision received from the communication device.

[0035] The vehicle occupant monitoring device of the present invention includes: a camera device, which is configured to be able to photograph occupants riding in the vehicle; and a control unit, which is configured to be able to execute multiple determination processes based on the images captured by the camera device, the multiple determination processes including a motion detection process for detecting the actions of the occupants, and the control unit executes the motion detection process in priority to determination processes other than the motion detection process when predicting or detecting a collision of the vehicle.

[0036] Effects of the Invention

[0037] In the present invention, the control unit that can execute multiple determination processes including at least the action detection process for the driver on the vehicle based on the captured image stops the determination processes other than the action detection process and executes the action detection process when a collision of the vehicle is predicted or detected. Therefore, after the collision of the vehicle is predicted or detected, the control unit of the occupant monitoring device can update the action detection information of the occupant at very short intervals, and can ensure the immediacy of the action detection information of the occupant. It can be expected that the occupant's action at the time of collision can be detected with high accuracy to a degree that can be appropriately used to protect the occupant, thereby improving the occupant protection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] 1(A)-(C) are schematic illustrations of a car that can be used in a transportation system according to a first embodiment of the present invention;

[0039] Figure 2 1 (A) - (C) of the vehicle control system schematic diagram;

[0040] Figure 3 Is has Figure 2 A diagram showing the configuration of an occupant monitoring device of an occupant monitoring ECU;

[0041] Figure 4 is an explanatory diagram showing an example of a plurality of functions implemented by an occupant monitoring device;

[0042] Figure 5 yes Figure 3 A flowchart of the occupant monitoring ECU performing startup processing of multiple functions;

[0043] Figure 6 is a diagram illustrating an example of an AI function executed by an occupant monitoring ECU;

[0044] Figure 7 It is the same as the AI ​​function. Figure 5 A flowchart of the corresponding startup processing process;

[0045] Figure 8 This is a flowchart of the activation process of a plurality of functions performed by the occupant monitoring ECU in the traffic system according to the second embodiment of the present invention.

[0046] Explanation of symbols

[0047] 1…Automobile (vehicle), 10…Seat belt device, 20…Airbag device, 31…External camera, 40…Occupant monitoring device, 41…Internal camera, 58…Occupant monitoring ECU (control unit), 75…Acceleration sensor, 80…Traffic system, 81…Communication base station, 82…Other automobiles, 83…Server device, 101…In-vehicle communication unit, 102…Memory, 103…Timer, 104…Lens, 105…Drive unit, 106…Internal bus, 107…Out-vehicle emergency communicator, 108…Auxiliary acceleration sensor, 109…Auxiliary external camera, 110…Battery DETAILED DESCRIPTION

[0048] Embodiments of the present invention will be described below based on the drawings.

[0049] [First embodiment]

[0050] 1(A)-(C) are schematic explanatory diagrams of a car 1 that can be used in a transportation system 80 according to a first embodiment of the present invention.

[0051] The automobile 1 shown in FIG. 1 (A) to (C) is an example of a vehicle. The automobile 1 has a vehicle cabin 2 in the center of the vehicle body for passengers to sit in. Inside the vehicle cabin 2 are seats 3 for passengers to sit in. A footrest 4 is provided at the front of the vehicle cabin 2. On the front side of the seat 3, a steering wheel 5 protrudes backward from the footrest 4. Passengers can sit on the seat 3 and operate operating components such as the steering wheel 5.

[0052] Figure 2 1(A) to (C) are schematic explanatory diagrams of a control system 50 of the automobile 1 . Figure 2 Occupant monitoring device 40 and other communication devices of transportation system 80 are shown.

[0053] Figure 2 In FIG. 1 , a plurality of control modules constituting a control system 50 of the automobile 1 are represented and indicated by respective incorporated control ECUs (Electronic Control Units).

[0054] Specifically, Figure 2The driving ECU 51, steering ECU 52, brake ECU 53, automatic driving / driving support ECU 54, driving operation ECU 55, detection ECU 56, air conditioning ECU 57, passenger monitoring ECU 58, protection ECU 59, external communication ECU 60, UI ECU 61, and system ECU 62 are shown. The plurality of control ECUs are connected to a central gateway 67 (CGW) 67 as a relay device via a vehicle network 66 such as CAN (Controller Area Network) or LIN (Local Interconnect Network) adopted in the automobile 1.

[0055] Furthermore, in each control module, the control ECU is connected to an electronic device used in the automobile 1. The activated control ECU executes various processes to control the operation of the respectively connected electronic devices according to the information (data) acquired from the vehicle network 66. In addition, the control ECU also outputs information (data) such as the operation status of the respectively connected electronic devices to the vehicle network 66.

[0056] For example, the driving operation ECU 55 is connected to operation detection sensors such as the steering wheel 5, the brake pedal 71, the accelerator pedal 72, and the shift lever 73, which are operated by the occupant to control the travel of the automobile 1. The driving operation ECU 55 outputs control information corresponding to the operation amount to the vehicle network 66. The drive ECU 51, the steering ECU 52, and the brake ECU 53 obtain information from the vehicle network 66 to control the travel of the automobile 1.

[0057] The detection ECU 56 is connected to the speed sensor 74 of the car 1, the acceleration sensor 75 for detecting acceleration in a collision, etc., the above-mentioned external camera 31, etc. The detection ECU 56 outputs the values ​​of the speed sensor 74 and the acceleration sensor 75 of the car 1, the image of the external camera 31, etc. to the vehicle network 66. The detection ECU 56 can also predict a collision based on the image of the external camera 31 and output the prediction result to the vehicle network 66. The detection ECU 56 can output a collision detection to the vehicle network 66 when the acceleration of the acceleration sensor 75 is greater than or equal to a threshold value higher than that in normal driving caused by an impact. The detection ECU 56 can output a collision detection with a pedestrian to the vehicle network 66 when the acceleration of the acceleration sensor 75 is greater than or equal to a threshold value higher than usual after predicting a collision with a pedestrian or a bicycle based on the image of the external camera 31. The central gateway 67 forwards the information. The UIECU 61 obtains information from the vehicle network 66 and displays the information on the display device 76 connected thereto. The UIECU 61 is connected to the operation device 77 and the display device 76 operated by the occupant.

[0058] The above-mentioned interior camera 41 and microphone 78 are connected to the passenger monitoring ECU 58. The passenger monitoring ECU 58 is a control ECU of the passenger monitoring device 40, and performs various processes related to passengers riding in the car 1 based on the image of the interior camera 41, the sound of the microphone 78, and information such as acceleration of impact acquired from the vehicle network 66. The passenger monitoring ECU 58 outputs images, sounds, and other information (data) to the vehicle network 66 as needed.

[0059] The airbag device 20 and the seat belt device 10 are connected to the protection ECU 59 . The protection ECU 59 controls the operation of the airbag device 20 and the seat belt device 10 based on information acquired from the vehicle network 66 .

[0060] The external communication ECU 60 wirelessly communicates with, for example, a communication base station 81 or a communication device of another car 82 that exists outside the car 1. The communication base station 81 or the communication device of another car 82 together with the server device 83 constitute the transportation system 80. The external communication ECU 60 wirelessly transmits information acquired from the vehicle network 66 to the communication base station 81 or the communication device of another car 82. The transmitted information can be used, for example, in the server device 83 or in another car 82. In addition, the external communication ECU 60 receives information from the communication base station 81 or the communication device of another car 82 and outputs it to the vehicle network 66. Thus, the occupant monitoring ECU 58 of the car 1, for example, can transmit and receive information (data) to and from the server device 83 or another car 82 outside the car through the external communication ECU 60.

[0061] In addition, the battery 91 installed in the vehicle 1 supplies power to each part, so that Figure 2 The control system 50 shown operates. Power supply lines from the battery 91 to the various components are laid in the automobile 1 together with, for example, a communication cable of the vehicle network 66. In addition to the battery 91, power may be supplied to the control system 50 from a generator or a power receiver.

[0062] As described above, the automobile 1 is provided with occupant protection devices such as the seat belt device 10 and the airbag device 20 in order to protect the occupants in the event of a collision.

[0063] The seat belt device 10 includes a seat belt 11 that is laid over the front of an occupant sitting on the seat 3. The seat belt device 10 applies tension to the seat belt 11 during a collision to restrain the occupant so that the occupant is less likely to leave the seat 3.

[0064] The airbag device 20 includes, for example, an airbag 21 that is deployed in front of or on the side of the seat 3. The airbag device 20 deploys the airbag 21 during a collision to support an occupant who is about to fall or leave the seat 3.

[0065] In this way, the automobile 1 can protect the passengers in the automobile during a collision.

[0066] However, the automobile 1 may collide not only in a specific direction such as the front, but also in a direction such as an oblique direction as shown by arrows in FIG. 1(A) to (C).

[0067] In this special collision situation, the occupant protection device may not be able to fully protect the occupants in the vehicle.

[0068] Furthermore, even if the occupant protection device can prevent serious injuries to the occupant, it cannot necessarily prevent minor injuries. The occupant protection device cannot guarantee that the occupant will not suffer any injuries in all collisions that are expected to occur to the automobile 1.

[0069] Therefore, in order to further enhance the protection performance of the occupants and others in the automobile 1 , further improvements are required.

[0070] The automobile 1 is provided with an external camera 31 for monitoring the front direction of the automobile 1 and the like, and by analyzing the captured images, a collision can be predicted in advance or the automobile 1 can be automatically controlled based on the prediction.

[0071] In addition, a passenger monitoring device 40 (DMS: Driver Monitoring System) 40 may be provided in the automobile 1 to monitor passengers riding in the automobile 1. The passenger monitoring device 40 is provided, for example, at the center of the vehicle width direction on the upper part of the footrest 4 to identify passengers riding in the automobile 1. Thus, in the automobile 1, the driving control of the automobile 1 can be permitted or prohibited according to the identified passengers.

[0072] Furthermore, it is conceivable to determine the position and movement of the occupant, especially the head, at the time of the collision based on the image of the internal camera 41 of the occupant monitoring device 40, and control the operation of the occupant protection device accordingly. In particular, it is foreseeable that it is desirable to detect the movement of not only the driver but also other occupants other than the driver at the time of the collision. In this way, the protection of the occupant by the occupant protection device can be optimized.

[0073] In addition, the occupant monitoring device 40 of the automobile 1 is required to determine the posture of the driver, etc. The occupant monitoring device 40 is required to have various functions.

[0074] In order to execute these various detection functions and determination functions, the occupant monitoring ECU 58 used in the occupant monitoring device 40 is required to have a high processing capability.

[0075] Furthermore, the occupant monitoring ECU 58 used in the occupant monitoring device 40 may only have a high processing capability and may not be able to obtain sufficient performance for occupant protection, for example.

[0076] For example, if the occupant monitoring ECU 58 is inserting the processing of other functions such as the occupant determination function during the period when the occupant monitoring ECU 58 is trying to quickly determine the position and behavior of the occupant in the event of a collision, the occupant monitoring ECU 58 may not be able to determine the latest position and behavior of the occupant required in the event of a collision. In this case, the occupant protection device will perform occupant protection based on the non-latest position and behavior of the occupant in the recent past.

[0077] In this way, the occupant monitoring device 40 of the automobile 1 is required to appropriately execute a plurality of determination functions according to the situation of the automobile 1 .

[0078] Figure 3 Is has Figure 2 A configuration diagram of the occupant monitoring device 40 of the occupant monitoring ECU 58.

[0079] Figure 3 The occupant monitoring device 40 includes an in-vehicle communication unit 101, an occupant monitoring ECU 58, a memory 102, a timer 103, a microphone 78, an internal camera 41, a driver 105 for a lens 104, and an internal bus 106 connecting them. For example, the occupant monitoring device 40 may also independently connect an external emergency communicator 107, an auxiliary acceleration sensor 108, and an auxiliary external camera 109 for photographing pedestrians, etc. to the internal bus 106. Thus, the occupant monitoring device 40 can independently ensure a function for executing a process at the time of a collision.

[0080] In addition, power is supplied from the battery 91 of the automobile 1 to each part of the passenger monitoring device 40. A storage battery 110 storing backup power is connected to the power supply path. The passenger monitoring device 40 may be provided to be detachable from the automobile 1.

[0081] The in-vehicle communication unit 101 is connected to the vehicle network 66. The in-vehicle communication unit 101 transmits and receives information (data) via the vehicle network 66 with other control ECUs such as the detection ECU 56, the protection ECU 59, and the external communication ECU 60 shown in the figure.

[0082] The interior camera 41 is an imaging device that captures an image of the interior 2 of the automobile 1. The interior camera 41 captures an image of passengers sitting in the vehicle.

[0083] The drive unit 105 drives the lens 104 to control the position of the lens 104. By controlling the position of the lens 104, the shooting angle of the interior camera 41 is changed. When the lens 104 is controlled to the farthest position, as shown in the double-dotted line frame of FIG. 1 (B), the interior camera 41 captures the upper body including the head of the passenger who is a driver sitting on the seat 3 to operate the steering wheel 5, etc. When the lens 104 is controlled to the closest position, as shown in the single-dotted line frame of FIG. 1 (B), the interior camera 41 captures the entire vehicle interior 2. In this case, in addition to the passenger who is the driver, the interior camera 41 can also capture other passengers sitting on the front passenger seat or the seat 3 in the rear seat.

[0084] The microphone 78 converts the sound of the vehicle interior 2 of the vehicle 1 into an electric signal.

[0085] The timer 103 measures elapsed time or the time.

[0086] The off-vehicle emergency communicator 107 is a communication device that can communicate with the communication base station 81 or other vehicles 82 outside the vehicle 1 in an emergency such as an accident of the vehicle 1. The off-vehicle emergency communicator 107 can communicate with the communication base station 81 or other vehicles 82 in the same communication method as the external communication ECU 60.

[0087] The auxiliary acceleration sensor 108 detects the acceleration acting on the vehicle 1 in the same manner as the acceleration sensor 75 .

[0088] The memory 102 records various images and detection values ​​acquired by the occupant monitoring device 40. In addition, the memory 102 records a program for occupant monitoring.

[0089] The occupant monitoring ECU 58 reads and executes the occupant monitoring program from the memory 102. Thus, the control unit of the occupant monitoring device 40 is realized in the occupant monitoring ECU 58. The control unit of the occupant monitoring device 40 controls the overall operation of the occupant monitoring device 40 so that the occupant monitoring device 40 realizes the occupant monitoring function.

[0090] Figure 4 It is an explanatory diagram showing an example of a plurality of functions implemented by the occupant monitoring device 40 .

[0091] Depend on Figure 4 The multiple functions implemented by the occupant monitoring device 40 can be divided into a basic function of the occupant monitoring device 40, a safety function for determining the safety of the occupant, and a convenient function for improving the convenience of the occupant.

[0092] The basic functions include, for example, a camera function of the interior camera 41, a function of recording images in the memory 102, an interior communication function of the interior communication unit 101, and an exterior communication function of the interior communication unit 101 communicating with the exterior communication ECU 60. The camera function may also be realized by the interior camera 41 and the auxiliary exterior camera 109. The exterior communication function may also be realized by the exterior emergency communicator 107.

[0093] Safety functions include, for example, a driver action detection function for detecting the driver's action based on a captured image including the driver, an other occupant action detection function for detecting the action of other occupants based on a captured image including the driver, a squint (line of sight) determination function for determining the driver's squint based on a camera image including the driver, a drowsiness and sleepiness determination function for determining the driver's drowsiness and sleepiness based on a camera image including the driver, and a deceased person determination function for determining the life or death of the driver based on a camera image including the driver.

[0094] The convenient functions include, for example, a posture determination function for determining the posture of a driver based on a captured image including the driver, an emotion determination function for determining the emotion of a driver based on a captured image including the driver, an individual identification function for identifying an individual driver based on a captured image including the driver, and an other determination function for performing other determinations based on a captured image including the driver. The convenient functions include an update function for updating an AI determination engine used for AI determination processing in the above functions.

[0095] The occupant monitoring ECU 58 controls execution / stop of these multiple functions (processes) according to the situation of the automobile 1 . Figure 4 The normal start state is shown, in which the automobile 1 can run normally; the post-collision prediction state is shown, in which a collision of the automobile 1 is predicted; and the post-collision detection state is shown, in which an actual collision of the automobile 1 is detected.

[0096] When in the normal start state, the passenger monitoring ECU 58 executes the operation of the passenger monitoring device 40 Figure 4 Among them, the occupant monitoring ECU58 performs other occupant action detection functions only when the lens 104 is at a wide angle. The occupant monitoring ECU58 can perform multiple determination processes.

[0097] When it is in the post-collision prediction state, the occupant monitoring ECU58 only performs part of the functions of the occupant monitoring device 40 and stops other functions. For example, the occupant monitoring ECU58 stops all convenient functions. In addition, the occupant monitoring ECU58 stops the squint (line of sight) judgment function, the drowsiness and sleepiness judgment function, and the death judgment function among the safety functions. The occupant monitoring ECU58 performs the basic functions, the driver action detection function and other occupant action detection functions in addition to these. In addition, the occupant monitoring ECU58 can also stop the external vehicle communication function among the basic functions.

[0098] When it is in the post-collision detection state, the occupant monitoring ECU58 only performs the minimum functions less than the post-collision prediction state, and stops other functions. For example, the occupant monitoring ECU58 stops all convenience functions and safety functions. In addition, the occupant monitoring ECU58 stops the external vehicle communication function among the basic functions. The occupant monitoring ECU58 performs the camera function, image recording function, and in-vehicle communication function in addition to these. In addition, the occupant monitoring ECU58 can also perform the external vehicle communication function instead of the in-vehicle communication function, or perform the in-vehicle communication function and the external vehicle communication function together. The stopped functions will not consume the processing power of the occupant monitoring ECU58. In addition, the power consumption of the occupant monitoring ECU58 can also be suppressed.

[0099] Figure 5 yes Figure 3 A flowchart of the occupant monitoring ECU 58 performing activation processing for multiple functions.

[0100] When the occupant monitoring device 40 and the automobile 1 are started together, the occupant monitoring ECU 58 executes Figure 5 In addition, the occupant monitoring ECU 58 can be executed even when the automobile 1 is parked, for example, by releasing the door key. Figure 5 Startup processing.

[0101] In step ST1, the passenger monitoring ECU 58 starts the passenger monitoring device 40. Figure 4 All functions shown in the normal startup state. The occupant monitoring device 40 performs the following personal identification processing: for example, a still image of the driver is captured by the internal camera 41 provided for occupant monitoring, and the driver is determined based on the image. When authenticated as a driver pre-registered in the car 1, the occupant monitoring ECU58 outputs the result of authentication passing to the vehicle network 66 through the in-vehicle communication unit 101. For example, the driving operation ECU55 allows the operation of the steering wheel 5, etc. according to the result of authentication passing. As a result, the car 1 can travel. When the driver gets off the car, the occupant monitoring ECU58 outputs the result of authentication failure indicating that the driver has not been authenticated to the vehicle network 66. The driving operation ECU55 prohibits the operation of the steering wheel 5, etc. according to the result of authentication failure. As a result, the car 1 cannot travel.

[0102] In step ST2, the occupant monitoring ECU58 determines whether there is a collision forecast. For example, the occupant monitoring ECU58 obtains the collision prediction output by the detection ECU56 to the vehicle network 66 from the in-vehicle communication unit 101, and determines whether there is a collision forecast. In addition, when the external communication ECU60 receives information about the possibility of a collision with the vehicle or other vehicles 82 traveling in front of it from other vehicles 82 and the communication base station 81, it will output the collision information ahead of the road ahead to the vehicle network 66. The occupant monitoring ECU58 can also obtain the collision information ahead of the road ahead and determine whether there is a collision forecast. When there is no collision forecast, the occupant monitoring ECU58 causes the processing to enter step ST7. When there is a collision forecast, the occupant monitoring ECU58 causes the processing to enter step ST3.

[0103] In step ST3 after the collision is predicted, the occupant monitoring ECU 58 instructs the drive unit 105 to switch the viewing angle from a narrow angle to a wide angle. The interior camera 41 switches from a state of photographing only the driver to a state of photographing the entire vehicle cabin 2. In addition, since it is in the normal start state, the occupant monitoring ECU 58 performs driver action detection processing and other occupant action detection processing based on the captured images.

[0104] In step ST4, the occupant monitoring ECU 58 sets the operating state of the occupant monitoring device 40 to the first retracted state (first retracted state) after the collision prediction. In the first retracted state, the occupant monitoring ECU 58 only executes Figure 4 In the post-collision prediction state, the function of continuing (O) is added, and the judgment processing (function) other than the action detection processing is added with stopping (×). Therefore, the occupant monitoring ECU58 executes the driver action detection function and other occupant action detection functions without being disturbed by the insertion of other judgment processing. The occupant monitoring ECU58 can quickly and repeatedly execute the driver action detection function and other occupant action detection functions in a short and stable cycle. As a result, the occupant monitoring ECU58 can detect the current positions and current actions of multiple occupants including the driver based on the latest wide-angle image taken by the internal camera 41. The occupant monitoring ECU58 can output the latest result as the judgment result of the positions and actions of multiple occupants including the driver to the vehicle network 66. In the protection ECU59, the operation of the airbag device 20 and the seat belt device 10 can be prepared based on the latest judgment result of the positions and actions of multiple occupants including the driver.

[0105] In step ST5, the occupant monitoring ECU 58 acquires the acceleration acting on the automobile 1 due to the collision and detects the collision. The occupant monitoring ECU 58 acquires the collision detection output by the detection ECU 56 to the vehicle network 66 from the in-vehicle communication unit 101, for example, and detects the collision. The occupant monitoring ECU 58 can determine that a collision is detected when the acceleration of the auxiliary acceleration sensor 108 is greater than or equal to a threshold value of whether there is a collision higher than that during normal driving. If no collision is detected, the process returns to step ST2. If a collision is detected, the process proceeds to step ST6.

[0106] In step ST6, the occupant monitoring ECU 58 sets the operating state of the occupant monitoring device 40 to the second retracted state after the collision detection. In the second retracted state, the occupant monitoring ECU 58 only executes Figure 4 The function of continuing (O) is added in the state after collision detection, and the function of stopping (×) is added. As a result, the occupant monitoring ECU 58 only performs the processing of recording the images of multiple occupants taken by the internal camera 41 into the memory 102, and will not be disturbed by the insertion of other determination processing. The occupant monitoring ECU 58 only performs the processing of repeatedly recording the images of multiple occupants into the memory 102, thereby being able to operate with the minimum power consumption required.

[0107] If no collision is predicted in step ST2, the occupant monitoring ECU 58 advances the process to step ST7. The occupant monitoring ECU 58 executes a process for returning to the normal state.

[0108] In step ST7 , the occupant monitoring ECU 58 instructs the drive unit 105 to switch the viewing angle from a wide angle to a narrow angle. The interior camera 41 switches from photographing the entire vehicle interior 2 to photographing only the driver.

[0109] In step ST8, the occupant monitoring ECU 58 executes a restart process to return the operating state of the occupant monitoring device 40 from the first retreat state after the collision prediction to the normal activation state. The occupant monitoring ECU 58 activates the stopped determination process (function) other than the action detection process, and activates Figure 4 All functions shown in the normal startup state. Figure 4 The occupant monitoring ECU 58 prioritizes the safety function determination process over the convenience function determination process.

[0110] Here, the AI ​​updating function of the occupant monitoring device 40 will be described.

[0111] Figure 6 is a diagram for explaining an example of the AI ​​function executed by the passenger monitoring ECU 58. Figure 4Each function has Figure 6 AI functions.

[0112] Figure 6 The AI ​​function includes an AI update engine 111 , setting data 112 , and a main reasoning algorithm 113 .

[0113] The setting data 112 are, for example, various parameters.

[0114] The main inference algorithm 113 is a program that executes the processing of each function of the occupant monitoring device 40 or a part of the processing thereof using the setting data 112. The main inference algorithm 113 generates the expected value related to the feature of the judgment object based on the image data, for example. In this case, Figure 4 Each function can perform determination processing on the image portion that obtains the highest expected value according to the main inference algorithm 113.

[0115] The AI ​​update engine 111 is a program that updates the setting data 112 and / or the main inference algorithm 113 according to the learning data.

[0116] For example, the AI ​​update engine 111 acquires image data as learning data, extracts its features, performs inference learning processing based on the features, and updates the setting data 112 and / or the main inference algorithm 113 based on the results. Thus, the setting data 112 and / or the main inference algorithm 113 are updated.

[0117] Figure 7 It is the same as the AI ​​function. Figure 5 Flowchart of the corresponding startup processing flow.

[0118] In step ST11 , the occupant monitoring ECU 58 activates the main reasoning algorithm 113 .

[0119] In step ST12 , the occupant monitoring ECU 58 activates the AI ​​update engine 111 .

[0120] In step ST13 , the occupant monitoring ECU 58 as the AI ​​update engine 111 acquires image data as learning data. The image data may be an image captured by the interior camera 41 .

[0121] In step ST14 , the occupant monitoring ECU 58 as the AI ​​update engine 111 extracts features included in the image data.

[0122] In step ST15 , the occupant monitoring ECU 58 as the AI ​​update engine 111 executes an inference learning process based on the extracted features and image data.

[0123] In step ST16, the occupant monitoring ECU 58 as the AI ​​update engine 111 updates the setting data 112 and / or the main inference algorithm 113 based on the result of the inference learning process. Thus, the setting data 112 and / or the main inference algorithm 113 are updated.

[0124] In step ST17, the occupant monitoring ECU 58 determines whether there is a collision prediction. Figure 5 The same as step ST2 of . When no collision is predicted, the occupant monitoring ECU 58 returns the process to step ST12. The occupant monitoring ECU 58 repeats the above process, continuously updating the setting data 112 and / or the main reasoning algorithm 113. When a collision is predicted, the occupant monitoring ECU 58 proceeds to step ST18.

[0125] In step ST18, the occupant monitoring ECU 58 stops the AI ​​update engine 111. Thereafter, the occupant monitoring ECU 58 returns the process to step ST17. The occupant monitoring ECU 58 continues to stop the AI ​​update engine 111 while the collision is being predicted. The occupant monitoring ECU 58 no longer executes the processing of the AI ​​update engine 111 while the collision is being predicted. The occupant monitoring ECU 58 can stop the update function and the determination processing other than the action detection processing, and repeatedly execute the action detection processing.

[0126] As described above, in the present embodiment, when the collision of the automobile 1 is predicted or detected, the occupant monitoring ECU (which performs a plurality of determination processes including at least a motion detection process for the driver on board based on the captured image) stops determination processes other than the motion detection process and repeatedly performs the motion detection process. Therefore, after the collision of the automobile 1 is predicted or detected, the occupant monitoring ECU of the occupant monitoring device 40 can update the occupant's motion detection information at very short intervals, and can ensure the immediacy of the occupant's motion detection information. It can be expected that the occupant's motion at the time of the collision can be detected with high accuracy to a degree that can be appropriately used to protect the occupant, thereby improving the occupant protection performance.

[0127] In particular, in the present embodiment, when a collision of the automobile 1 is predicted or detected, the state of photographing the driver is switched to the state of photographing the driver and other passengers by the switching unit. For example, when a collision of the automobile 1 is predicted, the passenger monitoring ECU switches from the state of photographing the driver to the state of photographing the driver and other passengers by the switching unit, performs the action detection processing of the driver and other passengers, and stops the determination processing other than the action detection processing. Therefore, the passenger monitoring ECU can perform the action detection processing of the driver and other passengers, and detect the actions of each passenger at the time of the collision for multiple passengers including the driver riding in the automobile 1. The passenger monitoring ECU stops the determination processing other than the action detection processing, so even if the action detection processing of other passengers other than the driver is performed, the action detection information of each passenger can be updated at very short intervals, and high-precision detection can be performed to a degree that can be appropriately used for passenger protection.

[0128] In addition, in the present embodiment, when a collision of the automobile 1 is detected, the passenger monitoring ECU stops the determination process including the action detection process, and repeatedly records the image captured by the camera device in the memory 102. Therefore, after the collision is detected, the passenger monitoring device 40 only performs the process of repeatedly recording the image of the vehicle interior 2 in the memory 102. As a result, the passenger monitoring device 40 can record the post-collision status of multiple passengers riding in the automobile 1 in a cumulative manner. The power consumption generated by the operation of the passenger monitoring ECU can be suppressed, and the remaining power in the automobile 1 can be effectively utilized to cumulatively record the post-collision status of multiple passengers riding in the automobile 1.

[0129] In this embodiment, when the occupant monitoring ECU does not detect a collision after predicting a collision of the automobile 1, it prioritizes restarting the determination processing of safety functions such as the driver's squint determination, drowsiness determination, sleepiness determination, and death determination through determination processing of convenient functions such as posture determination, emotion determination, and personal determination. Therefore, the occupant monitoring device 40 can immediately perform the driver's squint determination, drowsiness determination, sleepiness determination, and death determination after restarting.

[0130] In the present embodiment, the occupant monitoring ECU stops the determination processing other than the action detection processing and the update function of the determination processing when predicting or detecting the collision of the automobile 1. Therefore, determinations such as actions, postures, emotions, squinting, drowsiness, sleepiness, and the dead can be performed by the determination processing that has been updated before the collision. These determinations can be appropriately performed by performing the determination processing that can be updated to suit the occupant after the occupant gets on the vehicle.

[0131] [Second Embodiment]

[0132] Next, a transportation system 80 according to a second embodiment of the present invention will be described. In the following description, differences from the above embodiment will be mainly described. In this embodiment, the same components as those in the above embodiment are given the same reference numerals, and their descriptions are omitted.

[0133] Figure 8 This is a flowchart of the activation process of a plurality of functions of the occupant monitoring ECU 58 in the transportation system 80 according to the second embodiment of the present invention.

[0134] When the occupant monitoring device 40 and the automobile 1 are started together, the occupant monitoring ECU 58 executes Figure 8 In addition, the occupant monitoring ECU 58 can be executed even when the automobile 1 is parked, for example, by releasing the door key. Figure 8 Startup processing.

[0135] When the occupant monitoring ECU 58 detects a collision in step ST5 , it sets the operating state of the occupant monitoring device 40 to the second retracted state after the collision detection in step ST6 .

[0136] In the subsequent step ST21, the occupant monitoring ECU 58 determines whether the predetermined post-collision detection period (post-collision detection period) is measured by the timer 103. If the post-collision detection period is not measured, the occupant monitoring ECU 58 repeats the process of step ST21. Thus, the occupant monitoring ECU 58 can accumulate and store the images of a plurality of occupants taken by the internal camera 41 in the memory 102 during the post-collision detection period. If the post-collision detection period is measured, the occupant monitoring ECU 58 advances the process to step ST8.

[0137] In step ST8, the occupant monitoring ECU 58 executes a restart process to return the operating state of the occupant monitoring device 40 from the second retracted state after the collision detection to the normal activation state. The occupant monitoring ECU 58 activates the stopped determination process (function) other than the action detection process, and activates Figure 4 All functions shown in the normal startup state. Figure 4 The occupant monitoring ECU 58 prioritizes the safety function determination process over the convenience function determination process.

[0138] In addition, the occupant monitoring ECU 58 stores various determination results based on the images of the plurality of occupants captured by the interior camera 41 in the memory 102. Thus, the memory 102 stores the determination results determined by the occupant monitoring device 40 together with the images of the occupants after the collision. Figure 4 The judgment results in each function of .

[0139] In addition, the occupant monitoring ECU 58 can output the images and determination results accumulated and recorded in the memory 102 after the collision from the in-vehicle communication unit 101 to the vehicle network 66. The external communication ECU 60 obtains the images and the like after the collision from the vehicle network 66 and transmits them to the transportation system 80. The occupant monitoring ECU 58 can transmit the images and the determination results in the memory 102 from the emergency communicator 107 outside the vehicle to the transportation system 80. By receiving the images and the determination results from the in-vehicle communication unit 101, the server device 83 and other cars 82 of the transportation system 80 can grasp the post-collision state of the relevant car 1 in detail. The server device 83 and other cars 82, which are other communication devices communicating with the external communication ECU 60 in the transportation system 80, can obtain the images and the determination results as emergency notifications.

[0140] As described above, in this embodiment, the occupant monitoring ECU starts the determination process other than the action detection process after a predetermined period has passed since the collision of the automobile 1. Thus, the occupant monitoring device 40 can perform, for example, the posture determination, emotion determination, squint determination, drowsiness determination, sleepiness determination, and death determination of the occupant after the collision, and record these determination results in the memory 102.

[0141] In addition, in this embodiment, when a collision is predicted, the basic operation of the system does not stop, only the determination process is stopped. As a result, in this embodiment, not only can data be shared by each determination function, but also the detection accuracy of the occupant's behavior after the collision is predicted can be improved.

[0142] In addition, in the present embodiment, since only the determination process is stopped, the information on the occupant before the determination function is stopped, the settings of the calibration and determination value in each function, etc. can be retained in the memory 102 after the stop. In this case, when the prediction of the collision disappears and the determination is restarted, it is only necessary to activate the stopped determination function. Each function that is restarted can use the settings before the stop and resume appropriate operation immediately after the restart.

[0143] In addition, in the present embodiment, when a collision is predicted, a plurality of determination functions are stopped. Thus, it is possible to ensure resources for processing a wide range of images including passengers other than the driver after the collision prediction. By utilizing the recording area opened after the stop, it is possible to process a wide range of images including passengers other than the driver without increasing the memory 102.

[0144] The above-described embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited thereto, and various modifications and changes can be made without departing from the gist of the present invention.

Claims

1. A vehicle occupant monitoring device, comprising: a camera device that captures passengers riding in the vehicle; a control unit capable of executing a plurality of determination processes based on the image captured by the imaging device, the plurality of determination processes including a motion detection process for detecting the motion of the occupant; as well as a memory capable of recording images captured by the camera device, wherein, when the control unit predicts a collision of the vehicle, the control unit executes the action detection process and stops the determination process other than the action detection process; When a collision of the vehicle is detected, the determination process including the action detection process is stopped, and the image captured by the imaging device is recorded in the memory.

2. The vehicle occupant monitoring device according to claim 1, wherein: The action detection process detects an action of at least a driver among the passengers.

3. The vehicle occupant monitoring device according to claim 2, further comprising: a switching unit configured to switch the viewing angle of the camera device between a first viewing angle for photographing the driver and a second viewing angle wider than the first viewing angle for photographing the driver and other passengers, in, When a collision of the vehicle is predicted or detected, the switching unit switches the perspective of the photographing from the first perspective to the second perspective, and the control unit detects the actions of the driver and the other passengers as the action detection process.

4. The vehicle occupant monitoring device according to any one of claims 1 to 3, wherein: The control unit restarts the determination process other than the behavior detection process when canceling the prediction of the collision after predicting the collision of the vehicle and before actually detecting the collision of the vehicle.

5. The vehicle occupant monitoring device according to claim 4, wherein: As the plurality of determination processes, the control unit may execute a determination process of a plurality of safety functions for the safety of the occupant including the action detection process and a determination process of a convenience function for the convenience of the occupant. The control unit, when resuming the determination process other than the action detection process, prioritizes the resumption of the determination process of the safety function over the determination process of the convenience function.

6. The vehicle occupant monitoring device according to claim 5, wherein: The control unit includes, in the safety function determination process, at least one of the following in addition to the action detection process: driver's side-eye determination, drowsiness determination, sleepiness determination, and death determination; The convenient function determination process includes at least one of the following: posture determination, emotion determination, and personal determination.

7. The vehicle occupant monitoring device according to any one of claims 1 to 3, further comprising: The memory is configured to be capable of recording the determination results of the plurality of determination processes performed by the control unit, Here, when a predetermined period of time has elapsed after the collision of the vehicle is detected, the control unit restarts the determination process other than the action detection process and records the determination result in the memory.

8. The vehicle occupant monitoring device according to any one of claims 1 to 3, wherein: The control unit can execute an update function for updating the determination process, and when a collision of the vehicle is predicted or detected, the control unit stops the determination process other than the behavior detection process and the update function, and executes the behavior detection process.

9. A transportation system comprising: The vehicle occupant monitoring device according to any one of claims 1 to 8, which is arranged in the vehicle; as well as The other communication device outside the vehicle is configured to be able to communicate with the occupant monitoring device, wherein the communication device sends at least the possibility of a collision to the occupant monitoring device, The occupant monitoring device predicts or detects a collision of the vehicle based on the possibility of the collision received from the communication device.

10. A vehicle occupant monitoring device, comprising: A camera device configured to capture images of passengers riding in the vehicle; as well as a control unit configured to be capable of executing the following processing: processing of basic functions including an imaging function of the imaging device and an image recording function of storing the captured images; and executing a plurality of determination processes related to at least a driver of the vehicle based on the captured image, The plurality of determination processes include an action detection process for detecting an action of the occupant, The control unit is configured to execute the basic function processing and the action detection processing and stop the determination processing other than the action detection processing when a collision of the vehicle is predicted; When a collision of the vehicle is detected, the processing of the basic function is executed, and the determination processing including the action detection processing is stopped.

Images