Vehicle safety support systems

Abstract

To provide a safety support system for a vehicle, capable of grasping a state of a crew member.SOLUTION: A safety support system for a vehicle of the present invention is a safety support system for a vehicle that supports the safety of one or more crew members who ride on a vehicle. The safety support system for a vehicle comprises: a monitoring unit that can monitor the crew member in the vehicle; a recognition unit that recognizes the state of the crew member on the basis of monitoring information from the monitoring unit; and a control unit that performs a control of an operation of the vehicle or gives a notification to the crew member on the basis of the recognition information from the recognition unit. The monitoring unit includes a monitoring camera and a monitoring radar, which are used for specifying the crew member. An evaluation function obtained by synthesizing camera data of the monitoring camera and radar data of the monitoring radar is used in order to specify the crew member. A periodic oscillation detected by the monitoring radar is the radar data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a vehicle safety support system. [Background technology]

[0002] Vehicles equipped with sensors, cameras, etc. that grasp the situation outside the vehicle to ensure safe driving are known (see, for example, Japanese Patent Application Laid-Open No. 2014-85331). The vehicle described in the above publication uses ultrasonic sensors, radar, and video cameras to recognize the space on the shoulder of the road, and can safely move to the shoulder and stop there. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-85331 Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, there have been reports of fatal accidents involving small children or pets when all occupants exited the vehicle while the children or pets were left inside. Accidents can also occur when the health of occupants, especially the driver, deteriorates while the vehicle is in motion. Vehicle safety support systems are also needed to address these types of accidents.

[0005] The present invention has been made in view of the above circumstances, and has an object to provide a vehicle safety support system that can grasp the state of an occupant. [Means for solving the problem]

[0006] A vehicle safety support system according to one embodiment of the present invention is a vehicle safety support system that supports the safety of one or more occupants in a vehicle, and includes a monitoring unit capable of monitoring occupants in the vehicle, a recognition unit that recognizes the state of the occupants based on monitoring information from the monitoring unit, and a control unit that controls the operation of the vehicle or notifies the occupants based on the recognition information from the recognition unit, and the monitoring unit has a surveillance camera. [Effects of the Invention]

[0007] The vehicle safety support system of the present invention is capable of grasping the state of an occupant. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a vehicle safety support system according to one embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram showing a vehicle equipped with the vehicle safety support system of FIG. [Figure 3] FIG. 3 is a schematic diagram showing the configuration of the monitoring unit in FIG. [Figure 4] FIG. 4 is a schematic diagram showing the configuration of the recognition unit in FIG. [Figure 5] FIG. 5 is a flow chart showing a control method for the vehicle safety support system of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] [Description of the embodiment of the present invention] First, embodiments of the present invention will be listed and described.

[0010] A vehicle safety support system according to one embodiment of the present invention is a vehicle safety support system that supports the safety of one or more occupants in a vehicle, and includes a monitoring unit capable of monitoring occupants in the vehicle, a recognition unit that recognizes the state of the occupants based on monitoring information from the monitoring unit, and a control unit that controls the operation of the vehicle or notifies the occupants based on the recognition information from the recognition unit, and the monitoring unit has a surveillance camera.

[0011] This vehicle safety support system monitors the occupants inside the vehicle using a surveillance camera, recognizes their behavior using a recognition unit, and controls the vehicle's operation or notifies the occupants using a control unit, thereby ensuring safe driving of the vehicle.

[0012] The monitoring unit may have a personal item identification means for identifying a personal item that may be brought into the vehicle, the monitoring information may include personal item information identified by the personal item identification means, the recognition information may include personal item detection information for identifying whether the personal item is present in the vehicle and disembarkation information for identifying whether all of the occupants have disembarked from the vehicle, and when the disembarkation information identifies that all of the occupants have disembarked from the vehicle and the personal item detection information identifies that the personal item is present in the vehicle, the control unit may notify the occupants that the personal item is present in the vehicle. If a personal item that was brought into the vehicle and identified by the personal item identification unit is still present in the vehicle after all of the occupants have disembarked, the personal item can be identified as an item left behind in the vehicle, and the control unit may notify the occupants that the personal item is present in the vehicle, thereby preventing items from being left behind.

[0013] The monitoring unit may include a surveillance radar. By using the surveillance radar in addition to the surveillance camera, it becomes possible to identify an object covered by a blanket, for example, thereby improving the monitoring accuracy.

[0014] The carrying object identification means may use periodic vibrations detected by the surveillance radar. If the object is a living thing, its heartbeat and breathing are detected as periodic vibrations by the surveillance radar. Therefore, by using the periodic vibrations detected by the surveillance radar, it is possible to identify whether the object is a living thing or a non-living thing. Furthermore, if the period, i.e., the heartbeat and breathing rate, is within a certain range, it is possible to infer that the object is a child. Therefore, by using the periodic vibrations detected by the surveillance radar to identify carrying objects, it becomes possible to detect with high accuracy whether a child has been left behind.

[0015] The recognition unit may include a digitization unit that expresses the occupant's state from the monitoring information as a numerical value using an evaluation function, an optimization unit that optimizes a threshold value for identifying the occupant's state based on a magnitude relationship with the numerical value, and a recognition information identification unit that identifies the occupant's state from the numerical value and the threshold value as the recognition information. Optimizing the threshold value for identifying the occupant's state in this manner can improve the accuracy of identifying the occupant's state. Note that optimizing the threshold value includes not only a method of adjusting the numerical value of the threshold value itself, but also a method of relatively adjusting the threshold value by adding a bias value to the evaluation function.

[0016] The monitoring unit includes a database storing reference images of occupants captured in advance, and an occupant identification means for identifying the occupant by comparing the image of the occupant captured by the monitoring camera with the reference image, the occupant identification means being configured to calculate a degree of match between the captured image and the reference image, and the optimization means of the recognition unit preferably uses the degree of match. The degree of match is considered to represent the validity of the individual identification of the occupant, or in other words, the reliability of the captured image. The optimization means may use the degree of match to optimize a threshold value, thereby further improving the accuracy of identifying the state of the occupant.

[0017] The monitoring unit may have a photometer that measures the light intensity around the vehicle, and the optimization means of the recognition unit may use the light intensity. It is believed that the discrimination ability of a surveillance camera decreases when the light intensity is low, while the discrimination ability of a surveillance radar increases relatively when the light intensity is low. In other words, the optimization means of the recognition unit may adjust the priority of information from the surveillance camera or surveillance radar depending on the light intensity, thereby further improving the accuracy of identifying the state of the occupant.

[0018] The monitoring unit may have a vital sensor for acquiring biometric information of the occupant, and the monitoring information may include the biometric information acquired from the vital sensor. By including the biometric information in the monitoring information in this manner, it is possible to prevent vehicle accidents caused by the deterioration of the occupant's health condition.

[0019] The monitoring unit may have an ultrasonic sensor capable of detecting a person approaching the vehicle from outside the vehicle and an exterior camera or radar for recognizing the person, the monitoring information may include vehicle exterior information obtained from the ultrasonic sensor and the exterior camera or radar, the recognition information may include suspicious person identification information for identifying whether the person is a suspicious person from the vehicle exterior information, and the control unit may use the suspicious person identification information. By using the suspicious person identification information in this way, damage to the vehicle or items inside the vehicle can be prevented.

[0020] [Details of the embodiment of the present invention] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A vehicle safety support system according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

[0021] The vehicle safety support system 1 shown in Fig. 1 is a vehicle safety support system that supports the safety of one or more occupants aboard a vehicle X shown in Fig. 2. The vehicle safety support system 1 includes a monitoring unit 10 that can monitor occupants in the vehicle X, a recognition unit 20 that recognizes the state of the occupants based on monitoring information 10a from the monitoring unit 10, and a control unit 30 that controls the operation of the vehicle X or notifies the occupants based on the recognition information 20a from the recognition unit 20.

[0022] <Monitoring section> As shown in FIG. 3, the monitoring unit 10 includes a photometer 101, a monitoring camera 102, a monitoring radar 103, an ultrasonic sensor 104, an exterior camera 105, an exterior radar 106, a vital sign sensor 107, a microphone 108, a database 109, an occupant identification means 110, and a carried item identification means 111.

[0023] (photometer) The photometer 101 measures the light intensity 112 around the vehicle X. The photometer 101 may measure the light intensity outside the vehicle X, but it is preferable to measure the light intensity inside the vehicle X. Since the vehicle safety support system 1 mainly monitors the occupants inside the vehicle X, using the light intensity inside the vehicle X improves monitoring accuracy. The measured light intensity 112 is sent to the recognition unit 20.

[0024] (surveillance cameras and radar) The surveillance camera 102 and surveillance radar 103 are used to monitor the occupants in the vehicle X and to identify the occupants and belongings they are carrying, as will be described later. The surveillance camera 102 identifies the object using an image, while the surveillance radar 103 identifies the object using reflected waves. The surveillance camera 102 can easily capture the shape and movement of the object, but on the other hand, it cannot monitor the object if it cannot be seen. For this reason, by using the surveillance radar 103 in addition to the surveillance camera 102, it is possible to identify objects, for example, even those covered by a blanket, thereby improving surveillance accuracy.

[0025] The surveillance camera 102 and surveillance radar 103 may monitor the entire living space of the vehicle X, but it is preferable to provide one set of surveillance camera 102 and surveillance radar 103 for each passenger seat, as shown in Fig. 2. Providing one set of surveillance camera 102 and surveillance radar 103 for each passenger increases the surveillance accuracy.

[0026] When a surveillance camera 102 and a surveillance radar 103 are provided for each passenger, the surveillance area 102a of the surveillance camera 102 should be an area centered around the location of the passenger's face, as shown in Fig. 2. The surveillance camera 102 can observe the passenger's complexion, facial angle, gaze angle, number of blinks, whether or not they yawn, etc. On the other hand, the surveillance area 103a of the surveillance radar 103 should be a wide area that also includes the area around the seat, from the perspective of improving the accuracy of detecting left-behind items, which will be described later.

[0027] The monitoring information 10a includes in-vehicle information 113 obtained by the monitoring camera 102 and the monitoring radar 103. By including the in-vehicle information 113 in the monitoring information 10a in this way, the state of the occupants can be ascertained.

[0028] (Ultrasonic sensors, exterior cameras and exterior radar) The ultrasonic sensor 104, the exterior camera 105, and the exterior radar 106 monitor people approaching the vehicle X. Specifically, the ultrasonic sensor 104 can detect people approaching the vehicle X from outside the vehicle. The exterior camera 105 and the exterior radar 106 are provided to recognize the people. It is preferable to use both the exterior camera 105 and the exterior radar 106 from the viewpoint of improving recognition accuracy, but recognition is possible with just one of them. Therefore, either the exterior camera 105 or the exterior radar 106 may be omitted.

[0029] The ultrasonic sensor 104, the exterior camera 105, and the exterior radar 106 monitor people approaching the vehicle X, and therefore can be disposed on both sides of the vehicle X, as shown in FIG.

[0030] Because the ultrasonic sensor 104 can easily detect objects at a relatively long distance, the monitoring area 104a of the ultrasonic sensor 104 is set to be relatively wide and is set to detect people at a distance of, for example, 5 to 20 meters, preferably 10 to 20 meters. Also, because the monitoring camera 102 captures people using its images, its monitoring area 105a is set to a short distance, for example, 5 meters or less, where people can be captured relatively clearly. The external radar 106 mainly covers intermediate distances, and its monitoring area 106a is set to a distance of, for example, 5 to 10 meters.

[0031] The monitoring information 10a includes outside-vehicle information 114 obtained from the ultrasonic sensor 104, the outside-vehicle camera 105, and the outside-vehicle radar 106. By including the outside-vehicle information 114 in the monitoring information 10a in this way, it is possible to identify whether the person is a suspicious person or not.

[0032] (Vital Sensor) The vital sensor 107 acquires the occupant's biological information 115. Specifically, the vital sensor 107 can capture the occupant's pulse rate, heart rate, heartbeat interval, blood pressure, blood sugar level, respiratory rate, etc. Among these, the pulse rate and respiratory rate are preferably used.

[0033] A non-contact type is preferable as the vital sensor 107, and among them, a Doppler sensor is particularly preferable from the viewpoint of accuracy. When a non-contact type vital sensor 107 is used, the vital sensor 107 can be placed in the same place as the surveillance camera 102, etc. In this case, from the viewpoint of ease of installation on the vehicle X, it is preferable to unitize the vital sensor 107 with the surveillance camera 102 and the surveillance radar 103.

[0034] A contact type may also be used as the vital sensor 107. As such a contact type vital sensor 107, a mat sensor is well known, and can be attached to the surface of a seat of the vehicle X, for example.

[0035] The monitoring information 10a includes biological information 115 obtained from the vital sensor 107. By including the biological information 115 in the monitoring information 10a in this way, it is possible to prevent accidents involving the vehicle X caused by the deterioration of the health condition of the occupant.

[0036] (microphone) The microphone 108 acquires speech uttered by a passenger inside the vehicle X as voice information 116.

[0037] The microphone 108 can be placed in the same place as the surveillance camera 102, etc. In this case, from the viewpoint of ease of installation on the vehicle X, it is preferable to unitize the microphone 108 with the surveillance camera 102 and the surveillance radar 103.

[0038] The monitoring information 10a includes audio information 116 obtained from the microphone 108. By including the audio information 116 in the monitoring information 10a in this way, it is possible to prevent accidents involving the vehicle X caused by the deterioration of the health of the occupants.

[0039] (Database) The database 109 stores reference images of passengers that have been photographed in advance, as well as reference images of personal belongings that may be brought into the vehicle X.

[0040] The database 109 is configured by, for example, a known storage device, and the data (reference images) thereof are referenced by an occupant identification means 110 and a carried object identification means 111, which will be described later.

[0041] (Occupant identification means) The occupant identification means 110 identifies the occupant by comparing the image of the occupant captured by the surveillance camera 102 with reference images of the occupant registered in the database 109. In other words, the occupant identification means 110 can obtain occupant information 117 that identifies who is sitting in which seat.

[0042] The occupant identification means 110 is realized by, for example, a microcontroller, or may be realized by a dedicated matching circuit.

[0043] The comparative search method can be a method of preparing a predetermined evaluation function and determining whether the evaluation function is large or small, or it can be a method of using an estimation model that has been trained by machine learning, so-called AI (artificial intelligence). Note that, for estimation using such an estimation model, known estimation techniques related to AI can be used.

[0044] The monitoring information 10a includes occupant information 117 obtained from the occupant identification means 110. The normal condition of occupants varies from person to person. By including the occupant information 117 in the monitoring information 10a in this way, it becomes possible to grasp the health conditions of occupants taking into account individual differences, thereby improving the monitoring accuracy.

[0045] Furthermore, the occupant identification means 110 is configured to be able to calculate a degree of match 118 between the captured image and the reference image. When an evaluation function is used for the comparative search, if the evaluation function is configured so that, for example, the evaluation function value is 0 when the captured image and the reference image completely match, and the evaluation function value increases as the difference increases, the evaluation function value itself can be used as the degree of match 118. The calculated degree of match 118 is sent to the recognition unit 20.

[0046] (Means for identifying belongings) The carried article identification means 111 identifies carried articles that may be brought into the vehicle X.

[0047] The carried item identification means 111 is realized by, for example, a microcontroller. This microcontroller can be used in combination with the microcontroller that realizes the occupant identification means 110. Alternatively, like the occupant identification means 110, it may be realized by a dedicated matching circuit.

[0048] Examples of the belongings include infants, pets, bags, mobile phones, etc. These belongings are detected by the recognition unit 20 (described later) to determine whether the belongings remain in the vehicle X even after all occupants have exited the vehicle, i.e., whether they are left behind.

[0049] As a method for identifying the carried item, for example, a method using a reference image of the carried item registered in the database 109 can be adopted.

[0050] In this case, it is possible to detect in advance that a personal item has been brought into the vehicle by image analysis of the surveillance camera 102 (or the exterior camera 105) while the occupant is getting into the vehicle. Specifically, this can be achieved by a method using an evaluation function or a method using AI, as with the occupant identification means 110. By detecting in advance that a personal item has been brought into the vehicle in this way, it is possible to improve the accuracy of detecting left-behind items. Furthermore, as will be described later, it is advisable to use the surveillance radar 103 (or the exterior radar 106) in combination, which may make it possible to detect personal items that cannot be seen directly from the outside and are difficult to recognize from images alone.

[0051] Alternatively, the belongings of the occupant before boarding vehicle X may be identified by image analysis or AI from the exterior camera 105 or the surveillance camera 102, and the identified items may be regarded as the carried items. In this case, the database 109 may be unnecessary. Instead of the database 109, it is effective to extract characteristics of the carried items (for example, characteristics such as red and rectangular for a bag) to generate metadata, that is, tag information for search. Furthermore, metadata linked to boarding information (such as the place and time of boarding vehicle X) may be generated.

[0052] On the other hand, belongings that may be brought into vehicle X when the occupants disembark may be read from database 109. In this case, a comprehensive search is performed to determine whether or not any belongings registered in database 109 are still in vehicle X after all occupants have disembarked. Such a comprehensive search can improve the accuracy of detecting left-behind items.

[0053] Furthermore, the above-mentioned carried items may be changed depending on the occupant aboard the vehicle X, that is, for each occupant identified by the occupant identification means 110.

[0054] The monitoring information 10a includes the carried item information 119 obtained from the carried item identification means 111. By including the carried item information 119 in the monitoring information 10a in this way, it becomes possible to detect left-behind items.

[0055] The carried object identification means 111 preferably uses periodic vibrations detected by the surveillance radar 103. If the object is a living thing, its heartbeat and breathing are detected as periodic vibrations by the surveillance radar 103. Therefore, by using the periodic vibrations detected by the surveillance radar 103, it is possible to identify whether the object is a living thing or a non-living thing. Furthermore, if the period, that is, the heartbeat and breathing rate, is within a certain range, it is possible to infer that the object is a child. Therefore, by using the periodic vibrations detected by the surveillance radar 103 to identify carried objects, it becomes possible to detect with high accuracy whether a child has been left behind.

[0056] When detecting whether an infant has been left behind, it is advisable to detect whether an infant is in the vehicle while the occupants are still in the vehicle. Since the completion of the occupants in the vehicle can be determined when the vehicle X starts moving, specifically, the belongings identification means 111 should use periodic vibrations detected by the surveillance radar 103 after the vehicle X starts moving. Furthermore, in terms of identification accuracy, it is advisable to use information from the vital signs sensor 107 to identify the infant. In this way, by identifying belongings using the periodic vibrations or the like after the vehicle X starts moving, it becomes possible to detect whether an infant has been left behind with even higher accuracy.

[0057] <Recognition part> As shown in FIG. 4, the recognition unit 20 has a quantification means 201 that expresses the state of the occupant from the monitoring information 10a as a numerical value using an evaluation function, an optimization means 202 that optimizes a threshold value for identifying the state of the occupant based on the magnitude relationship with the numerical value, and a recognition information identification means 203 that identifies the state of the occupant from the numerical value and the threshold value as recognition information 20a.

[0058] By optimizing the threshold value for identifying the state of the occupant in this way, it is possible to improve the accuracy of identifying the state of the occupant. Hereinafter, the recognition information 20a recognized by the recognition unit 20 of the vehicle safety support system 1 will be specifically described.

[0059] (Detection of lost items) The vehicle safety support system 1 can detect a personal item that is identified by the personal item identification means 111 as being likely to be brought into vehicle X as a lost item. That is, the recognition information 20a includes personal item detection information 204 that identifies whether the personal item is present in vehicle X, and disembarkation information 205 that identifies whether all of the occupants have disembarked from vehicle X. This is because when the disembarkation information 205 identifies that all of the occupants have disembarked from vehicle X, and the personal item detection information 204 identifies that the personal item is present in vehicle X, the personal item can be identified as a lost item.

[0060] The carried object detection information 204 can be extracted using the in-vehicle information 113, that is, the information from the monitoring camera 102 and the monitoring radar 103, from the monitoring information 10a.

[0061] The extraction method for extracting the carried object detection information 204 includes, for example, a coordinate axis addition step, a threshold determination step, a synthesis step, and a judgment step. The extraction method can be performed by software, but is preferably performed by hardware from the viewpoint of processing speed.

[0062] In the coordinate axis addition step, the camera data and the radar data are each created with x and y coordinates added. Adding the x and y coordinates makes it easier to overlay the camera data and the radar data.

[0063] This processing is performed by the digitizing means 201. Before adding the x and y coordinates, preprocessing such as noise removal may be performed on the information from the surveillance camera 102 and the surveillance radar 103.

[0064] In the threshold determination step, an adaptive bias value to be used in a synthesis step (to be described later) and / or a threshold value of an evaluation function to be used in a judgment step (to be described later) are determined. This processing is performed by the optimization means 202.

[0065] The adaptive bias value is a weighting value used when combining multiple types of signals. The carried object identification means 111 combines the camera data R and radar data C, i.e., obtains an evaluation function V through data fusion. In this case, for example, weighting variables a1 and a2 are used, and the evaluation function V is calculated as shown in Equation 1 below. In this case, f(R) is an evaluation function (scalar) obtained from the camera data R, g(C) is an evaluation function (scalar) obtained from the radar data C, and a1 and a2 represent adaptive bias values. In this way, the monitoring information 10a includes multiple pieces of information (information from the monitoring camera 102 and the monitoring radar 103), and by providing an evaluation function that combines these pieces of information, the accuracy of recognizing the state of the occupant can be improved. V = a1 × f(R) + a2 × g(C) 1

[0066] The luminous intensity 112 of the monitoring unit 10 may be used to determine this adaptive bias value. It is believed that the discrimination ability of the monitoring camera 102 decreases when the luminous intensity 112 is low, while the discrimination ability of the monitoring radar 103 increases relatively when the luminous intensity 112 is low. In other words, by adjusting the priority of the information from the monitoring camera 102 or the monitoring radar 103 depending on the luminous intensity 112, and determining the weighting variables a1 and a2 so that a2 is large when the luminous intensity 112 is low, for example, it is possible to further improve the accuracy of determining whether or not the occupant has left something behind.

[0067] The adaptive bias value may be determined using the degree of agreement 118 of the monitoring unit 10. The degree of agreement 118 is considered to represent the validity of the individual identification of the occupant, in other words, the reliability of the captured image. For example, if the degree of agreement 118 is low, it can be determined that the discrimination ability of the monitoring camera 102 has decreased. For example, by optimizing the weighting variables a1 and a2 so that a2 becomes larger when the degree of agreement 118 is low, it is possible to further improve the accuracy of determining whether the occupant has left something behind. Note that the luminous intensity 112 and the degree of agreement 118 can also be used in combination for the weighting variables a1 and a2. The same applies to other elements below.

[0068] Furthermore, weighting variables may be selected according to the characteristics of the image to be recognized. For example, when resizing an image from the surveillance camera 102 included in the in-vehicle information 113 to an input format suitable for the optimization means 202, such as a resolution, it is possible to determine whether the input image is a color image or a binary image (such as an infrared image). Therefore, it is advisable to prepare weighting variables for color images and weighting variables for binary images in advance and switch between them according to the determination result. This makes it possible to select an appropriate weighting variable according to the input image. It is also possible to select weighting variables according to brightness, saturation, or a combination of these, and in this case, three or more weighting variables may be prepared.

[0069] Furthermore, the value of the weighting variable may be changed depending on the type of the carried object to be detected as a lost item. For example, when detecting a bag, it is advisable to weight the information of the color that matches the color of the bag observable by the surveillance camera 102 so that it is given a higher weight. Furthermore, the shape and size of the carried object to be detected may also be used. Note that in these examples, optimization is not necessarily possible using only the weighting variables a1 and a2 in the above formula 1. For example, in order to weight the color information, a new weighting variable a1 may be added to the above formula 1. R , a1 G , a1 B This is realized by using the following formula 2, which separates the RGB color information. f(R)=a1 R ×f R (R)+a1 G ×f G (R)+a1 B ×f B (R)···2

[0070] In the vehicle safety support system 1, the belongings identification means 111 can detect whether an infant has been left behind by using periodic vibrations detected by the surveillance radar 103. A method for detecting an infant will be described below.

[0071] The vehicle safety support system 1 has a surveillance camera 102, and if the image of the target infant can be recognized by the surveillance camera 102, it is easy to detect the infant as a lost item. On the other hand, there are cases where the surveillance camera 102 cannot detect the infant, for example, when the infant is sleeping wrapped in a blanket.

[0072] In such a case, the vehicle safety support system 1 uses the surveillance radar 103. When the surveillance radar 103 observes a sleeping infant wrapped in a blanket as described above, periodic vibrations are detected. These periodic vibrations are based on the infant's heartbeat and breathing. Therefore, these periodic vibrations are the result of the periodic vibrations of the infant's heartbeat and breathing being superimposed on each other. Furthermore, it is known that the heart rate and breathing rate of an infant fall within a certain range. These are different from those of an ordinary adult or a pet.

[0073] From the above, when periodic vibrations are detected by the surveillance radar 103, the period of the vibrations is calculated, and if the vibrations consist of two periods corresponding to the heart rate and breathing rate described above, it can be assumed that an infant is present. In this way, it can be detected that an infant has been left behind.

[0074] In detecting a left-behind infant, the evaluation function V of the above formula 1 obtained by combining the camera data R and radar data C can be used. For the adaptive bias values ​​a1 and a2, for example, if the monitoring camera 102 can detect the infant, the evaluation function of the monitoring camera 102 can be used with a2=0, and if the monitoring camera 102 cannot detect the infant, the value of a2 can be increased. Whether the monitoring camera 102 can detect the infant may be determined based on the results of the monitoring camera 102, or may be determined based on environmental information. In this case, the environmental information includes the luminous intensity 112 and the blind spot of the monitoring camera 102. The environmental information may also include biological information 115 obtained from the vital sensor 107.

[0075] For example, either the heart rate or the respiratory rate of the infant may be detected by the vital sensor 107.

[0076] In the combining step, the image data is recognized by superimposing the camera data and the radar data, and the value of an evaluation function for determining whether the carried object is present inside the vehicle X is calculated. This processing is performed by the digitizing means 201.

[0077] Specifically, the procedure is as follows: the camera data and radar data acquired in time series are collected at specified time intervals and superimposed to obtain image data. By performing this superimposition process, it is possible to distinguish between moving objects (living things) and non-moving objects (non-living things), for example.

[0078] Next, the superimposed image data is compressed and time-series correlation is extracted. By extracting this time-series correlation, it is possible to detect, for example, the movement of a carried object into the blind spot of the surveillance camera 102, which makes it easier to optimize the weighting variables in the subsequent matching process.

[0079] Then, a matching process is performed to determine whether the image data contains the above-mentioned carried object from the time-series correlation. In this matching process, a matching evaluation value is calculated as the value of an evaluation function.

[0080] For example, AI can be used for this matching process. Specifically, a trained estimation model is constructed by machine learning using a variable template that changes from moment to moment in response to given environmental conditions (e.g., luminosity 112, etc.), and the time-series correlation is used as an input to perform matching with the carried item. This estimation model may be stored within vehicle X, for example, in the database 109 of the monitoring unit 10, or may be stored outside vehicle X, for example, in a cloud server, and accessed via wireless communication, etc. Note that a deep learning network such as Yolo (You Only Look Once) can be used as the AI.

[0081] This matching process can be achieved by combining low-level layer processing, which is a processing method that handles multiple data sets with a single microprocessor command, with high-level layer processing using program control. In this case, the low-level layer processing can be matched using local grayscale distributions or density gradients using Haar-Like or HOG feature extraction processing for low-frame-rate data, while the high-frame-rate data can be matched using edge strength features for each local edge direction. Furthermore, the high-level layer processing can be matched using the AdaBoost method, which integrates multiple features.

[0082] In the determination step, it is determined whether the carried object is included or not based on the magnitude relationship between the value of the evaluation function calculated in the synthesis step and the threshold value determined in the threshold value determination step. For example, if the value of the evaluation function is greater than the threshold value, it is determined that the carried object is included. Note that it may also be determined that the carried object is included if the value of the evaluation function is smaller than the threshold value depending on the evaluation function.

[0083] If it is not determined that the carried object is included, the matching process may be performed by changing the variable template so that the matching evaluation value is equal to or greater than a specified value. Furthermore, if the matching process is performed again after a specified time has elapsed and the evaluation value when added to the matching result up to that point is equal to or less than the specified value, the variable template may be similarly changed. For example, in the above-mentioned infant detection, if an infant is not detected using a template that uses only the evaluation function of the surveillance camera 102 with a2 = 0, the value of a2 may be increased and the template may be changed to one that uses both the evaluation function of the surveillance camera 102 and the evaluation function of the surveillance radar 103, thereby improving the infant detection capability.

[0084] The disembarking information 205 can be extracted using the occupant information 117 and the in-vehicle information 113 from the monitoring information 10a.

[0085] Specifically, the occupant information 117 specifies who is seated in which seat. Therefore, when the in-vehicle information 113 indicates that the occupant in the corresponding seat is no longer present, it can be estimated that the occupant has disembarked. It is more preferable to confirm that the occupant has left the vehicle using the out-vehicle information 114. Then, when it is estimated that all occupants who appeared have disembarked, it can be extracted that all occupants have disembarked.

[0086] (Health status detection) The vehicle safety support system 1 can recognize the health condition of the occupant from the in-vehicle information 113, the biological information 115, and the voice information 116. That is, the recognition information 20a includes health information 206 that indicates the health condition of each occupant.

[0087] Specifically, the vehicle safety support system 1 can recognize the health condition of the occupant as not only abnormal health but also drowsiness, absentmindedness, and inattentiveness. Note that these conditions are broadly included in the health condition because they may lead to an accident if the occupant is a driver.

[0088] The method for recognizing the health condition of the vehicle safety support system 1 can be performed in the same way as the extraction method for extracting the personal item detection information 204 when detecting lost items, except that the three evaluation functions shown in the following equation 3 are used. Dozing: Fx=a1×Dx+b1×Vx Random: Fy=a2×Dy+b2×Vy 3 Side glance: Fz=a3×Dz+b3×Vz Here, Dx, Dy, and Dz are evaluation function values ​​obtained from the in-vehicle information 113 and the voice information 116, Vx, Vy, and Vz are evaluation function values ​​obtained from the biological information 115, and a1 to a3 and b1 to b3 are weighting variables.

[0089] Drowsy driving (Dx), absent-mindedness (Dy), and inattentive driving (Dz) based on the in-vehicle information 113 and the audio information 116 can be determined, for example, by analyzing the line of sight and behavior of the occupants based on the in-vehicle information 113, or by the presence or absence of conversation among the occupants and the content of that conversation based on the audio information 116, and are expressed as two values, 0 (not applicable) and 1 (applicable), or numerical values ​​between them (for example, 11 levels in increments of 0.1). Note that AI may be used for this determination.

[0090] Biometric information 115 includes respiratory rate and heart rate. It is known that a person's respiratory rate and heart rate indicate the states shown in Table 1. Based on this knowledge, dozing (Vx), absentmindedness (Vy), and inattentiveness (Vz) are given numerical values ​​in parentheses in Table 1 (Vx, Vy, Vz from left to right).

[0091] [Table 1]

[0092] In Table 1, if the occupant's specific value for N (standard heart rate) is known, that value can be used; if it is unknown, a standard value of N=65, for example, can be used. Furthermore, if the occupant's age is unknown, 40 years old can be used as a representative value. Regarding the respiratory rate, Table 1 is created based on an average value of 18, but if the occupant's specific value is known, that value can also be used for determination. The occupant's specific value can be registered in the database 109 of the monitoring unit 10, for example, and referenced.

[0093] If the biological information 115 does not belong to any of the categories in Table 1, it is considered to be an abnormal situation, and is therefore determined to be a health abnormality regardless of the result of the above formula 3. In this case, for example, Vx=Vy=Vz=1 can be set.

[0094] The weighting variables a1 to a3 and b1 to b3 and the threshold values ​​c1, c2, and c3 of the evaluation functions Fx, Fy, and Fz in the above formula 3 are determined by the optimization means 202.

[0095] For example, in the case where a1 to a3, b1 to b3, and c1 to c3 are all 0.3 as standard settings, if a health abnormality is determined based on information from biological information 115, it is advisable to set a1 to a3 = 0 and b1 to b3 = 1 to ensure that a health abnormality is determined. The same applies when it is impossible to determine whether the driver is dozing (Dx), absent-minded (Dy), or looking aside (Dz) based on in-vehicle information 113 and audio information 116. The values ​​of a1 to a3, b1 to b3, and c1 to c3 may also be adjusted for each occupant.

[0096] By providing an evaluation function that combines multiple pieces of monitoring information in this way, the accuracy of recognizing the occupant's condition can be improved. In addition, since it has the same configuration as the health condition detection, the processing mechanism can be shared, which leads to power saving and cost reduction of the vehicle safety support system.

[0097] (Identifying suspicious persons) The vehicle safety support system 1 can recognize whether a person approaching the vehicle X from outside the vehicle is a suspicious person from the outside vehicle information 114. That is, the recognition information 20a includes suspicious person identification information 207 that identifies whether the person is a suspicious person from the outside vehicle information 114.

[0098] Whether or not a person approaching vehicle X is a suspicious person can be recognized as follows. First, an object approaching vehicle X is detected by ultrasonic sensor 104. If the distance between the approaching object and vehicle X is equal to or less than a specified value, for example, 10 m, the suspicious person is identified by exterior camera 105 and exterior radar 106. The suspicious person can be identified in the same way as by occupant identification means 110.

[0099] <Control unit> The control unit 30 is realized by, for example, a microcontroller. The control unit 30 also has an interface unit with the vehicle X and a communication unit that communicates with other devices outside the vehicle, such as a mobile phone or a cloud server. Note that known communication means can be used for communication with other devices. An example of such communication means is a CAN interface that allows communication without a host computer.

[0100] The interface unit is configured to be able to interface with all or part of, for example, the in-vehicle display, horn, lights, door locking mechanism, etc. This makes it possible to control the vehicle X to sound its horn or flash its lights when, for example, an approaching suspicious person is detected based on the suspicious person identification information 207.

[0101] The communication unit is configured to be able to communicate with the user's mobile phone and a cloud server via a wireless network, so that, for example, when a lost item is detected, a message to that effect can be sent to the user's mobile phone.

[0102] The control unit 30 operates based on the recognition information 20a. Below, we will explain the operation of the control unit 30 corresponding to the detection of left-behind items, detection of health conditions, and identification of suspicious individuals, which are the recognition information 20a recognized by the above-mentioned recognition unit 20. Note that the operation of the control unit 30 described below is just an example, and other operations may also be adopted.

[0103] (Detection of lost items) When the disembarking information 205 identifies that all of the occupants have disembarked from vehicle X, and the carried object detection information 204 identifies that the carried object is inside vehicle X, the carried object can be determined to be a lost item. In this case, the control unit 30 notifies the occupants that the carried object is inside vehicle X. By the control unit 30 notifying the occupants that the carried object is inside vehicle X, it is possible to prevent the occupants from leaving the object behind.

[0104] (Health status detection) If the health information 206 indicates that the driver is dozing, absent-minded, or looking away, the control unit 30 issues a warning to the driver by message or voice, for example, via an in-vehicle display or horn, thereby drawing the driver's attention and preventing accidents from occurring.

[0105] (Identifying suspicious persons) To identify a suspicious person, the control unit 30 uses the suspicious person identification information 207. When the person approaching vehicle X is identified as a suspicious person by the suspicious person identification information 207, the communication unit may notify the owner of vehicle X and may sound a horn or turn on a lamp to intimidate the suspicious person. By using the suspicious person identification information 207 in this way, damage to vehicle X or items inside vehicle X can be prevented.

[0106] Conversely, if the person approaching vehicle X is not suspicious, the door may be unlocked or a welcome message may be displayed, and if the occupant is the driver, the system may have the function of adjusting the seat position, height, mirror angle, etc. to suit the driver.

[0107] The control unit 30 may also have an environment adaptation processing unit that performs control in accordance with environmental conditions. The environment adaptation processing unit controls the operation modes of the surveillance camera 102, surveillance radar 103, etc., and performs waveform shaping of digital signals, etc., based on the environmental conditions, for example.

[0108] The setting control of the above operation mode involves changing the exposure based on, for example, the ambient brightness (luminosity 112). If the vehicle speed or the falling speed of raindrops during rainfall can be used, the shutter speed may be increased in proportion to this. Furthermore, if the temperature can be observed, the color temperature of the surveillance camera 102 may be set based on the ambient temperature. These are controlled appropriately based on data available within the vehicle X. Note that the vehicle speed and ambient temperature are information that is generally installed in the vehicle X, and therefore this information can be used. Furthermore, the falling speed of raindrops can be known, for example, by analyzing images captured by the external vehicle camera 105.

[0109] The waveform shaping includes pre-processing including image edge emphasis, white balance, dynamic range expansion, compression / expansion, S / N improvement, and interface settings for mutual cooperation between various devices.

[0110] <Method for controlling a vehicle safety support system> As shown in FIG. 5, the control method for the vehicle safety support system 1 includes a vehicle exterior monitoring step S1, a vehicle interior monitoring step S2, and a left-behind item detection step S3.

[0111] (External monitoring process) In the vehicle exterior monitoring step S1, a suspicious person approaching the vehicle X from outside the vehicle is monitored.

[0112] This outside-vehicle monitoring process S1 is started when the vehicle X stops and the outside-vehicle mode is set after the engine is stopped. The transition to the outside-vehicle mode may be set by a passenger in the vehicle X, such as the driver, or may be set automatically when the engine is stopped.

[0113] When the vehicle exterior monitoring mode is set, the ultrasonic sensor 104 is turned on to detect an object approaching the vehicle X. When the ultrasonic sensor 104 detects an approaching object, the recognition unit 20 recognizes whether the person is a suspicious person as described above, and includes suspicious person identification information 207 in the recognition information 20a.

[0114] If the suspicious person identification information 207 identifies a person approaching vehicle X as a suspicious person, the control unit 30 will, for example, notify the owner of vehicle X via the communication unit and scare off the suspicious person by sounding the horn or turning on the lamps. Conversely, if the person approaching vehicle X is not a suspicious person, the control unit 30 may determine that the occupant has arrived at vehicle X when the distance between the person (occupant) and vehicle X is within a specified value, for example, 1 meter, and may unlock the doors and display a welcome message, and if the occupant is the driver, adjust the seat position, height, mirror angle, etc. to suit the driver.

[0115] Furthermore, if it is determined based on information from the monitoring camera 102 or the like that all passengers have boarded the vehicle, the vehicle exterior monitoring process S1 is terminated and the process shifts to the vehicle interior monitoring mode (vehicle interior monitoring process S2).

[0116] (In-vehicle monitoring process) In the in-vehicle monitoring step S2, the state of the occupants, particularly the driver, is monitored.

[0117] In this in-vehicle monitoring step S2, the monitoring camera 102, monitoring radar 103, vital sensor 107, and microphone 108 are turned on to monitor the state of the occupants. Specifically, as described above, the monitoring camera 102 and monitoring radar 103 are used to monitor the behavior of the occupants in the vehicle X, particularly the behavior of the driver, and changes in the health state of the occupants are detected based on changes in the posture and complexion of the occupants obtained from the monitoring camera 102 and the monitoring radar 103, and information from the vital sensor 107 and microphone 108. At this time, it is advisable to store in advance in a database 109, for example, the individual's normal attributes such as heart rate and complexion, in order to detect abnormalities.

[0118] When an abnormality is detected, a notification is sent to a predetermined location using the communication unit of the control unit 30, audio and video are distributed, and control information for vehicle equipment is transmitted.

[0119] In this step, the system also checks whether all passengers have disembarked, and if it is confirmed that all passengers have disembarked, it moves on to the lost item detection step S3.

[0120] (Lost item detection process) In the left-behind item detection step S3, it is detected whether or not a specific item is present in the vehicle X.

[0121] The above-mentioned belongings include, for example, infants, pets, bags, mobile phones, etc. registered in the database 109. As described above, the recognition unit 20 detects lost items, and if it determines that an item has been left behind, it notifies a predetermined terminal via the communication unit of the control unit 30. If no lost item is detected, the lost item detection process S3 ends after a specified time, for example, 10 minutes, and the process moves to the exterior monitoring process S1. Note that if the process automatically moves to the exterior monitoring process S1 using engine stop as a trigger, the exterior monitoring process S1 may start before the end of the lost item detection process S3, and both processes may proceed simultaneously. In this case, the lost item detection process S3 ends, and the exterior monitoring process S1 continues.

[0122] <Advantages> The vehicle safety support system 1 monitors the occupants in the vehicle X using a surveillance camera 102, recognizes their behavior using a recognition unit 20, and controls the operation of the vehicle X or notifies the occupants using a control unit 30, thereby ensuring the safe driving of the vehicle X.

[0123] [Other embodiments] The above-described embodiments do not limit the configuration of the present invention. Therefore, the above-described embodiments may include omissions, substitutions, or additions of components based on the description in this specification and common general technical knowledge, and all of these should be construed as falling within the scope of the present invention.

[0124] In the above embodiment, the monitoring unit includes a photometer, a monitoring camera, a monitoring radar, an ultrasonic sensor, an exterior camera, an exterior radar, a vital sign sensor, a microphone, a means for identifying personal belongings, a means for identifying an occupant, and a database. However, some or all of the sensors and means other than the monitoring camera may be omitted. In a vehicle safety support system that does not use monitoring information obtained from these means, these may be omitted as appropriate.

[0125] The monitoring unit may also include other monitoring means. Examples of such monitoring means include a radio wave detector that detects radio waves from a mobile phone and a GPS device that acquires location information of the mobile phone. By including a radio wave detector and / or a GPS device in the monitoring unit, it is possible to easily identify the presence of a mobile phone in the vehicle, and therefore, if the mobile phone is among the personal belongings that may be brought into the vehicle, it can be more reliably detected as a lost item. In this case, the optimization means of the recognition unit may use the radio wave intensity of the radio wave detector and / or the GPS information of the GPS device.

[0126] In the above embodiment, the monitoring information includes occupant information identified by the occupant identification means, but the occupant information is not an essential component and can be omitted. In other words, the present invention also contemplates a configuration in which only the degree of coincidence calculated by the occupant identification means is used. [Industrial Applicability]

[0127] As described above, the vehicle safety support system of the present invention is capable of grasping the state of the occupants, and therefore, by using the vehicle safety support system, it is possible to safely and reliably monitor the driver and occupants continuously from the time the vehicle is in motion until they get out of the vehicle. [Explanation of symbols]

[0128] 1. Vehicle safety support systems 10 Monitoring Department 10a Monitoring Information 20 Recognition part 20a Identification Information 30 Control Unit 101 Photometer 102 Surveillance Camera 102a Monitoring area 103 Surveillance Radar 103a Monitoring area 104 Ultrasonic Sensor 104a Monitoring area 105 Exterior camera 105a Monitoring area 106 External Radar 106a Monitoring area 107 Vital Sensor 108 Microphone 109 Database 110 Occupant Identification Means 111 Means of identifying belongings 112 Luminosity 113 In-car information 114 Outside vehicle information 115 Biometric Information 116 Audio Information 117 Crew Information 118 Concordance 119 Carrying Information 201 Quantification Methods 202 Optimization Methods 203 Recognition information identification means 204 Carrying Item Detection Information 205 Disembarking Information 206 Health Information 207 Suspicious Person Identification Information X vehicle

Claims

1. A vehicle safety support system that supports the safety of one or more occupants of a vehicle, a monitoring unit capable of monitoring occupants in the vehicle; a recognition unit that recognizes the state of the occupant based on monitoring information from the monitoring unit; a control unit that controls the operation of the vehicle or notifies the occupant based on the recognition information from the recognition unit; Equipped with the monitoring unit has a monitoring camera, a monitoring radar, and a personal item identification means for identifying personal items that may be brought into the vehicle; The surveillance camera and the surveillance radar are used to identify the carried items, An evaluation function combining camera data from a surveillance camera and radar data from a surveillance radar is used to identify the carried item, A vehicle safety assistance system, wherein the radar data is periodic vibrations detected by the surveillance radar.

2. 2. The vehicle safety support system according to claim 1, wherein the identification of the belongings is identification of a child left behind.

Images