Vehicle interior monitoring system and program

Abstract

Provided is a vehicle interior monitoring system capable of improving the accuracy of information obtained from the inside of a vehicle interior. A vehicle interior monitoring system (1) has: a vehicle interior information acquisition unit (10) that acquires a plurality of pieces of information in a vehicle interior of a vehicle; a reliability calculation unit (15) that calculates the reliability of each of the plurality of pieces of information; and an information processing unit (18) that determines, on the basis of the reliability, which piece of information among the plurality of pieces of information is to be used as basic information, which is information serving as a basis for monitoring the vehicle interior. The vehicle interior information acquisition unit (10) includes an image information acquisition unit (11) that acquires visible light image information and infrared ray image information in the vehicle interior. The information processing unit (18) determines that the infrared ray image information is to be used as the basic information when the reliability of the infrared ray image information is equal to or greater than a predetermined value, and determines that the visible light image information is to be used as the basic information when the reliability of the infrared ray image information is less than the predetermined value.

Description

In-vehicle monitoring system and program

[0001] The present invention relates to an in-vehicle monitoring system and a program for monitoring the interior of a vehicle.

[0002] Conventionally, there is a technology for monitoring a driver in the interior of a vehicle (for example, Patent Document 1). In Patent Document 1, when monitoring a driver in the interior of a vehicle, the state of the driver is recognized by analyzing image information acquired from a camera. Specifically, a visible light camera for acquiring a visible light image and a near-infrared camera for acquiring a near-infrared image are arranged. When the vehicle is in a stopped state, the visible light camera is turned on and the near-infrared camera is turned off. On the other hand, when the vehicle is in a running state, the visible light camera is turned off and the near-infrared camera is turned on. Thus, depending on whether the vehicle is in a running state or a stopped state, the power supply of the visible light camera is turned on or the power supply of the near-infrared camera is turned on is switched.

[0003] However, although the technology of Patent Document 1 is simple control, in order to obtain biometric information of a driver during driving, there are cases where more diverse information can be obtained from a visible light image than from a near-infrared image, and there are also cases where a clear visible light image cannot be obtained during parking or at night. For this reason, there was a problem that it was insufficient for obtaining information in the vehicle interior with higher accuracy.

[0004] Japanese Unexamined Patent Application Publication No. 2020-152133

[0005] An object of the present invention is to provide an in-vehicle monitoring system capable of improving the accuracy of information obtained from the interior of a vehicle.

[0006] A first aspect of the present invention is an in-vehicle monitoring system comprising: an in-vehicle information acquisition unit that acquires a plurality of pieces of information inside the vehicle interior; a reliability calculation unit that calculates the reliability of each of the plurality of pieces of information; and an information processing unit that determines, based on the reliability, which of the plurality of pieces of information will be used as basic information, which is the basis for in-vehicle monitoring, wherein the in-vehicle information acquisition unit includes an image information acquisition unit that acquires visible light image information and infrared image information inside the vehicle interior, and the information processing unit determines that the infrared image information will be used as the basic information when the reliability of the infrared image information is equal to or greater than a predetermined value, and determines that the visible light image information will be used as the basic information when the reliability of the infrared image information is less than a predetermined value.

[0007] According to a first aspect of the present invention, when determining the basic information for in-vehicle monitoring, the basic information can be determined based on the reliability of multiple pieces of information acquired in the vehicle interior, thereby enabling the use of highly accurate information as the basic information. Furthermore, when acquiring image information, if the reliability of the infrared image information is determined to be above a predetermined value, the infrared image information can be used as the basic information. If the reliability of the infrared image information is below a predetermined value, the visible light image information can be used as the basic information. In this way, infrared image information can be acquired with priority over visible light image information. In in-vehicle monitoring, infrared image information forms the basis of more stable information than visible light image information. Therefore, if both the reliability of the infrared image information and the reliability of the visible light image information are high, it is preferable to prioritize the infrared image information. As a result, an in-vehicle monitoring system that can improve the accuracy of information obtained from inside the vehicle can be provided.

[0008] A second aspect of the present invention is a vehicle interior monitoring system according to the first aspect, characterized in that the vehicle interior information acquisition unit includes a biometric information acquisition unit that acquires biometric information, and the information processing unit determines to use the biometric information as basic information when the reliability of the biometric information is equal to or greater than a predetermined value.

[0009] According to a second aspect of the present invention, biometric information is acquired inside the vehicle, and if the reliability of the biometric information is above a predetermined value, the biometric information is used as basic information. This makes it possible to monitor the inside of the vehicle based on biometric information of the driver and others, which is useful information inside the vehicle.

[0010] A third aspect of the present invention is the in-vehicle monitoring system according to the second aspect, characterized in that the information processing unit puts the image information acquisition unit into a sleep state when the reliability of the biological information is less than a predetermined value.

[0011] According to a third aspect of the present invention, the image information acquisition unit is put into a sleep state when the reliability of the biological information is below a predetermined value. This allows the power supply to the image information acquisition unit to be stopped or suppressed by putting it into a sleep state when, for example, biological information is obtained based on image information. This prevents false detections and provides an energy-saving effect.

[0012] A fourth aspect of the present invention is the in-vehicle monitoring system according to the second aspect, wherein the in-vehicle information acquisition unit further comprises a non-image information acquisition unit that acquires information other than image information, and the biological information acquisition unit analyzes the information obtained from the image information acquisition unit and the non-image information acquisition unit to obtain the biological information.

[0013] According to a fourth aspect of the present invention, biological information can be obtained not only from image information but also from information other than images. Therefore, biological information can be obtained even when the reliability of the image information is low.

[0014] A fifth aspect of the present invention is the in-vehicle monitoring system according to the fourth aspect, characterized in that the biological information includes at least one of the following: gaze information, facial expression information, skeletal information, blood pressure information, heart rate information, and respiration information.

[0015] According to a fifth aspect of the present invention, the biological information includes at least one of the following: gaze information, facial expression information, skeletal information, blood pressure information, heart rate information, and respiratory information. This makes it possible to obtain various types of biological information from the driver and ensure the safety of the vehicle's operation by monitoring the interior of the vehicle based on this information.

[0016] A sixth aspect of the present invention is a vehicle interior monitoring system according to the first aspect, wherein the vehicle interior information acquisition unit includes an object information acquisition unit that acquires object information which is information about objects in the vehicle interior, and the object information acquisition unit obtains the object information by analyzing image information obtained from the image information acquisition unit.

[0017] According to a sixth aspect of the present invention, object information, which is information about objects including living organisms inside the vehicle, is acquired by analyzing image information obtained from an image information acquisition unit. This makes it possible to determine whether items have been left behind or abandoned inside the vehicle (children, pets, etc.).

[0018] A seventh aspect of the present invention is a vehicle interior monitoring system according to the first aspect, further comprising a vehicle information acquisition unit that acquires vehicle information which is information relating to the vehicle, wherein the basic information includes, as a type of basic information, at least biological information in the vehicle interior and object information in the vehicle interior, and the information processing unit determines, based on the vehicle information, whether to use the biological information or the object information as the basic information.

[0019] According to a seventh aspect of the present invention, the system determines whether the basic information should be biometric information or object information based on vehicle information. This enables in-vehicle monitoring that is appropriate to the vehicle's condition.

[0020] An eighth aspect of the present invention is the in-vehicle monitoring system according to the first aspect, characterized in that the information processing unit updates the basic information at predetermined intervals.

[0021] According to the eighth aspect of the present invention, basic information is updated at predetermined intervals. Therefore, even if the situation inside the vehicle changes from the situation at the start of monitoring, the situation inside the vehicle can be understood.

[0022] A ninth aspect of the present invention is an in-vehicle monitoring system according to the first aspect, further comprising an external information acquisition unit that acquires external information which is information obtained from outside the vehicle, and the reliability calculation unit calculates the reliability by referring to the external information.

[0023] According to a ninth aspect of the present invention, external information obtained from outside the vehicle is acquired, and the reliability is calculated by referring to the external information. Therefore, since the reliability of image information, etc., can be determined according to the external information, more appropriate basic information can be selected.

[0024] A tenth aspect of the present invention is the in-vehicle monitoring system according to the ninth aspect, characterized in that the external information includes one of the following: health information relating to a living organism in the vehicle interior, weather information in the vicinity of the vehicle, and traffic congestion information related to the vehicle.

[0025] According to a tenth aspect of the present invention, the external information includes one of the following: health information relating to a living organism inside the vehicle, weather information around the vehicle, or traffic congestion information related to the vehicle. As a result, for example, the accuracy of the biological information is improved by referring to the health information, the accuracy of selecting which image information to use is improved by referring to the weather information, and the accuracy of the vehicle information is improved by referring to the traffic congestion information. Thus, the accuracy of in-vehicle monitoring is improved.

[0026] An eleventh aspect of the present invention is the in-vehicle monitoring system according to the first aspect, characterized in that the image information acquisition unit obtains image information from an imaging device that has pixels for acquiring visible light and pixels for acquiring infrared light in a single image sensor.

[0027] According to an eleventh aspect of the present invention, the image information acquisition unit obtains image information from an imaging device that has pixels for acquiring visible light and pixels for acquiring infrared light on a single image sensor. As a result, visible light images and infrared images can be acquired with a single device. Consequently, the size of the image information acquisition unit can be reduced compared to the case where means for acquiring visible light images and means for acquiring infrared images are arranged separately.

[0028] A twelfth aspect of the present invention is a program that causes a computer to execute a vehicle interior monitoring method, wherein the vehicle interior monitoring method comprises: a vehicle interior information acquisition step of acquiring a plurality of pieces of information in the vehicle interior; a reliability calculation step of calculating the reliability of each of the plurality of pieces of information; and an information processing step of determining, based on the reliability, which of the plurality of pieces of information will be used as basic information, which is the information that forms the basis for vehicle interior monitoring, wherein the vehicle interior information acquisition step includes an image information acquisition step of acquiring visible light image information and infrared image information in the vehicle interior, and the information processing step is characterized in that, if the reliability of the infrared image information is greater than or equal to a predetermined value, the infrared image information is determined to be the basic information, and if the reliability of the infrared image information is less than a predetermined value, the visible light image information is determined to be the basic information.

[0029] According to a twelfth aspect of the present invention, the same operation as in the first aspect can be introduced into the in-vehicle monitoring system.

[0030] According to the present invention, it is possible to provide an in-vehicle monitoring system that can improve the accuracy of information obtained from inside the vehicle.

[0031] This is a conceptual diagram showing an example of the hardware configuration of an in-vehicle monitoring system. This is a block diagram showing an example of the software configuration of an in-vehicle monitoring system. This is a diagram showing an example of the relationships between each information acquisition unit. This is a flowchart of the entire in-vehicle monitoring process. This is a flowchart of object recognition when boarding. This is a flowchart of biometric recognition during operation. This is a flowchart of object recognition when stopped. This is a flowchart of object recognition when disembarking.

[0032] Hereinafter, embodiments of the vehicle interior monitoring system of the present invention will be described with reference to the drawings. Figure 1 is a conceptual diagram showing an example of the hardware configuration of the vehicle interior monitoring system 1. The main body of the vehicle interior monitoring system 1 is positioned close to the mirror M installed in the vehicle interior of the vehicle 2. In this embodiment, a configuration in which the main body is placed in the space behind the rear-view mirror in the vehicle interior is described as an example. In this embodiment, the mirror M is a mirror, but it may be any device that functions as a rear-view mirror, and may be a display or the like that allows the driver to see the situation behind the vehicle.

[0033] The in-vehicle monitoring system 1 is controlled by a control unit 5 (control device). The control unit 5 is a computer or a device containing a computer, and has a CPU, ROM, and RAM (not shown). The CPU reads the program stored in the ROM into the RAM and executes the processing. The arrangement of the control unit 5 is not particularly limited, but for example, the control unit 5 may be arranged integrally with the mirror M, or it may be arranged in a location other than the mirror M in the vehicle interior.

[0034] In the example shown in Figure 1, the vehicle 2 is equipped with an imaging device C and a bio-information detection radar R integrated with a mirror M. The mirror M is positioned to observe the entire interior of the vehicle and is opposite the driver's seat, allowing for a clear view of the interior and the driver, making it a suitable location for the imaging device C and bio-information detection radar R. The imaging device C in this embodiment has pixels for acquiring visible light and pixels for acquiring infrared light in a single image sensor, enabling the acquisition of visible light image information and infrared image information within the vehicle interior. Alternatively, the imaging device C may have pixels for acquiring visible light and pixels for acquiring near-infrared light in a single image sensor, enabling the acquisition of visible light image information and near-infrared image information within the vehicle interior. The imaging device C can be switched to acquire either visible light image information or infrared image information depending on the situation. The bio-information detection radar R acquires heart rate information and other data of living organisms within the vehicle interior.

[0035] Image information obtained by the imaging device C and heart rate information obtained by the biometric information detection radar R are transmitted to the control unit 5. In addition, the vehicle 2 is equipped with information acquisition means such as sensors and devices that acquire information to be transmitted to the control unit 5. This information acquisition means includes a vehicle information acquisition unit 6 that acquires vehicle information, which is information related to the vehicle 2, and an external information acquisition unit 7 that acquires external information, which is information obtained from outside the vehicle. The vehicle information acquisition unit 6 and the external information acquisition unit 7 transmit the acquired information to the control unit 5.

[0036] The vehicle information acquisition unit 6 obtains vehicle information relating to the state of the vehicle 2 from various devices installed in the vehicle 2. Examples of the vehicle information acquisition unit 6 are shown below, but are not limited to these. The gear information sensor acquires gear information, indicating which gear the vehicle is in due to a shift change by the shift lever S. The steering sensor acquires steering angle information of the steering wheel WL. The vehicle speed sensor acquires vehicle speed information. The lighting information sensor acquires on / off information for headlights, interior lights, etc. The wiper sensor acquires operation information for the wipers WP. The seating sensor acquires seating information, such as whether a person is sitting in the seat. In addition, engine and motor drive information may also be acquired.

[0037] The external information acquisition unit 7 obtains external information other than that of the vehicle 2 from various devices other than the vehicle 2. Examples of the external information acquisition unit 7 and external information are shown below, but are not limited to these. The GPS and internet server work together to acquire navigation information containing the location information and surrounding information of the vehicle 2. Wearable sensors such as smartwatches acquire the current health information of the occupants. Note that the health information may be past data stored on the server. In addition, the external information acquisition unit 7 may acquire weather information and traffic congestion information via the internet. Note that this external information is displayed on the in-vehicle display D as needed. In addition, the external information acquisition unit 7 can also acquire time information. The clarity of image information, which will be described later, can be inferred from the time information and navigation information. Furthermore, the reliability of biometric information (described later) can be determined from wearable sensors and the driver's health information.

[0038] Figure 2 is a block diagram showing an example of the software configuration of the in-vehicle monitoring system 1. The in-vehicle monitoring system 1 includes an in-vehicle information acquisition unit 10 that acquires multiple pieces of information from inside the vehicle 2, a reliability calculation unit 15 that calculates a reliability score indicating the certainty of each of the multiple pieces of information, and an information processing unit 18 that determines, based on the reliability score, which of the multiple pieces of information will be used as basic information, which is the basis for in-vehicle monitoring. The process of acquiring information by the in-vehicle information acquisition unit 10 is the in-vehicle information acquisition process. The process of calculating the reliability score by the reliability calculation unit 15 is the reliability calculation process. The process of determining the basic information by the information processing unit 18 is the information processing process.

[0039] The vehicle interior information acquisition unit 10 includes an image information acquisition unit 11 that acquires visible light image information and infrared image information from an imaging device C such as a camera inside the vehicle interior. The image information acquisition unit 11 includes a visible light image information acquisition unit 11a and an infrared image information acquisition unit 11b. Therefore, the image information acquisition unit 11 acquires at least multiple pieces of image information (image information acquisition step).

[0040] The information processing unit 18 decides to use infrared image information as the base information if the reliability of the infrared image information is above a predetermined value. On the other hand, if the reliability of the infrared image information is below a predetermined value, it decides to use visible light image information as the base information. The information processing unit 18 updates the base information at predetermined intervals. The update interval may be set by the information processing unit 18 as appropriate. For example, the update interval may be different depending on the type of information to be acquired, such as updating every few tens of seconds when acquiring infrared image information and every few minutes when acquiring visible light image information.

[0041] Basic information refers to the fundamental information used in in-vehicle monitoring. For example, the basic information used to calculate the reliability of image information is the image information itself, and the basic information used to calculate the reliability of biometric information, as described later, is the biometric information itself. Furthermore, the information processing unit 18 can put the image information acquisition unit 11 into sleep mode if the reliability of the biometric information is below a predetermined value. Sleep mode is a state in which the information processing unit 18 has decided not to acquire image information from the image information acquisition unit 11, and this includes turning off the power to the imaging device C or changing the imaging device C to power-saving mode. This results in power saving.

[0042] The method used by the reliability calculation unit 15 to calculate reliability is not limited to any particular method; a predetermined calculation method may be incorporated into the program beforehand, or the calculation method may be determined using deep learning. As a result, the reliability may be higher for images that are considered clear, lower for images with overexposure or underexposure, lower for images where a predetermined point in the image is not detected, or lower for images where the driver or other person is wearing infrared-blocking sunglasses.

[0043] The vehicle interior information acquisition unit 10 includes a biometric information acquisition unit 13 that acquires biometric information. The information processing unit 18 decides to use the biometric information as basic information when the reliability of the biometric information is above a predetermined value. The vehicle interior information acquisition unit 10 also has a non-image information acquisition unit 12 that acquires information other than image information. The biometric information acquisition unit 13 obtains biometric information by analyzing the information obtained from the image information acquisition unit 11 and the non-image information acquisition unit 12. In this embodiment, the information acquired by the non-image information acquisition unit 12 is heart rate information acquired from the biometric information detection radar R. From the heart rate information, the driver's drowsiness can be calculated, and if the drowsiness exceeds a predetermined value, the information processing unit 18 can take measures such as changing the control method. The non-image information acquisition unit 12 may acquire information other than heart rate information.

[0044] The passenger compartment information acquisition unit 10 includes an object information acquisition unit 14 that acquires object information, which is information on objects including living bodies in the passenger compartment. The object information acquisition unit 14 analyzes the information obtained from the image information acquisition unit 11 to obtain object information. The object information acquisition unit 14 is used for detecting forgotten items and abandoned items, which will be described later.

[0045] The passenger compartment monitoring system 1 further has a vehicle information acquisition unit 16 that acquires vehicle information, which is information related to the vehicle 2. The basic information includes at least biometric information and object information in the passenger compartment as types of basic information. The information processing unit 18 determines whether to use either biometric information or object information as the basic information based on the vehicle information. For example, the vehicle information acquisition unit 16 acquires information on which gear (parking gear, drive gear, neutral gear, etc.) is selected from the gear information sensor. Also, the vehicle information acquisition unit 16 acquires information on whether the vehicle 2 is in a moving state or a stopped state from the vehicle speed sensor.

[0046] The passenger compartment monitoring system 1 further has an external information acquisition unit 17 that acquires external information, which is information obtained from outside the vehicle 2. The reliability calculation unit 15 calculates the reliability by referring to the external information. The external information includes any one of health information related to a living body (such as a driver) in the passenger compartment, weather information around the vehicle 2, and traffic jam information related to the vehicle 2.

[0047] The biometric information obtained from the biometric information acquisition unit 13 has at least any one of gaze information, facial expression information, skeletal information, blood pressure information, heart rate information, and respiratory information. The gaze information is information on the gaze of a driver or the like, and can be used to grasp where the attention of a driver or the like is. The facial expression information is information on the facial expression of a driver or the like, and can be used to determine whether the facial expression is different from normal (for example, a sleepy facial expression). The skeletal information can be used to grasp the posture of a driver or the like. From the blood pressure information, heart rate information, and respiratory information, the state of the driver's blood pressure, heart rate, and respiration can be mainly grasped.

[0048] Figure 3 is a diagram showing an example of the relationship between each information acquisition unit. The image information acquisition unit 11, the non-image information acquisition unit 12, the biological information acquisition unit 13, and the object information acquisition unit 14 are related as shown in Figure 3. The biological information acquisition unit 13 includes a line-of-sight information acquisition unit 13a, a facial expression information acquisition unit 13b, a skeleton information acquisition unit 13c, a blood pressure information acquisition unit 13d, and a heartbeat / respiration information acquisition unit 13e. The information of the visible light image information acquisition unit 11a serves as the basis for the line-of-sight information acquisition unit 13a, the facial expression information acquisition unit 13b, the skeleton information acquisition unit 13c, the blood pressure information acquisition unit 13d, and the object information acquisition unit 14. That is, by analyzing the visible light image information, the above-mentioned respective information can be obtained. Also, the information of the infrared image information acquisition unit 11b serves as the basis for the line-of-sight information acquisition unit 13a, the facial expression information acquisition unit 13b, the skeleton information acquisition unit 13c, and the object information acquisition unit 14. That is, by analyzing the infrared image information, the above-mentioned respective information can be obtained. Further, the information of the non-image information acquisition unit 12 is the basis for the heartbeat / respiration information acquisition unit 13e.

[0049] Figure 4 is a flowchart of the entire in-vehicle monitoring process. The process in Figure 4 is performed by the CPU of the control unit 5. When the in-vehicle monitoring process starts, in step S11, vehicle information is acquired. In this embodiment, based on the vehicle speed information, gear information, or other information obtained from the vehicle information acquisition unit 16, it is determined whether the driver is in a state of boarding the vehicle 2, whether the vehicle 2 is in operation, whether the vehicle 2 has stopped, or whether the passengers have disembarked from the vehicle 2.

[0050] In step S12, when it is determined that the driver is in a state of boarding the vehicle 2 (boarding time), in the process of step S20, object recognition at the time of boarding is performed. On the other hand, in step S12, when it is not determined as the boarding time, the process of step S13 is performed.

[0051] In step S13, when it is determined that the vehicle 2 is in operation, in the process of step S30, biological recognition during operation is performed. On the other hand, in step S13, when it is not determined as the operation time, the process of step S14 is performed.

[0052] If it is determined in step S14 that vehicle 2 has stopped (is stationary), then in step S40, object recognition is performed while the vehicle is stationary. On the other hand, if it is not determined in step S14 that the vehicle is stationary, then the process in step S15 is performed.

[0053] If it is determined in step S15 that a passenger has disembarked from vehicle 2 (disembarkation), then object recognition at the time of disembarkation is performed in step S50. On the other hand, if it is not determined in step S15 that a passenger has disembarked, then the process in step S16 is performed.

[0054] In step S16, it is determined whether or not to terminate the in-vehicle monitoring. The termination and start of in-vehicle monitoring are not particularly limited, but may be predetermined by time, detected by the opening and closing of the doors of vehicle 2, or performed by an operation on the in-vehicle monitoring system 1 by the driver or other person. If it is determined in step S16 that the in-vehicle monitoring should be terminated, the in-vehicle monitoring is terminated. On the other hand, if it is determined in step S16 that the in-vehicle monitoring should not be terminated, the process returns to step S11. Here, the time it takes to return from step S16 to step S11 may vary depending on the circumstances.

[0055] Figure 5 is a flowchart of object recognition during boarding. The process of object recognition during boarding (monitoring targets: personal belongings, number of passengers, etc.) corresponds to step S20 described above. The process of step S20 starts from step S201.

[0056] In step S201, the reliability calculation unit 15 calculates the reliability of the infrared image information acquired by the infrared image information acquisition unit 11b. If the reliability of the infrared image information is equal to or greater than a predetermined value, the process in step S202 is performed. In step S202, the infrared image information is analyzed by the object information acquisition unit 14, and object detection is performed in infrared mode (IR mode). On the other hand, if the reliability of the infrared image information in step S201 is not equal to or greater than a predetermined value, the process in step S203 is performed.

[0057] In step S203, the reliability calculation unit 15 calculates the reliability of the visible light image information acquired by the visible light image information acquisition unit 11a. If the reliability of the visible light image information is equal to or greater than a predetermined value, the process proceeds to step S204. In step S204, the visible light image information is analyzed by the object information acquisition unit 14, and object detection is performed in visible light mode (RGB mode). On the other hand, if the reliability of the visible light image information in step S203 is not equal to or greater than a predetermined value, the process in step S205 is performed. In step S205, it is decided not to perform object detection during boarding. After this, the process in step S16 in Figure 4 is performed.

[0058] Figure 6 is a flowchart of biometric recognition during operation. The process of biometric recognition during operation (monitoring target: driver, etc.) corresponds to step S30 described above. The process of step S30 starts from step S301.

[0059] In step S301, the reliability calculation unit 15 calculates the reliability of the infrared image information acquired by the infrared image information acquisition unit 11b. If the reliability of the infrared image information is above a predetermined value, the process in step S302 is performed. In step S302, the infrared image information is analyzed by the gaze information acquisition unit 13a and the facial expression information acquisition unit 13b, and face and gaze detection is performed in infrared mode. On the other hand, if the reliability of the face and gaze information is not above a predetermined value in step S301, the process in step S303 is performed. Even if the infrared image is clear, examples of cases where the reliability of the face and gaze information is not above a predetermined value include the following. For example, if the driver is wearing a hat or mask, the reliability of the facial expression information tends to decrease. Also, if the driver is wearing infrared-blocking sunglasses, the reliability of the gaze information tends to decrease.

[0060] In step S303, the reliability calculation unit 15 calculates the reliability of the visible light image information acquired by the visible light image information acquisition unit 11a. If the reliability of the visible light image information is above a predetermined value, the process in step S304 is performed. In step S304, the visible light image information is analyzed by the gaze information acquisition unit 13a and the facial expression information acquisition unit 13b, and face and gaze detection is performed in visible light mode. On the other hand, if the reliability of the face and gaze information is not above a predetermined value in step S303, the process in step S305 is performed. Even if the visible light image is clear, examples of cases where the reliability of the face and gaze information is not above a predetermined value include the following. For example, if the driver is wearing a hat or mask, the reliability of the facial expression information tends to decrease. Also, if the driver is wearing sunglasses, the reliability of the gaze information tends to decrease.

[0061] In step S305, the reliability calculation unit 15 calculates the reliability of the non-image information acquired by the non-image information acquisition unit 12. If the reliability of the non-image information is equal to or greater than a predetermined value, the process in step S306 is performed. In step S306, the non-image information is analyzed by the heart rate / respiratory information acquisition unit 13e, and heart rate / respiratory detection is performed in sleep mode. Here, sleep mode refers to a mode in which the image information acquisition unit 11 is put into a sleep state. On the other hand, if the reliability of the non-image information is not equal to or greater than a predetermined value in step S305, the process in step S307 is performed.

[0062] In step S307, the reliability calculation unit 15 again calculates the reliability of the infrared image information acquired by the infrared image information acquisition unit 11b. If the reliability of the infrared image information is equal to or greater than a predetermined value, the process in step S308 is performed. In step S308, the infrared image information is analyzed by the skeleton information acquisition unit 13c, and skeleton detection is performed in infrared mode. On the other hand, if the reliability of the infrared image information in step S307 is not equal to or greater than a predetermined value, the process in step S309 is performed.

[0063] In step S309, the reliability calculation unit 15 calculates the reliability of the visible light image information acquired by the visible light image information acquisition unit 11a. If the reliability of the visible light image information is equal to or greater than a predetermined value, the process in step S310 is performed. In step S310, the visible light image information is analyzed by the skeleton information acquisition unit 13c, and skeleton detection is performed in visible light mode. On the other hand, if the reliability of the visible light image information is not equal to or greater than a predetermined value in step S309, the process in step S311 is performed. In step S311, it is decided not to perform biological detection during operation. After this, the process in step S16 in Figure 4 is performed.

[0064] Figure 7 is a flowchart of object recognition while the vehicle is stopped. The process of object recognition while the vehicle is stopped (monitoring target: lost items, etc.) corresponds to step S40 described above. The process of step S40 starts from step S401.

[0065] In step S401, the reliability calculation unit 15 calculates the reliability of the infrared image information acquired by the infrared image information acquisition unit 11b. If the reliability of the infrared image information is equal to or greater than a predetermined value, the process in step S402 is performed. In step S402, the infrared image information is analyzed by the object information acquisition unit 14, and object detection is performed in infrared mode. On the other hand, if the reliability of the infrared image information in step S401 is not equal to or greater than a predetermined value, the process in step S403 is performed.

[0066] In step S403, the reliability calculation unit 15 calculates the reliability of the visible light image information acquired by the visible light image information acquisition unit 11a. If the reliability of the visible light image information is equal to or greater than a predetermined value, the process proceeds to step S404. In step S404, the visible light image information is analyzed by the object information acquisition unit 14, and object detection is performed in visible light mode. On the other hand, if the reliability of the visible light image information in step S403 is not equal to or greater than a predetermined value, the process in step S405 is performed. In step S405, it is decided not to perform object detection when the vehicle is stopped. After this, the process in step S16 in Figure 4 is performed.

[0067] Figure 8 is a flowchart of object recognition upon disembarking. The process of object recognition upon disembarking (monitoring target: abandoned object, etc.) corresponds to step S50 described above. The process of step S50 starts from step S501.

[0068] In step S501, the reliability calculation unit 15 calculates the reliability of the infrared image information acquired by the infrared image information acquisition unit 11b. If the reliability of the infrared image information is equal to or greater than a predetermined value, the process in step S502 is performed. In step S502, the infrared image information is analyzed by the object information acquisition unit 14, and object detection is performed in infrared mode. On the other hand, if the reliability of the infrared image information in step S501 is not equal to or greater than a predetermined value, the process in step S503 is performed.

[0069] In step S503, the reliability calculation unit 15 calculates the reliability of the visible light image information acquired by the visible light image information acquisition unit 11a. If the reliability of the visible light image information is equal to or greater than a predetermined value, the process proceeds to step S504. In step S504, the visible light image information is analyzed by the object information acquisition unit 14, and object detection is performed in visible light mode. On the other hand, if the reliability of the visible light image information in step S503 is not equal to or greater than a predetermined value, the process in step S505 is performed. In step S505, it is decided not to perform object detection when the vehicle is disembarked. After this, the process in step S16 in Figure 4 is performed.

[0070] As described above, in this embodiment, when determining the basic information for in-vehicle monitoring, the basic information is determined based on the reliability of multiple pieces of information acquired in the vehicle interior, thereby enabling the use of highly accurate information as basic information. Furthermore, when acquiring image information, if the reliability of the infrared image information is above a predetermined value, it is decided to use the infrared image information as basic information, and if the reliability of the infrared image information is below the predetermined value, it is decided to use the visible light image information as basic information. This allows infrared image information to be acquired with priority over visible light image information. In other words, in in-vehicle monitoring, infrared image information forms the basis of more stable information than visible light image information, so when both the reliability of the infrared image information and the reliability of the visible light image information are high, it is preferable to prioritize the infrared image information. As a result, an in-vehicle monitoring system that can improve the accuracy of information obtained from inside the vehicle can be provided.

[0071] Furthermore, in this embodiment, biometric information is acquired inside the vehicle, and if the reliability of the biometric information is above a predetermined value, the biometric information is used as basic information. This makes it possible to monitor the inside of the vehicle based on biometric information of the driver and other relevant individuals, which is useful information inside the vehicle.

[0072] Furthermore, in this embodiment, the image information acquisition unit 11 is put into sleep mode when the reliability of the biological information falls below a predetermined value. This allows the power supply to the imaging device C to be stopped or suppressed by putting the image information acquisition unit 11 into sleep mode when, for example, biological information is obtained based on image information. This prevents false detections and also provides an energy-saving effect.

[0073] Furthermore, in this embodiment, biological information can be obtained not only from image information obtained by the imaging device C, but also from information other than images obtained by the biological information detection radar R. Therefore, biological information can be obtained even when the reliability of the image information is low.

[0074] Furthermore, in this embodiment, the biometric information includes at least one of the following: gaze information, facial expression information, skeletal information, blood pressure information, heart rate information, and respiratory information. This allows, for example, the driver to provide diverse types of biometric information, and by monitoring the vehicle interior based on this information, the safety of driving the vehicle can be ensured.

[0075] Furthermore, in this embodiment, object information, which includes information about living objects inside the vehicle, is acquired by analyzing information obtained from the image information acquisition unit 11. This makes it possible to determine whether items have been left behind or abandoned inside the vehicle (children, pets, etc.).

[0076] Furthermore, in this embodiment, the system determines whether to use biometric information or object information as basic information based on vehicle information such as vehicle speed information and gear information. This enables in-vehicle monitoring that is appropriate to the vehicle's condition. In this embodiment, object detection is performed when boarding, stopping, and alighting to prevent forgotten or abandoned items. On the other hand, during operation, biometric detection of the driver or other persons is performed to ensure the safety of driving vehicle 2.

[0077] Furthermore, in this embodiment, basic information is updated at predetermined intervals. Therefore, even if the situation inside the vehicle changes from the situation at the start of monitoring, the situation inside the vehicle can be understood. In this embodiment, basic information is updated at predetermined intervals for situations during operation, stopping, and alighting, which differ from the situation at the start of vehicle monitoring (boarding). Therefore, basic information appropriate to the situation is selected. Note that the update time is not uniform; for example, it is preferable that the update time for operation and other situations differs from each other, and even during operation, it is preferable that the update time differs for each process.

[0078] Furthermore, in this embodiment, external information obtained from outside the vehicle 2 is acquired, and the reliability is calculated by referring to the external information. Therefore, since the reliability of image information, etc., can be determined according to the external information, more appropriate basic information can be selected.

[0079] Furthermore, in this embodiment, the external information includes one of the following: health information relating to the body inside the vehicle, weather information around the vehicle 2, or traffic congestion information related to the vehicle 2. This allows, for example, the accuracy of the biometric information to be improved by referring to health information, the accuracy of selecting which image information to use to be improved by referring to weather information, and the accuracy of the vehicle information to be improved by referring to traffic congestion information. For example, by analyzing the weather information, the brightness around the vehicle 2, and further analyzing the brightness inside the vehicle, the reliability of the biometric information obtained from the image information can be determined.

[0080] Furthermore, in this embodiment, the image information acquisition unit 11 obtains image information from an imaging device C that has pixels for acquiring visible light and pixels for acquiring infrared light on a single image sensor. As a result, visible light images and infrared images can be acquired with a single device. Consequently, the size of the image information acquisition unit 11 can be reduced compared to the case where means for acquiring visible light images and means for acquiring infrared images are arranged separately.

[0081] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. Furthermore, parts of the embodiments described above may be deleted or combined as appropriate. In addition, each process shown in the flowchart of the embodiments described above can be made into a program that can be implemented on a computer, into a recording medium on which the program is stored, or into a computer program product.

[0082] This application claims priority based on Japanese Patent Application No. 2024-166632, filed on September 25, 2024, and incorporates all the contents of said Japanese Patent Application.

[0083] 1...In-vehicle monitoring system, 5...Control unit, 10...In-vehicle information acquisition unit, 11...Image information acquisition unit, 12...Non-image information acquisition unit, 13...Biometric information acquisition unit, 14...Object information acquisition unit, 15...Reliability calculation unit, 16...Vehicle information acquisition unit, 17...External information acquisition unit, 18...Information processing unit, C...Imaging device, M...Mirror, R...Biometric information detection radar

Claims

1. An in-vehicle monitoring system comprising: an in-vehicle information acquisition unit that acquires multiple pieces of information inside the vehicle's interior; a reliability calculation unit that calculates the reliability of each of the multiple pieces of information; and an information processing unit that determines, based on the reliability, which of the multiple pieces of information will be used as basic information for in-vehicle monitoring, wherein the in-vehicle information acquisition unit includes an image information acquisition unit that acquires visible light image information and infrared image information inside the vehicle's interior; and the information processing unit determines to use the infrared image information as the basic information when the reliability of the infrared image information is above a predetermined value, and determines to use the visible light image information as the basic information when the reliability of the infrared image information is below a predetermined value.

2. The vehicle interior monitoring system according to claim 1, wherein the vehicle interior information acquisition unit includes a biometric information acquisition unit that acquires biometric information, and the information processing unit determines to use the biometric information as basic information when the reliability of the biometric information is equal to or greater than a predetermined value.

3. The vehicle interior monitoring system according to claim 2, characterized in that the information processing unit puts the image information acquisition unit into a sleep state when the reliability of the biological information is less than a predetermined value.

4. The vehicle interior monitoring system according to claim 2, wherein the vehicle interior information acquisition unit further comprises a non-image information acquisition unit that acquires information other than image information, and the biological information acquisition unit obtains the biological information by analyzing the information obtained from the image information acquisition unit and the non-image information acquisition unit.

5. The in-vehicle monitoring system according to claim 4, characterized in that the biological information includes at least one of the following: gaze information, facial expression information, skeletal information, blood pressure information, heart rate information, and respiration information.

6. The vehicle interior monitoring system according to claim 1, wherein the vehicle interior information acquisition unit includes an object information acquisition unit that acquires object information which is information about objects in the vehicle interior, and the object information acquisition unit obtains the object information by analyzing image information obtained from the image information acquisition unit.

7. The vehicle interior monitoring system according to claim 1, further comprising a vehicle information acquisition unit that acquires vehicle information which is information relating to the vehicle, wherein the basic information includes, as types of basic information, at least, biological information in the vehicle interior and object information in the vehicle interior, and the information processing unit determines, based on the vehicle information, whether to use the biological information or the object information as the basic information.

8. The vehicle in-car monitoring system according to claim 1, characterized in that the information processing unit updates the basic information at predetermined intervals.

9. The vehicle interior monitoring system according to claim 1, further comprising an external information acquisition unit that acquires external information which is information obtained from outside the vehicle, wherein the reliability calculation unit calculates the reliability by referring to the external information.

10. The in-vehicle monitoring system according to claim 9, characterized in that the external information includes one of the following: health information relating to a living organism inside the vehicle, weather information in the vicinity of the vehicle, and traffic congestion information related to the vehicle.

11. The in-vehicle monitoring system according to claim 1, characterized in that the image information acquisition unit obtains image information from an imaging device that has pixels for acquiring visible light and pixels for acquiring infrared light in a single image sensor.

12. A program for causing a computer to execute a vehicle interior monitoring method, wherein the vehicle interior monitoring method comprises: a vehicle interior information acquisition step for acquiring multiple pieces of information in the vehicle interior; a reliability calculation step for calculating the reliability of each of the multiple pieces of information; and an information processing step for determining, based on the reliability, which of the multiple pieces of information will be used as basic information, which is the information that forms the basis for vehicle interior monitoring, wherein the vehicle interior information acquisition step includes an image information acquisition step for acquiring visible light image information and infrared image information in the vehicle interior, and the information processing step is characterized in that, if the reliability of the infrared image information is greater than or equal to a predetermined value, the infrared image information is determined to be the basic information, and if the reliability of the infrared image information is less than a predetermined value, the visible light image information is determined to be the basic information.

Images