Vehicle interior monitoring system and program

Abstract

A vehicle interior monitoring system is provided that can increase the accuracy of information captured from inside a cabin. A vehicle interior monitoring system 1 includes a cabin information acquisition unit 10, which captures a plurality of information elements in a vehicle cabin; a reliability calculation unit 15, which calculates the reliability of each of the plurality of information elements; and an information processing unit 18, which, based on the reliability, determines which of the plurality of information elements is to be designated as the basic information that serves as the basis for vehicle interior monitoring.The cabin information acquisition unit 10 includes an image information acquisition unit 11, which acquires visible light and near-infrared image information in the cabin, and the information processing unit 18 determines to set the visible light image information as the basic information if the reliability of the visible light image information is equal to or higher than a predetermined value, and determines to set the near-infrared image information as the basic information if the reliability of the visible light image information is less than the predetermined value.

Description

Technical field

[0001] The present invention relates to a vehicle interior monitoring system that monitors the interior of a cabin, and to a program. Technical background

[0002] Conventionally, a technology exists for monitoring a driver in a vehicle cabin (e.g., patent literature 1). According to patent literature 1, the driver's condition is detected by analyzing image information captured by a camera. Specifically, a visible light camera, which records images of visible light, and a near-infrared camera, which records near-infrared images, are arranged. When the vehicle is stationary, the visible light camera is activated and the near-infrared camera is deactivated. Conversely, when the vehicle is in motion, the visible light camera is deactivated and the near-infrared camera is activated.In this way, the power supply to the visible light camera is switched on and the power supply to the near-infrared camera is switched on depending on whether the vehicle is in driving mode or stationary.

[0003] While the technology described in patent literature 1 is a simple control system, in some cases a visible light image can capture more diverse types of information than a near-infrared image to capture the driver's biological information while the vehicle is moving, or a clear visible light image cannot be captured when the vehicle is stationary at night. Therefore, the technology is inadequate for accurately capturing information within the cabin. List of oppositions patent literature

[0004] Patent literature 1 Japanese patent disclosure no. 2020-152133 Brief description of the invention: Technical problem

[0005] One object of the present invention is to provide a vehicle interior monitoring system that can increase the accuracy of information captured from inside a cabin. Solution to the problem

[0006] A first aspect of the present invention is a vehicle interior monitoring system that includes: a cabin information acquisition unit that captures a plurality of information elements in a vehicle cabin; a reliability calculation unit that calculates the reliability of each of the plurality of information elements; and an information processing unit that determines, based on reliability, which of the majority of information elements is to be defined as basic information, which is the information that serves as the basis for vehicle interior monitoring, in which The cabin information acquisition unit includes an image information acquisition unit that captures visible light and near-infrared image information in the cabin, and The information processing unit determines to set the visible light image information as the basic information if the reliability of the visible light image information is equal to or higher than a predetermined value, and determines to set the near-infrared image information as the basic information if the reliability of the visible light image information is lower than the predetermined value.

[0007] According to the first aspect of the present invention, when determining the basic information for vehicle interior monitoring, information can be defined as the basic information with high accuracy by determining the basic information based on the reliability of the majority of information elements acquired in the cabin. Furthermore, when acquiring image information, it is possible to acquire visible light image information with priority over near-infrared image information by defining the visible light image information as the basic information if its reliability is equal to or greater than a predetermined value, and defining the near-infrared image information as the basic information if its reliability is less than a predetermined value.In vehicle interior monitoring, it is preferable to prioritize visible light image information when both the reliability of the visible light and near-infrared image information is high, as visible light image information provides a more comprehensive basis for understanding than near-infrared image information. Consequently, it is possible to implement a vehicle interior monitoring system that increases the accuracy of information captured from inside the cabin.

[0008] A second aspect of the present invention is the vehicle interior monitoring system according to the first aspect, in which The cabin information acquisition unit includes a biological information acquisition unit that captures biological information, and The information processing unit determines whether to set the biological information as the basic information if the reliability of the biological information is equal to or higher than a predetermined value.

[0009] According to the second aspect of the present invention, biological information is acquired in the cabin, and if the reliability of the biological information is equal to or higher than the predetermined value, the biological information is set as the baseline information. This makes it possible to perform vehicle interior monitoring based on biological information, for example, of the driver or the like, which is useful information in the cabin.

[0010] A third aspect of the present invention is the vehicle interior monitoring system according to the second aspect, wherein the information processing unit puts the image information acquisition unit into a sleep state when the reliability of the biological information is less than the predetermined value.

[0011] According to the third aspect of the present invention, the image information acquisition unit is put into a sleep state when the reliability of the biological information is less than the predetermined value. Accordingly, for example, when biological information is acquired based on image information, the energy supply to the image information acquisition unit is stopped or reduced by putting the unit into a sleep state. This prevents false detection and also results in energy savings.

[0012] A fourth aspect of the present invention is the vehicle interior monitoring system according to the second aspect, in which The cabin information acquisition unit further includes a non-image information acquisition unit that captures information other than image information, and The biological information acquisition unit acquires biological information by analyzing information acquired by the image information acquisition unit and the non-image information acquisition unit.

[0013] According to the fourth aspect of the present invention, it is possible to acquire biological information from non-image data in addition to image data. Thus, it is possible to acquire biological information even when the reliability of the image data is low.

[0014] A fifth aspect of the present invention is the vehicle interior monitoring system according to the fourth aspect, wherein the biological information includes at least any one of the following: line-of-sight information, facial expression information, skeletal information, blood pressure information, heart rate information, and respiratory information.

[0015] According to the fifth aspect of the present invention, the biological information comprises at least any one of the following: line-of-sight information, facial expression information, skeletal information, blood pressure information, heart rate information, and respiratory information. Accordingly, it is possible to ensure the operational safety of the vehicle by, for example, recording various types of biological information from the driver and conducting vehicle interior monitoring based on this information.

[0016] A sixth aspect of the present invention is the vehicle interior monitoring system according to the first aspect, in which The cabin information acquisition unit includes an object information acquisition unit that captures object information, which is information about an object in the cabin, and The object information acquisition unit acquires the object information by analyzing image information acquired by the image information acquisition unit.

[0017] According to the sixth aspect of the present invention, object information, which is information about an object containing a living body in the cabin, is acquired by analyzing image information acquired by the image information acquisition unit. This makes it possible to determine whether an object or a living body (a child or an abandoned pet) has been left in the cabin.

[0018] A seventh aspect of the present invention is the vehicle interior monitoring system according to the first aspect, which further includes a vehicle information acquisition unit that acquires vehicle information, which is information about the vehicle, in which the basic information as types of basic information include at least biological information in the cabin and object information in the cabin, and The information processing unit determines, based on the vehicle information, which of the biological information or the object information are to be designated as the basic information.

[0019] According to the seventh aspect of the present invention, the vehicle information is used to determine which of the biological or object information is to be designated as the basic information. This makes it possible to carry out vehicle interior monitoring that is appropriate for the vehicle's situation.

[0020] An eighth aspect of the present invention is the vehicle interior monitoring system according to the first aspect, wherein the information processing unit updates the basic information at every predetermined time.

[0021] According to the eighth aspect of the present invention, the basic information is updated at predetermined intervals. Thus, the situation in the cabin can be detected even if the situation in the cabin has changed compared to the situation at the beginning of the monitoring.

[0022] A ninth aspect of the present invention is the vehicle interior monitoring system according to the first aspect, which further includes a detection unit for external information that detects external information, which is information detected from outside the vehicle, in which The reliability calculation unit calculates reliability by referring to external information.

[0023] According to the ninth aspect of the present invention, external information acquired from outside the vehicle is recorded, and the reliability is calculated by reference to this external information. Thus, more suitable base information can be selected, since the reliability of the image information and the like can be determined based on the external information.

[0024] A tenth aspect of the present invention is the vehicle interior monitoring system according to the ninth aspect, wherein the external information includes any health information about a living body in the cabin, weather information around the vehicle, and traffic jam information related to the vehicle.

[0025] According to the tenth aspect of the present invention, the external information includes any health information about the living body in the cabin, weather information around the vehicle, and traffic information related to the vehicle. Accordingly, for example, the accuracy of the biological information is increased by referencing the health information, the accuracy of determining which image information to select is increased by referencing the weather information, and the accuracy of the vehicle information is increased by referencing the traffic information. Thus, the accuracy of the vehicle interior monitoring is increased.

[0026] An eleventh aspect of the present invention is the vehicle interior monitoring system according to the first aspect, wherein the image information acquisition unit acquires image information from an imaging device, the imaging device comprising pixels that capture visible light and pixels that capture near-infrared rays in an imaging element.

[0027] According to the eleventh aspect of the present invention, the image information acquisition unit acquires image information from the imaging device, which includes the pixels that capture visible light and the pixels that capture near-infrared rays in a single imaging element. This makes it possible to acquire a visible light image and a near-infrared image using a single device. Consequently, the size of the image information acquisition unit can be smaller than that in a case where means for acquiring a visible light image and means for acquiring a near-infrared image are arranged separately.

[0028] A twelfth aspect of the present invention is a program that causes a computer to execute a vehicle interior monitoring method, which includes the vehicle interior monitoring procedure: a cabin information acquisition step of capturing a plurality of information elements in a vehicle cabin; a reliability calculation step of calculating a reliability of each of the plurality of information elements; and an information processing step of determining which of the majority of information elements is to be defined as basic information, which is the information that serves as the basis for vehicle interior monitoring, based on reliability, in which the cabin information acquisition step includes an image information acquisition step of acquiring visible light and near-infrared image information in the cabin, and In the information processing step, it is determined to set the visible light image information as the basic information if the reliability of the visible light image information is equal to or higher than a predetermined value, and it is determined to set the near-infrared image information as the basic information if the reliability of the visible light image information is lower than the predetermined value.

[0029] According to the twelfth aspect of the present invention, the same functionality as in the first aspect can be introduced in the vehicle interior monitoring system. Advantageous effect of the invention

[0030] According to the present invention, it is possible to provide a vehicle interior monitoring system that can increase the accuracy of information captured from inside a cabin. Brief description of the drawings [ Fig. 1] Fig. Figure 1 is a conceptual diagram showing an example of a hardware configuration of a vehicle interior monitoring system. [ Fig. 2] Fig. Figure 2 is a block diagram showing an example of a software configuration of the vehicle interior monitoring system. [ Fig. 3] Fig. Figure 3 is a diagram that shows an example of the relationships between information acquisition units. [ Fig. 4] Fig. 4 is a flowchart of the entire vehicle interior monitoring process. [ Fig. 5] Fig. 5 is a flowchart of object detection during boarding. [ Fig. 6] Fig. 6 is a flowchart of biological detection while driving. [ Fig. 7] Fig. 7 is a flowchart of object detection during a stop. [ Fig. 8] Fig. 8 is a flowchart of object detection during exiting. Description of embodiments

[0031] In the following, an embodiment of a vehicle interior monitoring system of the present invention is described with reference to the drawings. Fig. Figure 1 is a conceptual diagram showing an example of a hardware configuration for a vehicle interior monitoring system 1. A main component of the vehicle interior monitoring system is located near a mirror M installed in the cabin of a vehicle 2. In the present embodiment, an example configuration is described in which the main component is located in a space on the rear surface of a rearview mirror M inside the cabin. Although the mirror M in the present embodiment is a mirror, it can be anything that functions as a rearview mirror and could, for example, be a display or the like that allows the driver to visually perceive a situation behind them.

[0032] In the vehicle interior monitoring system 1, control is performed by a control unit 5 (control device). The control unit 5 is a computer or a device incorporating a built-in computer and includes a CPU, ROM, and RAM, which are not shown. The CPU reads a program stored in the ROM into the RAM and executes a process. Although the placement of the control unit 5 is not restricted to any specific location, the control unit 5 can, for example, be integrated with the mirror M or be located at a different position in the cabin than the mirror M.

[0033] In the example of Fig. In the vehicle 2, an imaging device C and a radar R for detecting biological information are integrally arranged with the mirror M. Since the mirror M is located at a position from which the interior of the entire cabin can be observed, and at a position facing the driver's seat, and thus at a position from which the interior of the cabin, the driver, and the like can be detected, this position is suitable as the placement position for the imaging device C and the radar R for detecting biological information. The imaging device C of the present embodiment includes pixels that capture visible light and pixels that capture near-infrared rays in one imaging element and can acquire visible light and near-infrared image information in the cabin.The imaging device C can be switched to capture either visible light or near-infrared image information, depending on the situation. The radar R, used for detecting biological information, captures, for example, heartbeat data and similar information from a living body in the cabin.

[0034] The image information acquired by the imaging device C and the heartbeat information and the like acquired by the radar R for the detection of biological information are transmitted to the control unit 5. Furthermore, the vehicle 2 is equipped with information acquisition devices, such as a sensor or a device, which acquire information to be transmitted to the control unit 5. The information acquisition devices include a vehicle information acquisition unit 6, which acquires vehicle information that is information about the vehicle 2, and an external information acquisition unit 7, which acquires external information that is information acquired from outside the vehicle. The vehicle information acquisition unit 6 and the external information acquisition unit 7 transmit the acquired information to the control unit 5.

[0035] The vehicle information acquisition unit 6 acquires vehicle information about the status of the vehicle 2 from each device installed in the vehicle 2. Although examples of the vehicle information acquisition unit 6 are described below, the vehicle information acquisition unit 6 is not limited to these. A gear information sensor acquires gear information indicating which gear was engaged by shifting gears using a gearshift lever S. A steering sensor acquires steering angle information from a steering wheel WL. A vehicle speed sensor acquires vehicle speed information. A lighting information sensor acquires on / off information from a headlight, interior light, and the like. A windshield wiper sensor acquires windshield wiper actuation information WP. A seat sensor acquires seat information indicating, for example, that a person has sat down in a seat.Furthermore, drive information from an internal combustion engine or an electric motor can be recorded.

[0036] The External Information Collection Unit 7 (II) collects external information about anything other than Vehicle 2 from any device other than Vehicle 2. Although examples of the II and the external information are described below, the II and the external information are not limited to these. The GPS and an internet server work together to collect navigation information, which includes location and environmental information of Vehicle 2. A wearable sensor, such as a smartwatch, collects the current health information of an occupant. It should be noted that the health information may be historical data stored on the server. Additionally, the II can collect weather and traffic information via the internet.It should be noted that these external information elements are displayed on a display D in the cabin as needed. Furthermore, the external information acquisition unit 7 can also acquire time information. The definition of image information, described below, can be derived from the time information and the navigation information. Additionally, the reliability of biological information (described below) from the portable sensor and the driver's health information can be determined.

[0037] Fig. Figure 2 is a block diagram showing an example of a software configuration for the vehicle interior monitoring system 1. The vehicle interior monitoring system 1 includes a cabin information acquisition unit 10, which acquires a plurality of information elements in the cabin of the vehicle 2; a reliability calculation unit 15, which calculates the reliability of each of the plurality of information elements; and an information processing unit 18, which, based on the reliability, determines which of the plurality of information elements is to be designated as the base information, which is the information that serves as the basis for vehicle interior monitoring. A step of acquiring information using the cabin information acquisition unit 10 is a cabin information acquisition step. A step of calculating reliability using the reliability calculation unit 15 is a reliability calculation step.One step of determining the basic information using the information processing unit 18 is an information processing step.

[0038] The cabin information acquisition unit 10 includes an image information acquisition unit 11, which acquires visible light and near-infrared image information from the imaging device C, such as a camera in the cabin. The image information acquisition unit 11 includes an acquisition unit 11a for visible light image information and a near-infrared image information acquisition unit 11b. Thus, the image information acquisition unit 11 acquires at least a plurality of image information elements (image information acquisition step).

[0039] Information Processing Unit 18 determines whether to set the visible light image information as the primary information if the reliability of the visible light image information is equal to or greater than a predetermined value. Conversely, if the reliability of the visible light image information is less than the predetermined value, it determines whether to set the near-infrared image information as the primary information. Information Processing Unit 18 updates the primary information at predetermined intervals. The update interval can be set by Information Processing Unit 18 as appropriate.For example, the update time can be designed differently depending on the type of information being captured; for instance, the update can be performed approximately every few tens of seconds when visible light image information is captured, and the update can be performed approximately every few minutes when near-infrared image information is captured.

[0040] It should be noted that the basic information is information that serves as the basis for vehicle interior monitoring. For example, when calculating the reliability of image information, the basic information is image information, and when calculating the reliability of biological information, as described below, the basic information is biological information. Furthermore, the information processing unit 18 can put the image information acquisition unit 11 into a sleep state if the reliability of the biological information is less than a predetermined value. The sleep state is a state in which the information processing unit 18 decides not to acquire any image information from the image information acquisition unit 11 and involves switching off the power supply to the imaging device C or switching the imaging device C into a power-saving mode. This provides a power-saving effect.

[0041] A method for calculating reliability using the reliability calculation unit 15 is not limited to any specific procedure, and a predetermined calculation method can be included in a program in advance, or the calculation method can be determined using deep learning. Consequently, the reliability of what is considered clear image information is high, the reliability is low if there is a blown-out highlight or a crushed shadow in the image, and the reliability is low if a predetermined point in the image is not detected.

[0042] The cabin information acquisition unit 10 includes a biological information acquisition unit 13, which acquires biological information. The information processing unit 18 determines whether the biological information should be set as the base information if the reliability of the biological information is equal to or greater than the predetermined value. The cabin information acquisition unit 10 also includes a non-image information acquisition unit 12, which acquires information other than image information. The biological information acquisition unit 13 acquires the biological information by analyzing the information acquired by the image information acquisition unit 11 and the non-image information acquisition unit 12. In the present embodiment, the non-image information acquisition unit 12 represents heartbeat information acquired by the biological information detection radar R.The driver's drowsiness can be calculated from the heartbeat information, and if the drowsiness reaches a predetermined value or exceeds it, it is possible to take action, such as changing the control procedure carried out by the information processing unit 18. It should be noted that the non-image information acquisition unit 12 can acquire information other than heartbeat data.

[0043] The cabin information acquisition unit 10 includes an object information acquisition unit 14, which acquires object information about an object containing a living body in the cabin. The object information acquisition unit 14 acquires the object information by analyzing the information acquired by the image information acquisition unit 11. The object information acquisition unit 14 is used to detect an abandoned object or living body, as described below.

[0044] The vehicle interior monitoring system 1 further includes a vehicle information acquisition unit 16, which acquires vehicle information, specifically information about the vehicle 2. The basic information includes, as types of basic information, at least the biological information in the cabin and the object information in the cabin. The information processing unit 18 determines, based on the vehicle information, which of the biological information or the object information is to be designated as the basic information. For example, the vehicle information acquisition unit 16 acquires information from the gear information sensor indicating which gear (park, drive, neutral, etc.) has been selected. Furthermore, the vehicle information acquisition unit 16 acquires information from the vehicle speed sensor indicating whether the vehicle 2 is in motion or stationary.

[0045] The vehicle interior monitoring system 1 further includes an external information acquisition unit 17, which acquires external information that is information acquired from outside the vehicle 2. The reliability calculation unit 15 calculates the reliability by referring to the external information. The external information includes any health information about a living person (such as the driver) in the cabin, weather information around the vehicle 2, and traffic congestion information related to the vehicle 2.

[0046] The biological information collected by the biological information acquisition unit 13 includes at least one of the following: line-of-sight information, facial expression information, skeletal information, blood pressure information, heart rate information, and respiratory information. Line-of-sight information is information about the driver's line of sight, for example, and allows the system to determine where the driver's attention is focused. Facial expression information is information about the driver's facial expression, for example, and allows the system to determine whether the facial expression differs from that under normal conditions (for example, whether the driver appears sleepy). Skeletal information allows the system to determine the driver's posture, for example.Based on blood pressure, heart rate, and respiration information, the driver's blood pressure, heart rate, and respiration status can be determined.

[0047] Fig. Figure 3 is a diagram that shows an example of the relationships between the information acquisition units. The image information acquisition unit 11, the non-image information acquisition unit 12, the biological information acquisition unit 13, and the object information acquisition unit 14 are as shown in Fig. Figure 3 shows the relationships between the units. The biological information acquisition unit 13 includes a line-of-sight information acquisition unit 13a, a facial expression information acquisition unit 13b, a skeletal information acquisition unit 13c, a blood pressure information acquisition unit 13d, and a heart rate and respiration information acquisition unit 13e. The information from the visible light image information acquisition unit 11a serves as the basis for the line-of-sight information acquisition unit 13a, the facial expression information acquisition unit 13b, the skeletal information acquisition unit 13c, the blood pressure information acquisition unit 13d, and the object information acquisition unit 14. That is, each of the information elements described above is acquired by analyzing the visible light image information.Furthermore, the information from the near-infrared image information acquisition unit 11b serves as the basis for the line-of-sight information acquisition unit 13a, the facial expression information acquisition unit 13b, the skeletal information acquisition unit 13c, and the object information acquisition unit 14. That is, each of the information elements described above is acquired by analyzing the near-infrared image information. Additionally, the information from the non-image information acquisition unit 12 serves as the basis for the heartbeat and respiration information acquisition unit 13e.

[0048] Fig. 4 is a flowchart of the entire vehicle interior monitoring process. The process in Fig. Step 4 is performed by the CPU of the control unit 5. When the vehicle interior monitoring process begins, vehicle information is acquired in step S11. In the present embodiment, vehicle speed information, gear information, or other information acquired by the vehicle information acquisition unit 16 is used to determine whether the driver has entered the vehicle 2, whether the vehicle 2 is moving, whether the vehicle 2 has stopped, and whether a person has exited the vehicle 2.

[0049] If step S12 determines that the driver has entered vehicle 2 (during entry), an object detection process is performed during entry in step S20. If, however, no entry is determined in step S12, a process in step S13 is performed.

[0050] If step S13 determines that vehicle 2 is moving, a biological detection during driving is performed in process S30. If, however, step S13 determines that driving is not occurring, a process from step S14 is performed.

[0051] If step S14 determines that vehicle 2 has stopped (during stopping), an object detection during stopping is performed in a process from step S40. If, however, no stopping is determined in step S14, a process from step S15 is performed.

[0052] If step S15 determines that the person has exited vehicle 2 (while exiting), an object detection during exit is performed in step S50. If, however, exit is not determined in step S15, a process from step S16 is performed.

[0053] Step S16 determines whether to terminate the vehicle interior monitoring. The termination and initiation of the vehicle interior monitoring can be triggered by a predetermined time, by detecting the opening and closing of a vehicle door 2, or by an operation, for example by the driver, on the vehicle interior monitoring system 1, but is not specifically limited to these methods. If step S16 determines that the vehicle interior monitoring should be terminated, the monitoring is ended. Conversely, if step S16 determines that the vehicle interior monitoring should not be terminated, the process returns to step S11. The time required to return from step S16 to step S11 can vary depending on the situation.

[0054] Fig. Figure 5 is a flowchart of object detection during boarding. One object detection process during boarding (a monitoring target: personal belongings, the number of occupants, etc.) corresponds to step S20 described above. The process of step S20 begins at step S201.

[0055] In step S201, the reliability calculation unit 15 calculates the reliability of visible light image information acquired by the visible light image acquisition unit 11a. If the reliability of the visible light image information is equal to or higher than the predetermined value, the process of step S202 is executed. In step S202, object detection is performed in a visible light mode (RGB mode) by analyzing the visible light image information in the object information acquisition unit 14. If, however, the reliability of the visible light image information in step S201 is not equal to or higher than the predetermined value, the process of step S203 is executed.

[0056] In step S203, the reliability calculation unit 15 calculates the reliability of near-infrared image information acquired by the near-infrared image information acquisition unit 11b. If the reliability of the near-infrared image information is equal to or higher than the predetermined value, the process continues to step S204. In step S204, object detection is performed in a near-infrared (IR) mode by analyzing the near-infrared image information in the object information acquisition unit 14. If, however, the reliability of the near-infrared image information in step S203 is not equal to or higher than the predetermined value, a process from step S205 is carried out. In step S205, it is determined not to perform object detection upon boarding. Then, a process from step S16 is carried out. Fig. 4 carried out.

[0057] Fig. Figure 6 is a flowchart of biological detection during driving. One process of biological detection during driving (the monitoring target: the driver, etc.) corresponds to step S30 described above. The process of step S30 begins at step S301.

[0058] In step S301, the reliability calculation unit 15 calculates the reliability of visible light image information acquired by the visible light image acquisition unit 11a. If the reliability of the visible light image information is equal to or higher than the predetermined value, the process of step S302 is executed. In step S302, face and line-of-sight detection is performed in visible light mode by analyzing the visible light image information in the line-of-sight information acquisition unit 13a and the facial expression information acquisition unit 13b. If, however, the reliability of the face and line-of-sight information in step S301 is not equal to or higher than the predetermined value, the process of step S303 is executed.It should be noted that an example of a case in which the reliability of the facial and line-of-sight information is not equal to or higher than the predetermined value, even when the visible light image is clear, is as follows. For example, if the driver or similar person is wearing a hat or mask, the reliability of the facial expression information tends to decrease. Similarly, the reliability of the line-of-sight information tends to decrease if the driver or similar person is wearing sunglasses.

[0059] In step S303, the reliability calculation unit 15 calculates the reliability of near-infrared image information acquired by the near-infrared image information acquisition unit 11b. If the reliability of the near-infrared image information is equal to or higher than the predetermined value, the process of step S304 is executed. In step S304, face and line-of-sight detection is performed in near-infrared mode by analyzing the near-infrared image information in the line-of-sight information acquisition unit 13a and the facial expression information acquisition unit 13b. If, however, the reliability of the face and line-of-sight information in step S303 is not equal to or higher than the predetermined value, the process of step S305 is executed.It should be noted that an example of a case in which the reliability of the facial and line-of-sight information is not equal to or higher than the predetermined value, even when the near-infrared image is clear, is as follows. For example, if the driver or similar person is wearing a hat or mask, the reliability of the facial expression information tends to decrease. Furthermore, the reliability of the line-of-sight information tends to decrease if the driver or similar person is wearing infrared-protective sunglasses.

[0060] In step S305, the reliability calculation unit 15 calculates the reliability of non-image information acquired by the non-image information acquisition unit 12. If the reliability of the non-image information is equal to or greater than the predetermined value, a process from step S306 is executed. In step S306, heartbeat and respiration detection is performed in a sleep mode by analyzing the non-image information in the heartbeat and respiration information acquisition unit 13e. Here, "sleep mode" refers to a mode in which the image information acquisition unit 11 is placed in a sleep state. If, however, the reliability of the non-image information in step S305 is not equal to or greater than the predetermined value, a process from step S307 is executed.

[0061] In step S307, the reliability calculation unit 15 recalculates the reliability of visible light image information acquired by the visible light image acquisition unit 11a. If the reliability of the visible light image information is equal to or higher than the predetermined value, the process of step S308 is executed. In step S308, skeleton detection is performed in visible light mode by analyzing the visible light image information in the skeleton information acquisition unit 13c. However, if the reliability of the visible light image information in step S307 is not equal to or higher than the predetermined value, the process of step S309 is executed.

[0062] In step S309, the reliability calculation unit 15 calculates the reliability of near-infrared image information acquired by the near-infrared image information acquisition unit 11b. If the reliability of the near-infrared image information is equal to or higher than the predetermined value, a process from step S310 is executed. In step S310, skeletal detection is performed in near-infrared mode by analyzing the near-infrared image information in the skeletal information acquisition unit 13c. If, however, the reliability of the near-infrared image information in step S309 is not equal to or higher than the predetermined value, a process from step S311 is executed. In step S311, it is determined not to perform biological detection while driving. Then, the process from step S16 is executed. Fig. 4 carried out.

[0063] Fig. Figure 7 is a flowchart of object detection during a stop. One object detection process during a stop (monitoring target: an abandoned object, etc.) corresponds to step S40 described above. The process of step S40 begins at step S401.

[0064] In step S401, the reliability calculation unit 15 calculates the reliability of visible light image information acquired by the visible light image acquisition unit 11a. If the reliability of the visible light image information is equal to or higher than the predetermined value, the process of step S402 is executed. In step S402, object detection is performed in visible light mode by analyzing the visible light image information in the object information acquisition unit 14. If, however, the reliability of the visible light image information in step S401 is not equal to or higher than the predetermined value, the process of step S403 is executed.

[0065] In step S403, the reliability calculation unit 15 calculates the reliability of near-infrared image information acquired by the near-infrared image information acquisition unit 11b. If the reliability of the near-infrared image information is equal to or higher than the predetermined value, the process continues to step S404. In step S404, object detection is performed in near-infrared mode by analyzing the near-infrared image information in the object information acquisition unit 14. If, however, the reliability of the near-infrared image information in step S403 is not equal to or higher than the predetermined value, a process from step S405 is executed. In step S405, it is determined not to perform object detection upon stopping. The process then continues from step S16. Fig. 4 carried out.

[0066] Fig. Figure 8 is a flowchart of object detection during disembarkation. One object detection process during disembarkation (monitoring target: an abandoned living body, etc.) corresponds to step S50 described above. The process of step S50 begins at step S501.

[0067] In step S501, the reliability calculation unit 15 calculates the reliability of visible light image information acquired by the visible light image acquisition unit 11a. If the reliability of the visible light image information is equal to or higher than the predetermined value, the process of step S502 is executed. In step S502, object detection is performed in visible light mode by analyzing the visible light image information in the object information acquisition unit 14. If, however, the reliability of the visible light image information in step S501 is not equal to or higher than the predetermined value, the process of step S503 is executed.

[0068] In step S503, the reliability calculation unit 15 calculates the reliability of near-infrared image information acquired by the near-infrared image information acquisition unit 11b. If the reliability of the near-infrared image information is equal to or higher than the predetermined value, the process continues to step S504. In step S504, object detection is performed in near-infrared mode by analyzing the near-infrared image information in the object information acquisition unit 14. However, if the reliability of the near-infrared image information in step S503 is not equal to or higher than the predetermined value, a process from step S505 is executed. In step S505, it is determined not to perform object detection on exit. Then the process continues to step S16. Fig. 4 carried out.

[0069] As described above, in the present embodiment, when determining the basic information for vehicle interior monitoring, information can be defined with high accuracy as the basic information by determining the basic information based on the reliability of the majority of information elements acquired in the cabin. Furthermore, when acquiring image information, it is determined to define the visible light image information as the basic information if its reliability is equal to or greater than the predetermined value, and to define the near-infrared image information as the basic information if its reliability is less than the predetermined value. This allows visible light image information to be acquired with priority over near-infrared image information.This means that in vehicle interior monitoring, it is preferable to prioritize visible light image information when both the reliability of the visible light and near-infrared image information is high, since visible light image information provides more information than near-infrared image information. Consequently, it is possible to implement a vehicle interior monitoring system that increases the accuracy of information captured from inside the cabin.

[0070] Furthermore, in the present embodiment, biological information is acquired in the cabin, and if the reliability of the biological information is equal to or higher than the predetermined value, the biological information is set as the baseline information. This makes it possible to perform vehicle interior monitoring based on the biological information, for example, of the driver, which is useful information in the cabin.

[0071] Furthermore, in the present embodiment, the image information acquisition unit 11 is put into a standby state if the reliability of the biological information is less than the predetermined value. Accordingly, for example, when biological information is acquired based on image information, the energy supply to the imaging device C is stopped or reduced by putting the image information acquisition unit 11 into a standby state. This makes it possible to prevent false detection and also achieve an energy-saving effect.

[0072] Furthermore, in the present embodiment, it is possible to acquire biological information from non-image data acquired by the radar R for biological information detection, in addition to the image information acquired by the imaging device C. Thus, it is possible to acquire biological information even when the reliability of the image information is low.

[0073] Furthermore, in the present embodiment, the biological information includes at least one of the following: line-of-sight information, facial expression information, skeletal information, blood pressure information, heart rate information, and respiratory information. Accordingly, it is possible to ensure the operational safety of the vehicle by, for example, recording various types of biological information from the driver and conducting vehicle interior monitoring based on this information.

[0074] Furthermore, in the present embodiment, object information, which is information about an object containing a living body in the cabin, is acquired by analyzing information acquired by the image information acquisition unit 11. This makes it possible to determine whether an object or a living body (a child or an abandoned pet) has been left in the cabin.

[0075] Furthermore, in the present embodiment, based on vehicle information such as vehicle speed and gear information, it is determined which biological information or object information should be designated as the primary information. This enables vehicle interior monitoring appropriate to the vehicle's situation. In the present embodiment, object detection is performed when entering, stopping, and exiting the vehicle to prevent the leaving of an object or a living body. While driving, biological detection is performed for, for example, the driver, to ensure the operational safety of the vehicle 2.

[0076] Furthermore, in the present embodiment, the basic information is updated at predetermined intervals. This allows the situation in the cabin to be detected even if it has changed since the start of the monitoring. In this embodiment, the basic information is updated at predetermined intervals during driving, stopping, and exiting situations, which differ from the situation at entry, the start of the vehicle interior monitoring. This ensures that appropriate basic information is selected for each situation. It should be noted that the update interval is not uniform and, for example, preferably differs between driving and the other situations, and also preferably differs between the processes occurring during driving.

[0077] Furthermore, in the present embodiment, external information acquired from outside the vehicle 2 is acquired, and the reliability is calculated by reference to this external information. Thus, more suitable base information can be selected, since the reliability of the image information and the like can be determined according to the external information.

[0078] Furthermore, in the present embodiment, the external information includes health information about the living body in the cabin, weather information around the vehicle 2, and traffic congestion information related to the vehicle 2. Accordingly, for example, the accuracy of the biological information is increased by referencing the health information, the accuracy in determining which image information to select is increased by referencing the weather information, and the accuracy of the vehicle information is increased by referencing the traffic congestion information. For example, the reliability of the biological information acquired from image information can be determined by analyzing the weather information and the ambient light around the vehicle 2, and furthermore, by analyzing the ambient light inside the cabin.

[0079] Furthermore, in the present embodiment, the image information acquisition unit 11 acquires image information from the imaging device C, which includes the pixels that capture visible light and the pixels that capture near-infrared rays in a single imaging element. This makes it possible to acquire both a visible light image and a near-infrared image using a single device. Consequently, the size of the image information acquisition unit 11 can be smaller than that in a case where means for acquiring a visible light image and means for acquiring a near-infrared image are arranged separately.

[0080] Although the embodiment of the present invention has been described above, the present invention is not limited to the embodiment described above. Furthermore, the embodiment described above can, if necessary, be partially deleted or combined. Moreover, any process shown in the flowcharts of the embodiment described above can be a program that can be implemented in a computer or a recording medium that records the program.

[0081] The present application claims priority over Japanese patent application No. 2023-053649, filed on March 29, 2023, which is hereby incorporated in its entirety by reference. Reference symbol list 1 Vehicle interior monitoring system 5 Control or regulating unit 10 cabin information acquisition units 11 Image information acquisition unit 12 Non-image information acquisition unit 13. Unit for recording biological information 14 Object information acquisition unit 15 Reliability calculation unit 16 Vehicle Information Acquisition Unit 17. Data capture unit for external information 18 Information processing unit C Imaging device M mirror Radar for the detection of biological information QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2023-053649

[0081]

Claims

[1] Vehicle interior monitoring system, comprising: a cabin information acquisition unit that captures a plurality of information elements in a vehicle cabin; a reliability calculation unit that calculates the reliability of each of the plurality of information elements; and an information processing unit that determines, based on reliability, which of the majority of information elements is to be designated as basic information, which is the information that serves as the basis for vehicle interior monitoring, wherein The cabin information acquisition unit includes an image information acquisition unit that captures visible light and near-infrared image information in the cabin, and The information processing unit determines to set the visible light image information as the basic information if the reliability of the visible light image information is equal to or higher than a predetermined value, and determines to set the near-infrared image information as the basic information if the reliability of the visible light image information is lower than the predetermined value. [2] Vehicle interior monitoring system according to claim 1, wherein The cabin information acquisition unit includes a biological information acquisition unit that captures biological information, and The information processing unit determines whether to set the biological information as the basic information if the reliability of the biological information is equal to or higher than a predetermined value. [3] Vehicle interior monitoring system according to claim 2, wherein the information processing unit puts the image information acquisition unit into a sleep state when the reliability of the biological information is less than the predetermined value. [4] Vehicle interior monitoring system according to claim 2, wherein The cabin information acquisition unit further includes a non-image information acquisition unit that captures information other than image information, and The biological information acquisition unit acquires biological information by analyzing information acquired by the image information acquisition unit and the non-image information acquisition unit. [5] Vehicle interior monitoring system according to claim 4, wherein the biological information includes at least any one of line-of-sight information, facial expression information, skeletal information, blood pressure information, heart rate information and respiratory information. [6] Vehicle interior monitoring system according to claim 1, wherein The cabin information acquisition unit includes an object information acquisition unit that captures object information, which is information about an object in the cabin, and The object information acquisition unit acquires the object information by analyzing image information acquired by the image information acquisition unit. [7] Vehicle interior monitoring system according to claim 1, further comprising a vehicle information acquisition unit that acquires vehicle information which is information about the vehicle, wherein the basic information as types of basic information include at least biological information in the cabin and object information in the cabin, and The information processing unit determines, based on the vehicle information, which of the biological information or the object information are to be designated as the basic information. [8] Vehicle interior monitoring system according to claim 1, wherein the information processing unit updates the basic information at every predetermined time. [9] Vehicle interior monitoring system according to claim 1, further comprising an external information acquisition unit which acquires external information which is information acquired from outside the vehicle, wherein the reliability calculation unit calculates the reliability by reference to the external information. [10] Vehicle interior monitoring system according to claim 9, wherein the external information includes any health information about a living body in the cabin, weather information around the vehicle and traffic jam information related to the vehicle. [11] Vehicle interior monitoring system according to claim 1, wherein the image information acquisition unit acquires image information from an imaging device, the imaging device comprising pixels that capture visible light and pixels that capture near-infrared rays in an imaging element. [12] Program that causes a computer to execute a vehicle interior monitoring procedure, wherein the vehicle interior monitoring procedure includes: a cabin information acquisition step of capturing a plurality of information elements in a vehicle cabin; a reliability calculation step of calculating a reliability of each of the plurality of information elements; and an information processing step of determining which of the majority of information elements is to be defined as basic information, which is the information that serves as the basis for vehicle interior monitoring, based on reliability, whereby The cabin information acquisition step includes an image information acquisition step for acquiring visible light and near-infrared image information in the cabin, and In the information processing step, it is determined to set the visible light image information as the basic information if the reliability of the visible light image information is equal to or higher than a predetermined value, and it is determined to set the near-infrared image information as the basic information if the reliability of the visible light image information is lower than the predetermined value.

Images