Condition monitoring devices and condition monitoring programs

Abstract

The status monitoring device (1) includes: an image processing unit (5) for image processing of images captured by a vehicle-mounted camera that captures passengers inside the vehicle; an identification condition information acquisition unit (6) for acquiring identification condition information specifically for the vehicle or a specific passenger; and a passenger detection unit (10) for setting image recognition conditions using the identification condition information, identifying the image after image processing by the image processing unit according to the image recognition conditions, and detecting a specific passenger.

Description

[0001] Cross-reference to related applications

[0002] This application is based on Japanese Patent Application No. 2020-213695, filed on December 23, 2020, the contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to condition monitoring devices and condition monitoring software products. Background Technology

[0004] The following structure is provided: an in-vehicle camera is provided to capture images inside the vehicle, and passengers are identified and the driver is detected based on the difference between the images captured by the in-vehicle camera before the doors are opened and closed and the images captured after the doors are opened and closed (for example, see Patent Document 1).

[0005] Patent Document 1: Japanese Patent Application Publication No. 2012-44404

[0006] However, depending on the camera's location and the vehicle environment, sometimes passengers other than the driver are captured within the camera's field of view. For example, in a sedan, passengers in the front passenger seat and back seat may be captured in addition to the driver; in a bus, passengers in the passenger seats may be captured in addition to the driver. In such cases, if image processing is performed on the image captured by the camera, multiple faces may be detected in the processed image, making it sometimes impossible to detect the driver from among the multiple passengers. Thus, if multiple passengers are captured within the camera's field of view and multiple faces are detected in the processed image, there is a problem of not being able to properly detect a specific passenger from among the multiple passengers. Summary of the Invention

[0007] The purpose of this disclosure is to enable the appropriate detection of a specific passenger from among multiple passengers, even when multiple passengers are captured within the field of view of the vehicle-mounted camera and multiple faces are detected in the image after image processing.

[0008] According to one aspect of this disclosure, an image processing unit processes images captured by an onboard camera that captures passengers inside the vehicle. A recognition condition information acquisition unit acquires recognition condition information specifically for the vehicle or a particular passenger. A passenger detection unit uses the recognition condition information to set image recognition conditions, and identifies the processed image (after image processing by the image processing unit) based on these conditions, thereby detecting the specific passenger.

[0009] Image recognition conditions are set using recognition condition information, and the processed image is recognized based on these conditions to detect specific passengers. By pre-setting the feature information of the vehicle or specific passengers as recognition condition information, the processed image can be recognized based on the feature information of the vehicle or specific passengers, thus enabling the appropriate detection of specific passengers. Therefore, even when multiple passengers are captured within the field of view of the vehicle-mounted camera and multiple faces are detected in the processed image, the specific passenger can still be appropriately detected from among the multiple passengers. Attached Figure Description

[0010] The foregoing objectives, as well as other objectives, features, and advantages of this disclosure, will become more apparent from the accompanying drawings and from the following detailed description. The accompanying drawings are...

[0011] Figure 1 This is a functional block diagram illustrating one implementation method.

[0012] Figure 2 It is a flowchart (1).

[0013] Figure 3 This is a diagram (1) representing characteristic information when a specific passenger is the driver.

[0014] Figure 4 This is a diagram (Figure 2) showing the characteristic information when a specific passenger is the driver.

[0015] Figure 5 This is a diagram (3) showing the characteristic information when a specific passenger is the driver.

[0016] Figure 6 It is a flowchart (part 2). Detailed Implementation

[0017] Hereinafter, one embodiment will be described with reference to the accompanying drawings. Figure 1 As shown, the status monitoring device 1 is a device that detects specific passengers in vehicles such as cars and buses and monitors the status of those specific passengers. For example, it can monitor the driver's status as a specific passenger, determine the driver's eye opening level, facial expression, etc., and thus determine whether the driving operation can be carried out normally or initiate attention arousal as needed.

[0018] The status monitoring device 1 includes an image input unit 3 and a control unit 4 that receive images from the vehicle-mounted camera 2. The vehicle-mounted camera 2 is positioned to capture the entire interior of the vehicle, and outputs the captured images to the status monitoring device 1. Because the vehicle-mounted camera 2 is positioned to capture the entire interior of the vehicle, the images captured by the vehicle-mounted camera 2 may sometimes include the faces of multiple passengers. Furthermore, the vehicle-mounted camera 2 may not necessarily be positioned to capture the entire interior of the vehicle; even if it cannot capture the entire interior, the images captured by the vehicle-mounted camera 2 may sometimes include the faces of multiple passengers.

[0019] If the image input unit 3 inputs an image output from the vehicle-mounted camera 2, it outputs the input image to the control unit 4. The control unit 4 is primarily a microcomputer, comprising a CPU, ROM, RAM, I / O, etc., and performs various processing operations based on a program stored in the ROM. The control unit 4 includes an image processing unit 5, a recognition condition information acquisition unit 6, and a passenger detection unit 10 as its processing components. Furthermore, the functions provided by the control unit 4 can be provided through software stored in a physical memory device (ROM) and a computer executing that software, software only, hardware only, or a combination thereof. The program executed by the control unit 4 includes a status monitoring program.

[0020] If the image processing unit 5 inputs an image output from the image input unit 3, it performs image processing on the input image and outputs the processed image to the personal authentication unit 9 and the passenger detection unit 10.

[0021] The identification condition information acquisition unit 6 acquires identification condition information specifically for vehicles or specific passengers. The identification condition information acquisition unit 6 includes a vehicle application information acquisition unit 7, a vehicle sensor information acquisition unit 8, and a personal authentication unit 9.

[0022] The vehicle operation information acquisition unit 7 acquires vehicle operation information and outputs it to the passenger detection unit 10. The vehicle operation information includes information indicating the location of the vehicle-mounted camera 2, information indicating the location of driving-related equipment such as the steering wheel and gearshift lever, information related to wearable devices, and information indicating actions. The vehicle operation information can be acquired in any way; for example, it can be acquired by reading from a storage medium containing the vehicle operation information, or by manual input from the user.

[0023] The vehicle sensor information acquisition unit 8 acquires vehicle sensor information from vehicle sensors, electronic control devices, and other devices mounted on the vehicle, and outputs the acquired vehicle operation information to the passenger detection unit 10. The vehicle sensor information includes information indicating vehicle speed, gear shift position, seating position, start button operation status, and seat belt wearing status, etc.

[0024] If the personal authentication unit 9 inputs an image processed by the image processing unit 5, it uses the input image processed by the image processing unit to perform personal authentication, and outputs the personal authentication result, which indicates the result of the personal authentication, to the passenger detection unit 10.

[0025] If the passenger detection unit 10 inputs vehicle operation information output from the vehicle operation information acquisition unit 7, vehicle sensor information output from the vehicle sensor information acquisition unit 8, and personal authentication result output from the personal authentication unit 9, then it uses the input vehicle operation information, vehicle sensor information, and personal authentication result to set image recognition conditions. In this case, the passenger detection unit 10 may use all of the vehicle operation information, vehicle sensor information, and personal authentication result to set image recognition conditions, or it may use only one of them to set image recognition conditions. That is, the passenger detection unit 10 may, for example, use only the vehicle operation information to set image recognition conditions, or it may use only the personal authentication result to set image recognition conditions. In addition, the passenger detection unit 10 may, for example, use both the vehicle operation information and the personal authentication result to set image recognition conditions.

[0026] If the passenger detection unit 10 inputs a processed image from the image processing unit 5 while the image recognition conditions are set, it identifies the input processed image according to the set image recognition conditions and detects a specific passenger. That is, if the passenger detection unit 10 detects the bus driver as a specific passenger, it can detect the bus driver by setting image recognition conditions using the feature information of the bus and the driver.

[0027] Next, refer to Figures 2-6 The function of the above-mentioned structures will be explained.

[0028] Control unit 4 waits for the establishment of a start event for status monitoring processing. If the start event for status monitoring processing is established, then status monitoring processing begins. Furthermore, the timing of status monitoring processing is arbitrary. In the case of detecting a driver as a specific passenger, if it is necessary to continuously monitor the driver's status while the vehicle is in motion, the start event for status monitoring processing can be set to be established at a predetermined period while the vehicle is in motion.

[0029] If the control unit 4 starts status monitoring processing, it performs image processing (S1) on the image input from the vehicle camera 2 via the image input unit 3. S1 corresponds to the image processing step. The control unit 4 acquires vehicle operation information (S2), acquires vehicle sensor information (S3), and acquires personal authentication results (S4). S2 to S4 correspond to the recognition condition information acquisition step. The control unit 4 uses at least one of the vehicle operation information, vehicle sensor information, and personal authentication results to set image recognition conditions (S5). In addition, the control unit 4 can acquire at least one of the vehicle operation information, vehicle sensor information, and personal authentication results, and use any one of the acquired ones to set image recognition conditions. Furthermore, the control unit 4 can also perform image processing, acquire vehicle operation information, acquire vehicle sensor information, acquire personal authentication results, and set image recognition conditions in parallel.

[0030] If the control unit 4 inputs an image processed by the image processing unit 5, it identifies the input image processed by the image processing unit based on the image recognition conditions set by the vehicle operation information, vehicle sensor information and personal authentication results (S6), detects a specific passenger (S7, equivalent to the passenger detection step), and ends the status monitoring process.

[0031] Here, the image recognition conditions are explained. Generally, the positional relationship of seats inside a vehicle is determined. That is, in a sedan, the positional relationship of the driver's seat, front passenger seat, and rear seats is determined, with the driver's seat and front passenger seat arranged in the width direction of the vehicle, and the rear seats located behind the driver's seat and front passenger seat. In a bus, the positional relationship of the driver's seat and passenger seats is determined, with the passenger seats located behind the driver's seat. Furthermore, the location of equipment related to driving operations, such as the steering wheel and gear shift lever, is determined within the vehicle. That is, the location of equipment related to driving operations is around the driver's seat. Additionally, in commercial vehicles such as buses and trucks, the driver's attire is often specified. Furthermore, the actions performed by the driver before and during driving are similar. For example, before driving, the driver usually adjusts the rearview mirror, seat position, or operates a navigation device to set a destination; while driving, the driver holds the steering wheel and gear shift lever. Image recognition conditions are set using vehicle application information that considers these characteristics.

[0032] Reference Figures 3-5 The corresponding technology for detecting situations where the passenger being detected is designated as the driver, and the driver is detected as a specific passenger, is explained. Furthermore, Figures 3-5 The content shown represents one example of the characteristics of the driver being detected, but is not limited to the illustrated content. Control unit 4 categorizes the timing of driver detection into broad categories. The timing of driver detection includes cases where detection is based on instantaneous information and cases where detection is based on accumulated information.

[0033] When detecting based on instantaneous information, the control unit 4 sets image recognition conditions using vehicle operation information, vehicle sensor information, and personal authentication results as a classification. In this case, the control unit 4 sets the location of the vehicle-mounted camera 2, driving operation-related equipment, and information related to clothing as vehicle operation information. For example, if the vehicle-mounted camera 2 is located on the A-pillar, there is a characteristic that the driver is closest to the vehicle-mounted camera 2. Therefore, the control unit 4 sets image recognition conditions to detect the person closest to the vehicle-mounted camera 2 as the driver. In addition, for example, in commercial vehicles such as buses and trucks, there is a characteristic that the driver wears prescribed clothing such as a hat or uniform. Therefore, the control unit 4 sets image recognition conditions to detect the person wearing the prescribed clothing as the driver. Image recognition conditions are also set for other characteristics to detect the person matching the characteristic as the driver.

[0034] When detecting based on accumulated information, the control unit 4 sets image recognition conditions using vehicle operation information and vehicle sensor information as a classification. In this case, the control unit 4 sets information related to the action as vehicle operation information. For example, if the driver enters the vehicle through the driver's side door, and it is a right-hand drive vehicle, the control unit 4 sets image recognition conditions so that a person entering from the left side outside the image after the driver's side door is opened is detected as the driver. For example, if the driver operates the start button, gear lever, or seat belt, the control unit 4 sets image recognition conditions so that a person who operates the start button, gear lever, or seat belt is detected as the driver. Image recognition conditions are also set for other features so that a person matching that feature is detected as the driver.

[0035] Specifically, refer to Figure 6 The process of detecting the driver's status by recognizing the processed image based on image recognition conditions is explained here. Furthermore, the cases where the steering wheel's position, vehicle status, and personal authentication status are set as image recognition conditions are explained here.

[0036] Control unit 4 determines whether multiple or a single face is identified in the image after image processing (S11, S12). If control unit 4 determines that multiple faces are identified in the image after image processing (S11: Yes), it determines whether the vehicle is a right-hand drive or a left-hand drive vehicle (S13, S14). If control unit 4 determines that the vehicle is a right-hand drive vehicle (S13: Yes), it identifies the face on the left side of the image (S15); if it determines that the vehicle is a left-hand drive vehicle (S14: Yes), it identifies the face on the right side of the image (S16). Control unit 4 determines whether the vehicle is in motion (S17). If it determines that the vehicle is in motion (S17: Yes), it determines whether personal authentication registration has been completed (S18). If it determines that the person is a registered person or a previously authenticated person, and personal authentication registration has been completed (S18: Yes), it detects the driver (S19).

[0037] The above explains the scenarios where the steering wheel's position, vehicle status, and personal authentication status are used as image recognition criteria. However, as mentioned above... Figures 3-5 As shown, drivers have a wide variety of characteristics, so the features used to detect drivers are arbitrary. Increasing the number of features used can improve reliability, but there are concerns about processing time. Therefore, the items and number of features used to detect drivers can be determined based on the required reliability and processing time.

[0038] Furthermore, the above examples illustrate the use of vehicle application information, vehicle sensor information, and personal authentication results to set image recognition conditions when detection is based on instantaneous information. However, it is also possible to use at least one of these three types of information to set image recognition conditions. The examples also illustrate the use of vehicle application information and vehicle sensor information to set image recognition conditions when detection is based on accumulated information. However, it is also possible to use at least one of these three types of information to set image recognition conditions.

[0039] Furthermore, while the above examples illustrate detecting the driver as a specific passenger, the method can also be applied to detecting passengers in the front passenger seat, rear seat, and passenger compartment as specific passengers. For example, if detecting a front passenger as a specific passenger in a right-hand drive vehicle, the front passenger has the characteristic of being on the right side of the image. Therefore, the control unit 4 can set image recognition conditions to detect people on the right side of the image as front passenger passengers. Similarly, if detecting a front passenger as a specific passenger involves a tendency to turn their face to the side while conversing with the driver, image recognition conditions can be set to detect people with this tendency. In other words, by setting image recognition conditions based on the characteristics of the passengers to be detected, any passenger can be detected.

[0040] As explained above, the following effects can be achieved according to this embodiment. In the status monitoring device 1, image recognition conditions are set using recognition condition information, and the image after image processing is recognized based on the image recognition conditions to detect specific passengers. By pre-setting the feature information of the vehicle or specific passenger as recognition condition information, the image after image processing can be recognized based on the feature information of the vehicle or specific passenger, and specific passengers can be appropriately detected. Therefore, even if multiple passengers are reflected in the field of view of the vehicle-mounted camera 2 and multiple faces are detected in the image after image processing, specific passengers can be appropriately detected from among multiple passengers.

[0041] Furthermore, in the status monitoring device 1, vehicle usage information is used to set image recognition conditions. Specific passengers can be detected based on vehicle usage information. Image recognition conditions are set using information indicating the location of the vehicle-mounted camera 2, the location of the steering wheel and gearshift, information related to clothing, and information indicating actions. This allows the information indicating the location of the vehicle-mounted camera 2, the location of the steering wheel and gearshift, information related to clothing, and information indicating actions to be utilized as characteristic information of specific passengers.

[0042] Furthermore, in the status monitoring device 1, vehicle sensor information is used to set image recognition conditions. Specific passengers can be detected based on this vehicle sensor information. By setting information indicating vehicle speed, gear shift position, seating position, start button operation status, and seatbelt wearing status as vehicle sensor information, these information can be utilized as characteristic information for specific passengers.

[0043] Furthermore, in the status monitoring device 1, personal authentication results are used to set image recognition conditions. This enables the detection of specific passengers based on the personal authentication results.

[0044] Furthermore, in the status monitoring device 1, image recognition conditions are set based on instantaneous information, thereby enabling the rapid detection of specific passengers. On the other hand, setting image recognition conditions based on accumulated information increases the amount of information used to detect specific passengers, thus improving detection accuracy.

[0045] This disclosure is described based on embodiments, but should be understood as not being limited to those embodiments or constructions. This disclosure also includes various modifications and equivalent variations. Furthermore, various combinations and methods, including only one element, one or more, or one or fewer other combinations and methods, are also included within the scope and spirit of this disclosure.

[0046] The control unit and method described in this disclosure can also be implemented using a dedicated computer, which is provided by comprising a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the control unit and method described in this disclosure can also be implemented using a dedicated computer provided by employing one or more dedicated hardware logic circuits to construct a processor. Alternatively, the control unit and method described in this disclosure can also be implemented using one or more dedicated computers, which are composed of a combination of a processor and memory programmed to perform one or more functions, and a processor composed of one or more hardware logic circuits. Furthermore, the computer program can also be stored as instructions to be executed by the computer on a computer-readable non-transferable tangible recording medium.

Claims

1. A status monitoring device, comprising: The image processing unit processes images captured by the vehicle-mounted camera that photographs passengers inside the vehicle. The identification condition information acquisition unit acquires identification condition information specifically for vehicles or specific passengers; as well as The passenger detection unit uses the aforementioned recognition condition information to set image recognition conditions, and identifies the image after image processing by the image processing unit based on the image recognition conditions, thereby detecting specific passengers. The identification condition information acquisition unit includes a vehicle usage information acquisition unit, a vehicle sensor information acquisition unit, and a personal authentication unit. The vehicle usage information acquisition unit acquires vehicle usage information, the vehicle sensor information acquisition unit acquires vehicle sensor information, and the personal authentication unit uses an image processed by the image processing unit to perform personal authentication. The passenger detection unit sets the image recognition conditions by making the items in the vehicle operation information, vehicle sensor information, and personal authentication result used when setting the image recognition conditions based on instantaneous information different from the items in the vehicle operation information, vehicle sensor information, and personal authentication result used when setting the image recognition conditions based on accumulated information.

2. The status monitoring device according to claim 1, wherein, The passenger detection unit uses at least one of the following as vehicle usage information to set image recognition conditions: information indicating the location of the vehicle-mounted camera, information indicating the location of devices related to driving operations, information related to wearables, and information indicating actions.

3. The status monitoring device according to claim 1 or 2, wherein, The passenger detection unit uses at least one of the following as vehicle sensor information to set image recognition conditions: information indicating vehicle speed, information indicating gear shift position, information indicating seat position, information indicating start button operation status, and information indicating seat belt wearing status.

4. A status monitoring program product that causes the control unit of a status monitoring device to execute: The image processing step involves processing the images captured by the vehicle-mounted camera that photographs the passengers inside the vehicle. The identification condition information acquisition step is to acquire identification condition information specifically for vehicles or specific passengers; as well as The passenger detection step involves setting image recognition conditions using the aforementioned recognition condition information, identifying the processed image (after image processing steps) based on these conditions, and detecting specific passengers. The identification condition information acquisition step includes a vehicle usage information acquisition step, a vehicle sensor information acquisition step, and a personal authentication step. The vehicle usage information acquisition step acquires vehicle usage information, the vehicle sensor information acquisition step acquires vehicle sensor information, and the personal authentication step uses the image processed in the image processing step to perform personal authentication. In the passenger detection step, the image recognition conditions are set by making the items in the vehicle operation information, vehicle sensor information, and personal authentication result that are to be used when setting the image recognition conditions based on instantaneous information different from the items in the vehicle operation information, vehicle sensor information, and personal authentication result that are to be used when setting the image recognition conditions based on accumulated information.

Images