Vehicle interior monitoring system

Abstract

To provide a cab monitor system that monitors behaviors of a passenger in accordance with a privacy policy for people outside a cab.SOLUTION: A cab monitor system 10 monitors behaviors of a passenger based on an image inside a cab taken by a camera 12. A masking part 30-5 of a processing unit 30 performs masking processing on a portion of the image taken by the camera 12 that corresponds to the outside of the cab.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a vehicle interior monitoring system that monitors the behavior of passengers based on images of the vehicle interior captured by a camera. [Background technology]

[0002] For autonomous shared buses and the like, it has been proposed to photograph and monitor the interior of the vehicle with a camera from the viewpoint of ensuring passenger safety. For example, Patent Document 1 proposes a control method for determining passengers who are at high risk of falling and preventing them from falling while realizing smooth operation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-003936 A Summary of the Invention [Problem to be solved by the invention]

[0004] When photographing people with a camera, it is necessary to protect the privacy of the people who are the subjects. When taking images of the interior of a public bus, it is necessary to explain to passengers that camera images will be taken and that the taken images will be used to maintain the safety of the bus, and obtain their consent in advance.

[0005] However, vehicles such as public buses have windows, and passengers can view the scenery outside. If the inside of the vehicle is photographed, people outside the window will also appear in the image, but it is difficult to obtain consent from these people to be photographed. Therefore, it is necessary to deal with people outside the vehicle who are photographed, rather than passengers. [Means for solving the problem]

[0006] The vehicle interior monitoring system according to the present invention is an interior monitoring system that monitors passenger behavior based on an image of the vehicle interior captured by a camera, a camera for capturing an image of the interior of a vehicle cabin to obtain the interior image of the vehicle cabin; a window recognition unit for recognizing a window area in the interior image of the vehicle cabin obtained by the camera using stored positions of window areas; and an interior / exterior separation unit for performing depth estimation on the interior image of the recognized window area to separate a portion corresponding to the interior of the vehicle cabin from a portion corresponding to the exterior of the vehicle cabin. The part of the image captured by the camera that corresponds to the outside of the vehicle is masked. A masking portion. .

[0007] The masking unit first masks the window area and then overwrites the area corresponding to the interior of the vehicle. It is possible.

[0008] The area outside the vehicle cabin, which is slightly larger than the area inside the vehicle cabin, is masked differently from the other areas outside the vehicle cabin. It is possible. Effect of the Invention

[0010] By masking people and objects outside the window that appear in the camera image, privacy policies regarding people outside the vehicle can be complied with. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an overall configuration of a vehicle interior monitoring system 10. FIG. [Diagram 2] 10 is a flowchart showing an example of image processing in a processing unit 30 for an image captured by a camera 12. [Diagram 3] FIG. 2 is a diagram showing an image before and after masking. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiment described herein.

[0013] "Overall Configuration" First, the vehicle in which the in-vehicle monitoring system according to the embodiment is mounted is a shared bus that carries passengers and travels along a predetermined route, and is assumed to be an autonomous vehicle that automatically performs acceleration / deceleration, steering, etc., but the system can also be applied to semi-autonomous vehicles and normal vehicles driven by a passenger. In addition, it is preferable to provide stops in addition to the start and end points on the travel route, and to automatically stop and start at the stops.

[0014] In such vehicles, the situation inside the vehicle must be monitored by an operation control center or the like. For example, when opening and closing the doors, passengers near the doors must be identified and the doors must be opened and closed after confirming that the passengers are a specified distance away from the doors. Also, when passengers get on, the vehicle will not depart until the passengers have finished moving and settled down. At this time, an announcement such as "We are now departing" is automatically made. Also, if a passenger gets up from their seat while the vehicle is moving, a warning will be issued and the vehicle will slow down.

[0015] The vehicle interior monitoring system 10 according to this embodiment is installed in an autonomous vehicle and transmits camera images of the vehicle interior to a driving control center. The driving control center monitors the transmitted images (usually videos) of the vehicle interior and controls the driving of the autonomous vehicle.

[0016] 1 is a block diagram showing the overall configuration of a vehicle interior monitoring system 10. The camera 12 is a device that photographs the interior of the vehicle interior and obtains images, and a plurality of cameras may be provided. Here, the image data obtained by the camera 12 is moving image data, but still image data with high resolution may also be obtained. The communication unit 14 can communicate with an operation control center 20 that manages the operation of the vehicle, a mobile terminal of a user, and the like, by using wireless communication (including short-distance direct wireless communication) and communication over a communication line such as the Internet. As described above, it is preferable for the operation control center 20 to manage the automatic driving of the vehicle.

[0017] The camera 12 and the communication unit 14 are connected to a processing unit 30. The processing unit 30 controls the processing of images captured by the camera 12, the exchange of necessary information with the operation control center 20, and the like.

[0018] The processing unit 30 is formed by a computer and performs various data processing, and its functions include an image recognition unit 30-1, a window recognition unit 30-2, a moving object recognition unit 30-3, an interior / exterior separation unit 30-4, and a masking unit 30-5.

[0019] The image recognition unit 30-1 performs processing such as edge recognition on the image obtained by the camera 12, and recognizes the contours of various objects.

[0020] The window recognition unit 30-2 stores an image of the interior of the vehicle as a default image when the camera 12 is installed in the vehicle or before driving, and also stores the positions of window areas in the image of the interior of the vehicle. Therefore, the window areas in the images obtained during driving can be recognized using data from the window recognition unit 30-2. Also, the window frame portion can be recognized in advance from the default image. Note that there may be cases where curtains are drawn on the windows, and in this case too, it is advisable to store images in advance and use them to recognize the curtains.

[0021] The moving object recognition unit 30-3 recognizes and tracks moving objects from successive images. In this case, images of the interior of the vehicle that have already been obtained can also be used. For example, a moving object can be recognized from a differential image of images that differ in time, and passengers can be recognized by such moving object recognition.

[0022] The vehicle interior / exterior separation unit 30-4 determines whether the image obtained of the window area is inside or outside the vehicle. Note that while the vehicle is traveling, stationary objects outside the vehicle are also recognized as moving objects because they move relatively.

[0023] The determination of whether an image is inside or outside the vehicle is basically performed by depth estimation. Depth estimation can be performed from an image obtained by the camera 12, but is not necessarily limited to images obtained by the camera 12, and depth estimation can also be performed by other means.

[0024] The image of the interior of the vehicle reflected by the window can be determined as a reflected image based on the distance. The reflected image may be displayed as is, or may be deleted. The inside surface of the window may be treated to prevent reflection.

[0025] The masking unit 30-5 masks the window area other than the object in the vehicle interior, and removes the image of that area. Processing such as mosaic, whitening, and blacking can be used, and various masking software can be used.

[0026] Here, for the image from the camera 12, the image of the object inside the vehicle cabin and the image outside the vehicle cabin can be recognized pixel by pixel, and the outside part of the vehicle cabin is masked. The window area may be masked once and the object inside the vehicle cabin may be overwritten, or the outside part of the vehicle cabin other than the inside part of the vehicle cabin may be masked.

[0027] It should be noted that, when the masking process is completed, images including images of the outside of the vehicle cabin before masking are preferably discarded and not stored.

[0028] In the above explanation, the image from the camera 12 is processed in the processing unit 30 mounted on the vehicle to obtain a masked image, but the image obtained by the camera 12 may be sent to an external server computer, for example, a server computer in the operation control center 20, and image processing may be performed in the external server computer to obtain a masked image.

[0029] <Depth estimation (distance estimation from camera 12 and other sensors)> The following methods can be used for this depth estimation. Uses motion capture to focus only on people in the room Passengers are asked to wear some kind of sensor, and the passenger's movements are judged based on signals from the sensor to recognize passengers in the vehicle. This is a reliable method of recognition, but the passengers must wear the sensor. It is also possible to mount a sensor on a ticket or a smartphone, and have the vehicle interior monitoring system 10 recognize the passenger and the sensor when the passenger gets on the vehicle. Time-of-Flight Camera A laser is used to measure the distance to an object. This makes it possible to recognize objects outside the vehicle. The data obtained by the TOF camera is a signal regarding distance, so a separate camera is required for remote monitoring by humans. -Depth estimation with stereo camera If images from two or more cameras are available, it is easy to detect the distance to the object and it provides the most reliable depth estimation, but the need for multiple cameras makes the system expensive. - Depth estimation with a monocular camera There are various proposals for depth estimation using machine learning with monocular cameras. Although the accuracy is lower than that of stereo cameras, it has the advantage in terms of cost.

[0030] Machine learning can use a technique called board calibration, for example. Board calibration uses a checkerboard-like object to determine various parameters of the camera image and make it possible to detect size, etc. In addition, a technique called semantic segmentation can associate labels and categories, such as what is depicted, with each pixel in the image, making it possible to recognize objects in the image as crew members or passengers.

[0031] Using this technique, it is possible to learn about crew members, passengers, the scenery outside the window, etc. from images taken with a monocular camera, and recognize crew members and passengers individually.

[0032] Since the basic purpose is to recognize objects outside the vehicle cabin and objects inside the vehicle cabin from images of the inside of the window, if an object is connected to the outside of the window, it can be determined to be an object inside the vehicle cabin. It is a good idea to use such determinations as well.

[0033] In addition, the entrance and exit points are fixed, and passengers enter and exit through these points. Therefore, from the moment the vehicle boards, it is possible to recognize and track moving objects, and thereby recognize that the object inside the window is an object inside the vehicle (a passenger or crew member).

[0034] "Masking Procedures" FIG. 2 is a flowchart showing an example of processing performed by the processing unit 30 on an image captured by the camera 12.

[0035] First, a camera image is acquired from the camera 12 (S11). The window recognition unit 30-2 reads out the window position stored in advance and recognizes the window position in the image (S12). Then, the vehicle interior / exterior separation unit 30-4 performs depth estimation for the image inside the window (S13). In this way, the amount of data processing can be limited by limiting the depth estimation to the window area. By processing the image inside the window, the depth of each object located in the image is estimated. That is, the distance to the subject is detected for the image inside the window by the various methods described above. Basically, the distance of each pixel of the image inside the window is estimated. In addition, by using the result of the determination of whether or not there is a moving object obtained by the moving object recognition unit 30-3, it is also possible to recognize the behavior of passengers.

[0036] In the image outside the vehicle cabin, the sky is an area where no objects exist, and pedestrians and the like are objects outside the vehicle cabin, and their distances are significantly different. Here, since it is sufficient to recognize objects outside the window and objects inside the window, it is not necessary to accurately recognize the distance of each object, and it is sufficient to determine the area where the depth is not an indoor object as being outside the vehicle cabin. In other words, the depth of the entire window area is determined, and only the part with a depth equal to or less than a predetermined value is extracted and recognized as an object inside the vehicle cabin.

[0037] Furthermore, when images from a monocular camera are judged using a machine learning model, if a well-trained model is created, it will be possible to instantly and correctly judge whether the vehicle is inside or outside the vehicle.

[0038] Even in the case of distance detection, the positions of the windows are known, and objects outside and inside the vehicle compartment can be distinguished by depth estimation.

[0039] Then, when it is possible to determine whether the image is inside or outside the vehicle, the masking unit 30-5 removes objects inside the vehicle and masks window areas in the image (S14).

[0040] As described above, the masking may be performed by masking the area excluding the area of ​​the objects inside the vehicle cabin, or the objects inside the vehicle cabin may be extracted, and the window area may be masked and then the objects inside the vehicle cabin may be overwritten.

[0041] Then, the obtained image in which the object outside the vehicle compartment has been masked is transmitted to the external operation control center 20 (S15). If an operator is in the vehicle, the image may be displayed on the display 40 in the vehicle. Note that the image may also be displayed when the operator is not in the vehicle.

[0042] The transmitted image data is stored in the operation control center, but may also be stored in the vehicle interior monitoring system 10. Meanwhile, the unmasked image data obtained from the camera 12 is automatically discarded at this stage (S16). In this way, by automatically discarding image data of the portion corresponding to the outside of the vehicle interior that may be problematic under privacy policy, privacy policy can be appropriately addressed.

[0043] "Masking status" FIG. 3 shows an image before masking and an image after masking. In the image before masking, there is a passenger P in the vehicle interior I in front of the window W, and a view O outside the vehicle interior seen through the window W. For such an image, the area of ​​the window W is masked except for the passenger P inside the vehicle interior. In the figure, the masking area is shown as sand. Note that, in this example, the masking method is changed for an area slightly larger than the passenger, which is an object inside the vehicle interior. This is for the purpose of simplifying the masking process, and this makes the passenger P's part easier to understand. Also, for this area, masking may be omitted and an image outside the vehicle interior may be left. This is because if the area is small, no privacy problem occurs. Of course, there is no need to provide an area surrounding such an object inside the vehicle interior, and an image of the object inside the vehicle interior may be displayed directly on the masking area. [Explanation of symbols]

[0044] 10 in-vehicle monitoring system, 12 camera, 14 communication unit, 20 operation control center, 30 processing unit, 30-1 image recognition unit, 30-2 window recognition unit, 30-3 moving object recognition unit, 30-4 vehicle interior / exterior separation unit, 30-5 masking unit, 40 display.

Claims

1. A vehicle interior monitoring system that monitors passenger behavior based on an image of the vehicle interior captured by a camera, A camera for capturing an image of the interior of the vehicle and obtaining the image of the interior of the vehicle; a window recognition unit that recognizes a window area in the vehicle interior image obtained by the camera using the stored positions of the window areas; an interior / exterior separation unit that performs depth estimation on the interior image of the recognized window area and separates a portion corresponding to the interior of the vehicle from a portion corresponding to the exterior of the vehicle; a masking unit that performs a masking process on a portion of the image captured by the camera that corresponds to the outside of the vehicle; Including, After the masking process, the unmasked image obtained by the camera is automatically discarded. In-vehicle monitoring system.

2. 2. The vehicle interior monitoring system according to claim 1, The masking unit first masks the window area and then overwrites the area corresponding to the interior of the vehicle. In-vehicle monitoring system.

3. 2. The vehicle interior monitoring system according to claim 1, The masking method for the area outside the vehicle cabin, which is slightly larger than the area inside the vehicle cabin, is different from that for the other areas outside the vehicle cabin. In-vehicle monitoring system.

Images