Vehicle remote monitoring system

Abstract

This system provides a vehicle remote monitoring system that can display event images on a vehicle's display at an appropriate time when an event occurs in the vehicle. [Solution] The system comprises vehicles 40A, 40B, and 40C having cameras, a group of sensors, and a vehicle-side display that photograph at least one of the interior and exterior of the vehicle, and a remote monitoring device 20 that is separate from the vehicle and can communicate wirelessly with the vehicle. The remote monitoring device includes a recording unit for image data acquired by the camera and transmitted from the vehicle, a determination unit that determines whether or not an event has occurred in the vehicle using the detection values ​​of the group of sensors wirelessly transmitted from the vehicle, an extraction unit that extracts image data from the recording unit at the time of the event if it is determined that an event has occurred, and a transmission unit that wirelessly transmits the extracted image data, which is the event image data, to the vehicle, and the vehicle-side display displays an image based on the event image data.

Description

Technical Field

[0001] The present invention relates to a vehicle remote monitoring system.

Background Art

[0002] The following Patent Document 1 discloses an invention in which when a vehicle control device determines that an event such as an accident has occurred in a vehicle, a drive recorder image when the event occurred when the ignition switch is turned off is automatically displayed on a display.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There may be a case where an image related to an event is desired to be displayed on a display at a timing different from when the ignition switch is turned off. In this regard, the invention of Patent Document 1 has room for improvement.

[0005] In consideration of the above facts, an object of the present invention is to obtain a vehicle remote monitoring system that can display an event image on a display provided in a vehicle at an appropriate timing when an event occurs in the vehicle.

Means for Solving the Problems

[0006] A vehicle remote monitoring system according to a first aspect of the present invention comprises a vehicle having a camera that photographs at least one of the interior and exterior of the vehicle, a group of sensors, and a vehicle-side display, and a remote monitoring device that is separate from the vehicle and capable of wireless communication with the vehicle, wherein the remote monitoring device has a recording unit for image data acquired by the camera and transmitted from the vehicle, a determination unit that determines whether or not an event has occurred in the vehicle using detection values ​​of the group of sensors wirelessly transmitted from the vehicle, an extraction unit that, if it is determined that an event has occurred, extracts the image data at the time of the event from the recording unit, and a transmission unit that wirelessly transmits the extracted image data, which is the event image data, to the vehicle, and the vehicle-side display displays an image based on the event image data.

[0007] A second aspect of the present invention is a vehicle remote monitoring system in the first aspect of the present invention, wherein the vehicle-side display displays an image based on the event image data in response to a predetermined operation by the administrator of the remote monitoring device or an occupant of the vehicle.

[0008] A third aspect of the present invention is a vehicle remote monitoring system in the second aspect of the present invention, wherein the remote monitoring device comprises a display capable of displaying an image based on the event image data, and an input device capable of performing an operation to cause the transmission unit to wirelessly transmit the event image data to the vehicle.

[0009] A fourth aspect of the present invention is a vehicle remote monitoring system according to the second or third aspect of the present invention, wherein the vehicle is equipped with a display operation unit capable of performing operations to display an image based on the event image data wirelessly transmitted from the transmission unit on the vehicle-side display.

[0010] A fifth aspect of the present invention is a vehicle remote monitoring system in any of the first to fourth aspects of the present invention, wherein the vehicle is equipped with a plurality of cameras, and the extraction unit selects at least one of the plurality of cameras as the source for acquiring the image data that will form the basis of the event image data, based on the detection values ​​of the sensor group transmitted wirelessly from the vehicle.

[0011] A sixth aspect of the present invention is a vehicle remote monitoring system in a fifth aspect of the present invention, wherein the extraction unit generates event image data from synthesized image data synthesized based on the image data of a plurality of cameras.

[0012] In the seventh aspect of the present invention, in the vehicle remote monitoring system of any of the first to sixth aspects of the present invention, when the determination unit determines that the event has occurred, an image based on the event image data is interrupted and displayed on a display provided in the remote monitoring device.

[0013] The eighth aspect of the present invention is a vehicle remote monitoring system in any of the first to seventh aspects of the present invention, wherein the determination unit assigns a rank to the event according to the importance of the event.

[0014] The ninth aspect of the present invention is a vehicle remote monitoring system in which the vehicle-side display is capable of displaying the detected values ​​of the sensor group, in any of the first to eighth aspects of the present invention. [Effects of the Invention]

[0015] The remote monitoring device of the vehicle remote monitoring system according to the first aspect of the present invention uses detection values ​​from a group of sensors wirelessly transmitted from the vehicle to determine whether or not an event has occurred in the vehicle, and if it is determined that an event has occurred, it extracts image data from the recording unit at the time the event occurred. Furthermore, the remote monitoring device wirelessly transmits the extracted image data, which is the event image data, to the vehicle. Furthermore, the vehicle's on-board display displays an image based on the received event image data. Thus, in the vehicle remote monitoring system according to the first aspect, a remote monitoring device separate from the vehicle wirelessly transmits the event image data to the vehicle. Therefore, by adjusting the timing of the transmission of the event image data by the remote monitoring device and the timing of the display of the received event image data on the vehicle's on-board display, the event image can be displayed on the vehicle's on-board display at an appropriate timing.

[0016] In a second aspect of the present invention, a vehicle remote monitoring system displays an image based on event image data on the vehicle's display, triggered by a predetermined operation by the administrator of the remote monitoring device or a vehicle occupant. Therefore, when an event occurs in the vehicle, the event image can be displayed on the vehicle's display at an appropriate time.

[0017] In a third aspect of the vehicle remote monitoring system of the present invention, the administrator of the remote monitoring device can determine whether or not to have the transmission unit wirelessly transmit event image data to the vehicle by viewing an image based on event image data displayed on the display of the remote monitoring device, using an input device. Therefore, when an event occurs in the vehicle, the administrator can have the event image displayed on the vehicle's display at an appropriate time.

[0018] In a fourth aspect of the vehicle remote monitoring system of the present invention, the vehicle occupant can use a display operation unit to decide whether or not to perform an operation to display an image based on event image data wirelessly transmitted from a transmitter unit on the vehicle's display. Therefore, when an event occurs in the vehicle, the occupant can have the event image displayed on the vehicle's display at an appropriate time.

[0019] The extraction unit of the vehicle remote monitoring system according to the fifth aspect of the present invention selects at least one camera from a plurality of cameras as a source for acquiring image data that serves as the basis for event image data based on the sensor information of the vehicle wirelessly transmitted from the vehicle. Therefore, the extraction unit can generate event image data that can appropriately display the event.

[0020] The extraction unit of the vehicle remote monitoring system according to the sixth aspect of the present invention generates event image data from the composite image data synthesized based on the image data of a plurality of cameras. Therefore, the extraction unit can generate event image data that can appropriately display the event.

[0021] In the vehicle remote monitoring system according to the seventh aspect of the present invention, when the determination unit determines that an event has occurred, an image based on the event image data is interruptedly displayed on the display provided in the remote monitoring device. Therefore, the possibility that the administrator of the remote monitoring device misses the occurrence of the event is reduced.

[0022] The vehicle remote monitoring system according to the eighth aspect of the present invention assigns a rank to an event according to the importance of the event by the determination unit. Therefore, for example, the administrator of the remote monitoring device can recognize the importance of the occurred event.

[0023] The vehicle-side display of the vehicle remote monitoring system according to the ninth aspect of the present invention can display the detection values of the sensor group of the vehicle. Therefore, for example, the passengers of the vehicle can recognize various states of the vehicle at the time of the event occurrence.

Brief Description of the Drawings

[0024] [Figure 1] It is a diagram showing the overall configuration of the vehicle remote monitoring system according to the embodiment. [Figure 2] It is a control block diagram of the control device of the remote monitoring device. [Figure 3] It is a functional block diagram of the control device. [Figure 4] It is a diagram showing the display of the remote monitoring device. [Figure 5] This diagram shows the event notification section of the display. [Figure 6] This is a control block diagram of the external server for the remote monitoring device. [Figure 7] This is a functional block diagram of the external server. [Figure 8] This is a control block diagram of the bus's ECU. [Figure 9] A plan view of the bus. [Figure 10] This is a functional block diagram of the ECU. [Figure 11] This is a diagram showing the on-board display of a bus. [Figure 12] This figure shows a vehicle-side display showing a different event image than that shown in Figure 11. [Figure 13] This flowchart shows the processes performed by the external server of the remote monitoring device. [Figure 14] This flowchart shows the processes performed by the control unit of the remote monitoring device. [Figure 15] This flowchart shows the processes performed by the bus's ECU. [Modes for carrying out the invention]

[0025] Hereinafter, embodiments of the vehicle remote monitoring system according to the present invention will be described with reference to the drawings.

[0026] As shown in Figure 1, the vehicle remote monitoring system 10 of the embodiment (hereinafter referred to as System 10) includes a remote monitoring device 20 and a plurality of buses (vehicles) 40A, 40B, and 40C.

[0027] The remote monitoring device 20 shown in Figure 1 is installed in a building (not shown) owned by the administrator of system 10. The remote monitoring device 20 comprises a control device 21, a display 24, an input device 27, an audio input / output device 28, and an external server 29.

[0028] As shown in Figure 2, the control device 21 has a hardware configuration that includes a CPU (Central Processing Unit) (processor) 21A, ROM (Read Only Memory) 21B, RAM (Random Access Memory) 21C, storage 21D, wireless communication interface 21E, and input / output interface 21G. The CPU 21A, ROM 21B, RAM 21C, storage 21D, wireless communication interface 21E, and input / output interface 21G are connected to each other so as to be able to communicate with one another via the internal bus 21Z. The CPU 21A and the CPUs 29A and 41A, which will be described later, can acquire time-related information from the timer.

[0029] The CPU 21A is the central processing unit, which executes various programs and controls various components. The CPU 21A reads programs from ROM 21B or storage 21D and executes them using RAM 21C as the working area. The CPU 21A controls each component and performs various calculations according to the programs recorded in ROM 21B or storage 21D.

[0030] ROM21B stores various programs and data. For example, multiple programs are installed in ROM21B. ROM21B also records information related to the vehicle ID. RAM21C temporarily stores programs or data as a working area. Storage21D consists of a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs and data.

[0031] The wireless communication interface (I / F21E) is an interface for wireless communication with various devices. For example, the wireless communication interface (I / F21E) can communicate wirelessly with the wireless communication interface (I / F41E) on buses 40A, 40B, and 40C via a network (e.g., the Internet). The wireless communication interface (I / F21E) can send and receive audio data, camera image data, and sensor group detection values, as described later.

[0032] The input / output interface 21G is an interface for communicating with various devices. For example, the input / output interface 21G is connected to a display 24, an input device 27, an audio input / output device 28, and an external server 29. The display 24 can display various images. For example, the display 24 can display camera images received wirelessly by the wireless communication interface 21E from the wireless communication interface 41E on buses 40A, 40B, and 40C, as well as event images and sensor group detection values ​​received from the external server 29, which will be described later. The input device 27 can input various information to the control device 21. The input device 27 is, for example, a keyboard. The audio input / output device 28 is equipped with a microphone and a speaker.

[0033] Figure 3 shows an example of the functional configuration of the control device 21 in a block diagram. The control device 21 has a functional configuration consisting of a transmit / receive control unit 211, a SIP control unit 212, a display control unit 213, and a server control unit 214. These functions are realized by the CPU 21A reading and executing a program stored in the ROM 21B.

[0034] The transmit / receive control unit 211 controls the wireless communication interface 21E. The wireless communication interface 21E controlled by the transmit / receive control unit 211 can, for example, receive camera image data, wirelessly transmit event images and sensor group detection values ​​(described later) to buses 40A, 40B, and 40C, and record received data in ROM 21B and storage 21D. Furthermore, when a call using SIP (described later) is being performed, the wireless communication interface 21E can wirelessly transmit audio data input to the audio input / output device 28 and receive audio data transmitted from the wireless communication interface 41E on buses 40A, 40B, and 40C.

[0035] The SIP control unit 212 controls communication between the voice input / output device 28, which uses SIP (Session Initiation Protocol), and buses 40A, 40B, and 40C (speaker 44, microphone 45). In other words, the control unit 21 functions as a SIP server.

[0036] When a first call request is transmitted from the remote monitoring device 20 to the selected vehicle (wireless communication I / F 41E) by touch operation on the first call request operation units 34A2, 34B2, and 34C2 (described later), the SIP control unit 212 enables communication between the audio input / output device 28 and the speaker 44 and microphone 45 of the selected vehicle. That is, the audio input to the audio input / output device 28 is output from the speaker 44 of the selected vehicle, and the audio input to the microphone 45 of the selected vehicle is output from the audio input / output device 28. The selected vehicle is the bus corresponding to the first call request operation unit 34A2, 34B2, and 34C2 that was touched.

[0037] When the SIP control unit 212 receives a second call request from buses 40A, 40B, or 40C, it establishes a call between the bus 40A, 40B, or 40C that sent the second call request and the control device 21. Specifically, the audio input to the microphone 45 of bus 40A, 40B, or 40C that sent the second call request is output from the audio input / output device 28, and the audio information input to the audio input / output device 28 is output from the speaker 44.

[0038] The display control unit 213 controls the touch panel display 24.

[0039] Figure 4 shows a display 24 controlled by a display control unit 213. The display 24 shows camera image display areas 33A, 33B, and 33C. Furthermore, camera image display areas 33A, 33B, and 33C have a first display area 33A1, 33B1, 33C1, a second display area 33A2, 33B2, 33C2, a third display area 33A3, 33B3, 33C3, and a fourth display area 33A4, 33B4, 33C4. The first display areas 33A1, 33B1, and 33C1 display camera images acquired by the in-vehicle cameras (cameras) 46 of buses 40A, 40B, and 40C in real time. The second display areas 33A2, 33B2, and 33C2 display camera images acquired by multiple external front cameras 47F of buses 40A, 40B, and 40C in real time while switching between them. The third display areas 33A3, 33B3, and 33C3 display camera images acquired by multiple external rear cameras 47R on buses 40A, 40B, and 40C in real time, switching between them. The fourth display areas 33A4, 33B4, and 33C4 display camera images acquired by multiple external front cameras 47S on buses 40A, 40B, and 40C in real time, switching between them.

[0040] Furthermore, the display 24 shows the second call request reception display units 34A1, 34B1, 34C1, the first call request operation units 34A2, 34B2, 34C2, the call termination operation units 34A3, 34B3, 34C3, and the transmit processing buttons 34A4, 34B4, 34C4. The second call request reception display unit 34A1, the first call request operation unit 34A2, the call termination operation unit 34A3, and the transmit processing button 34A4 correspond to bus 40A. The second call request reception display unit 34B1, the first call request operation unit 34B2, the call termination operation unit 34B3, and the transmit processing button 34B correspond to bus 40B. The second call request reception display unit 34C1, the first call request operation unit 34C2, the call termination operation unit 34C3, and the transmit processing button 34C4 correspond to bus 40C.

[0041] The second call request reception indicators 34A1, 34B1, and 34C1 are equipped with illumination fixtures and blink when the wireless communication I / F21E receives a second call request from buses 40A, 40B, and 40C.

[0042] As described above, when a touch operation is performed on the first call request operation units 34A2, 34B2, and 34C2, and when a second call request is issued from buses 40A, 40B, and 40C, a call is established between the corresponding buses 40A, 40B, and 40C and the control device 21 (voice input / output device 28).

[0043] When a touch operation is performed on the call termination operation units 34A3, 34B3, or 34C3, the SIP control unit 212 controls the communication between the remote monitoring device 20 and buses 40A, 40B, and 40C, rendering it impossible to communicate.

[0044] When a touch operation is performed on the transmission processing buttons 34A4, 34B4, or 34C4, the wireless communication I / F 21E controlled by the transmission / reception control unit 211 wirelessly transmits the event image data (time information at the time of event occurrence) and the sensor group detection value, as described later, to the buses 40A, 40B, and 40C corresponding to the touched transmission processing buttons 34A4, 34B4, or 34C4.

[0045] Furthermore, as shown in Figure 4, an event notification unit 36 ​​is provided on the right side of the display 24. As will be described later, when the external server 29 determines that an event has occurred and the event image data and sensor group detection values ​​are transmitted from the external server 29 to the control device 21, the event notification unit 36 ​​displays information about the event that occurred.

[0046] The server control unit 214 controls the external server 29 based on instructions input to the control device 21 using the input device 27.

[0047] As shown in Figure 6, the external server 29 has a hardware configuration that includes a CPU 29A, ROM 29B, RAM 29C, storage 29D, wireless communication interface 29E, and input / output interface 29G. The CPU 29A, ROM 29B, RAM 29C, storage 29D, wireless communication interface 29E, and input / output interface 29G are interconnected via an internal bus 29Z. As shown in Figure 1, the external server 29 is connected to the control unit 21 by a cable.

[0048] Figure 7 shows an example of the functional configuration of the external server 29 in a block diagram. The external server 29 has a functional configuration consisting of a transmit / receive control unit 291, an event determination unit 292, and a data management unit 293. These functions are realized by the CPU 29A reading and executing a program stored in the ROM 29B.

[0049] The transmit / receive control unit 291 controls the wireless communication interface 29E. The wireless communication interface 29E, controlled by the transmit / receive control unit 291, repeatedly receives camera image data and sensor group detection values ​​from the in-vehicle camera 46, the exterior front camera 47F, the exterior rear camera 47R, and the exterior side camera 47S (described later) via wireless communication from buses 40A, 40B, and 40C at predetermined intervals. As described later, the camera image data and sensor group detection values ​​are linked to information regarding the date and time of acquisition and the location information of buses 40A, 40B, and 40C at the time of acquisition. Furthermore, the transmit / receive control unit 291 transmits the event image and sensor group detection values ​​(described later) to the control device 21 via the cable.

[0050] The event determination unit 292 determines whether a predetermined event has occurred in buses 40A, 40B, and 40C based on the image data and sensor group detection values ​​of the external front camera 47F, external rear camera 47R, and external side camera 47S, which are received from buses 40A, 40B, and 40C and recorded in storage 29D. That is, the event determination unit 292 reads all the image data and sensor group detection values ​​of the external front camera 47F, external rear camera 47R, and external side camera 47S recorded in storage 29D and determines whether an event has occurred in any of buses 40A, 40B, or 40C. Here, "event" in this specification refers to a predetermined event related to buses 40A, 40B, and 40C that occurs outside the buses 40A, 40B, and 40C. The events include, for example, accidents in which buses 40A, 40B, and 40C collide with other vehicles, accidents in which buses 40A, 40B, and 40C collide with obstacles, and accidents in which buses 40A, 40B, and 40C collide with pedestrians. These other vehicles include vehicles other than four-wheeled vehicles; that is, for example, motorcycles and bicycles. Obstacles include, for example, guardrails, road curbs, and buildings.

[0051] The event determination unit 292 determines that an event has occurred, for example, if the camera image data acquired by the external front camera 47F includes another vehicle that has collided with the front end of the buses 40A, 40B, and 40C. The event determination unit 292 also determines that an event has occurred based on acceleration information if an acceleration in a predetermined direction that would not normally occur during normal driving occurs in the buses 40A, 40B, and 40C. The event determination unit 292 may also determine the direction in which the event occurred as seen from the buses 40A, 40B, and 40C based on the camera image data from the external front camera 47F, the external rear camera 47R, and the external side camera 47S, acceleration information, and steering angle information. Furthermore, the event determination unit 292 may determine whether or not an event has occurred by using a learning model (machine learning) that determines the relationship between camera image data, sensor group detection values, and the occurrence of an event.

[0052] Furthermore, when the event determination unit 292 determines that an event has occurred, it generates event image data using camera image data from the external front camera 47F, external rear camera 47R, and external side camera 47S during the time period between a predetermined time before and after the event occurrence time, which is the time the event occurred. This predetermined time period is, for example, 30 seconds. However, this predetermined time period may be longer or shorter than 30 seconds. This predetermined time period may also be zero seconds. In this specification, "event occurrence time" refers to this entire time period. For example, if only the camera image data captured by one camera contains an image representing the event, the event determination unit 292 uses this camera image data as event image data. For example, if the right side of the front end of bus 40A collides with a guardrail 65 provided along the right side of the road on which bus 40A is traveling, the camera image data from the right external front camera 47F is used as event image data (EV1), as shown in Figure 11. Furthermore, if, for example, another vehicle 70 traveling in the lane to the left of bus 40C collides with the left side of bus 40C, the event determination unit 292 generates event image data (EV2) based on the combined image data of the three left-side exterior cameras 47S, as shown in Figure 12.

[0053] Furthermore, as described above, if the direction of the event is determined, the event determination unit 292 may generate event image data using image data from at least one of the following cameras: the front external camera 47F, the rear external camera 47R, and the side external camera 47S, which captured a subject in the determined direction. In this way, event image data (described later) in which the occurred event is clearly recorded can be generated.

[0054] Furthermore, when the event determination unit 292 determines that an event has occurred, it determines the importance of the event based on the camera image data and sensor group detection values ​​at the time the event occurred. Specifically, it determines the importance of the event based on at least one of the following: the degree of damage to buses 40A, 40B, and 40C caused by the accident; the degree of damage to other vehicles that collided with buses 40A, 40B, and 40C; the condition of pedestrians that collided with buses 40A, 40B, and 40C; and the magnitude of the acceleration that occurred in buses 40A, 40B, and 40C. For example, the event determination unit 292 classifies the importance of events into levels 1 to 3. The higher the level number, the more important the event. For example, if an event occurs in which it is estimated that buses 40A, 40B, and 40C have suffered significant damage based on the magnitude of the acceleration that occurred in buses 40A, 40B, and 40C, the event determination unit 292 sets the importance of this event to "3". Furthermore, based on the camera image data, if the event determination unit 292 determines that the damage to other vehicles that collided with buses 40A, 40B, and 40C was minor, it sets the importance of this event to "2". Also, based on the camera image data, if the event determination unit 292 determines that there were no injuries to the occupants of other vehicles that collided with buses 40A, 40B, and 40C, it sets the importance of this event to "1". In this case as well, the event determination unit 292 may determine the importance of the event by using a learning model (machine learning) that finds a relationship between the camera image data, the sensor group detection values, and the importance of the event.

[0055] The event image data generated by the event determination unit 292, the sensor group detection values ​​for the time period in which the camera image data that forms the basis of the event image data was acquired, and information regarding the importance level are transmitted to the control device 21 via the cable by the transmission / reception control unit 291, while being linked with this time information, location information, and the ID information of the bus on which the camera is installed. In other words, the event determination unit 292 extracts camera image data and sensor group detection values ​​related to the occurrence of an event from among the camera image data and sensor group detection values ​​recorded in the storage 29D. Hereinafter, the event image data, sensor group detection values, information regarding the importance level, time information, location information, and bus ID information may be collectively referred to as "event occurrence information."

[0056] When an event occurs in this manner, the transmit / receive control unit 291 transmits event occurrence information to the control device 21. As a result, as shown in Figure 5, information about the event that occurred is displayed on the event notification unit 36 ​​of the display 24. The example in Figure 5 shows that event 1 occurred on bus 40A at a predetermined date and time, and event 2 occurred on bus 40C at a different date and time. Furthermore, when event occurrence information is transmitted to the control device 21, a pop-up screen 38 is displayed in the camera image display areas 33A, 33B, and 33C corresponding to the buses 40A, 40B, and 40C where the event occurred (see Figure 4). This pop-up screen 38 displays an image (video) based on the event image data corresponding to the received event occurrence information for a predetermined period of time.

[0057] Furthermore, when an administrator touches an image representing an event displayed on the event notification unit 36, an image (video) based on the event image data corresponding to this event, sensor group detection values, and information regarding the importance of the event are displayed in the camera image display areas 33A, 33B, and 33C (not shown). At this time, the second display areas 33A2, 33B2, and 33C2 display event images captured by the external front camera 47F, the third display areas 33A3, 33B3, and 33C3 display event images captured by the external rear camera 47R, and the fourth display areas 33A4, 33B4, and 33C4 display event images captured by the external side camera 47S. In addition, the first display areas 33A1, 33B1, and 33C1 display information representing the sensor group detection values ​​(same information as in Figures 11 and 12) and information regarding the importance of the event.

[0058] The data management unit 293 records various data in the ROM 29B or storage 29D, and deletes data recorded in storage 29D. For example, the data management unit 293 sequentially records camera image data and sensor group detection values ​​received by the transmission / reception control unit 291 into storage 29D.

[0059] Furthermore, the data management unit 293 automatically deletes data recorded in the storage 29D as needed. The data management unit 293 classifies the storage period of image data and sensor group detection values ​​from the in-vehicle camera 46, the exterior front camera 47F, the exterior rear camera 47R, and the exterior side camera 47S recorded in the storage 29D into levels 1 to 3. The higher the level number, the more important the data is, and the longer the storage period. Level 1 consists of camera image data and sensor group detection values ​​from time periods when no event was determined to have occurred. Level 2 consists of camera image data and sensor group detection values ​​from time periods when events with an importance level of 1 or 2 occurred. Level 3 consists of camera image data and sensor group detection values ​​from time periods when events with an importance level of 3 occurred. If the data has a storage period of level 1, it is automatically deleted from the storage 29D by the data management unit 293 when a first predetermined time has elapsed from the time when the event determination unit 292 performed the determination process for whether or not an event occurred. The first predetermined time is, for example, 30 minutes. For data with a retention period of level 2, the data management unit 293 automatically deletes it from storage 29D when a second predetermined time has elapsed from the time the event occurrence determination process is performed. The second predetermined time is, for example, one month. However, the first and second predetermined times can be changed to arbitrary values ​​using the input device 27. Data with a retention period of level 3 is not automatically deleted by the data management unit 293. Data with a retention period of level 3 is deleted from storage 29D when a predetermined operation is performed by the administrator of the remote monitoring device 20 using the input device 27.

[0060] As shown in Figure 8, buses 40A, 40B, and 40C are equipped with an ECU 41, a GNSS (Global Navigation Satellite System) receiver 43, a speaker 44, a microphone 45, an in-vehicle camera 46, an out-of-vehicle front camera 47F, an out-of-vehicle rear camera 47R, an out-of-vehicle side camera 47S, a second call request operation unit 49S, a call termination operation unit 49F, a touch-panel vehicle-side display 50, a sensor group 51, and a driver assistance operation unit 62. As shown in Figure 8, the ECU 41 has the same hardware configuration as the control unit 21. That is, the ECU 41 has a hardware configuration of a CPU 41A, ROM 41B, RAM 41C, storage 41D, wireless communication I / F 41E, input / output I / F 41G, and an internal bus 41Z. For example, multiple programs are installed in ROM 41B.

[0061] The input / output interface 41G of the ECU 41 is connected to a GNSS receiver 43, a speaker 44, a microphone 45, an in-vehicle camera 46, an out-of-vehicle front camera 47F, an out-of-vehicle rear camera 47R, an out-of-vehicle side camera 47S, a second call request operation unit 49S, a call termination operation unit 49F, a vehicle-side display 50, a sensor group 51, and a driver assistance operation unit 62.

[0062] The GNSS receiver 43 receives GNSS signals transmitted from GNSS satellites and acquires position information for buses 40A, 40B, and 40C equipped with an ECU 41. The acquired position information is repeatedly transmitted wirelessly to the remote monitoring device 20 and the external server 29 via the wireless communication interface 41E at predetermined intervals.

[0063] Speaker 44 can output various sounds. For example, if communication is established between the remote monitoring device 20 and buses 40A, 40B, and 40C, speaker 44 can output audio data based on the audio input to the audio input / output device 28.

[0064] Various types of audio are input to microphone 45. For example, when audio is input to microphone 45 when communication is established between the remote monitoring device 20 and buses 40A, 40B, and 40C, this audio is output from the audio input / output device 28 of the remote monitoring device 20.

[0065] The in-vehicle cameras 46 are installed inside the buses 40A, 40B, and 40C. For example, the in-vehicle cameras 46 are installed on the ceiling of the buses 40A, 40B, and 40C. The in-vehicle cameras 46 repeatedly acquire image data representing subjects inside the buses 40A, 40B, and 40C at predetermined intervals. There may be one or more in-vehicle cameras 46.

[0066] As shown in Figure 9, the external forward cameras 47F are installed at the front ends of buses 40A, 40B, and 40C. In this embodiment, three external forward cameras 47F are installed side by side in the left-right direction at the front ends of buses 40A, 40B, and 40C. Each external forward camera 47F repeatedly acquires image data representing an external subject located in front of the front end of buses 40A, 40B, and 40C at predetermined intervals. Note that in Figure 9, arrow FR indicates the front of the vehicle, and arrow LH indicates the left side of the vehicle.

[0067] As shown in Figure 9, the external rear cameras 47R are installed at the rear ends of buses 40A, 40B, and 40C. In this embodiment, three external rear cameras 47R are installed side by side at the rear ends of buses 40A, 40B, and 40C. Each external rear camera 47R repeatedly acquires image data representing an external subject located behind the rear end of buses 40A, 40B, and 40C at predetermined intervals.

[0068] As shown in Figure 9, the exterior cameras 47S are installed on both the left and right sides of buses 40A, 40B, and 40C. In this embodiment, three exterior cameras 47S are installed in a row in the front-to-back direction on the left and right sides of buses 40A, 40B, and 40C. The left exterior camera 47S repeatedly acquires image data representing an external subject located to the left of buses 40A, 40B, and 40C at predetermined intervals. The right exterior camera 47S repeatedly acquires image data representing an external subject located to the right of buses 40A, 40B, and 40C at predetermined intervals.

[0069] The second call request operation unit 49S and the call termination operation unit 49F are, for example, mechanical buttons installed inside the buses 40A, 40B, and 40C. However, the second call request operation unit 49S and the call termination operation unit 49F may also be images (operation images) displayed on the vehicle-side display 50.

[0070] The vehicle-side display 50 is installed inside the buses 40A, 40B, and 40C. For example, the vehicle-side display 50 is installed on the interior wall of the buses 40A, 40B, and 40C. There may be one or more vehicle-side displays 50. As shown in Figures 11 and 12, the vehicle-side display 50 has an event image display area 53 and a sensor group detection value display area 54.

[0071] Sensor group 51 is a collective term for the sensors installed on buses 40A, 40B, and 40C. These sensors include, for example, a vehicle speed sensor, an accelerator pedal sensor, a brake pedal sensor, a turn signal sensor, a LiDAR sensor, a radar sensor, air pressure sensors installed on the tires of each wheel, a steering angle sensor that detects the steering angle of the steering wheel, and an acceleration sensor that detects acceleration in various directions generated on the vehicle body. Each of these sensors repeatedly performs a detection operation after a predetermined time has elapsed. In this specification, the detection value acquired by each sensor included in sensor group 51 is referred to as the sensor group detection value.

[0072] The driver assistance control unit 62 is a switch located on the instrument panel of buses 40A, 40B, and 40C.

[0073] Figure 10 shows an example of the functional configuration of the ECU 41 in a block diagram. The ECU 41 has a functional configuration consisting of a transmit / receive control unit 411, a call control unit 412, a display control unit 413, and a driver assistance control unit 414. These functions are realized by the CPU 41A reading and executing a program stored in the ROM 41B.

[0074] The transmit / receive control unit 411 controls the wireless communication interface 41E. The wireless communication interface 41E, controlled by the transmit / receive control unit 411, can communicate wirelessly with the wireless communication interface 21E of the remote monitoring device 20 and the wireless communication interface 29E of the external server 29 via the network. For example, by controlling the wireless communication interface 41E, the transmit / receive control unit 411 wirelessly transmits image data, location information, and sensor group detection values ​​from the in-vehicle camera 46, the exterior front camera 47F, the exterior rear camera 47R, and the exterior side camera 47S to the remote monitoring device 20 and the external server 29 at predetermined intervals, while linking them with camera ID information and bus ID information. Furthermore, the wireless communication interface 41E receives event image data (including time information of event occurrence) and sensor group detection values ​​wirelessly transmitted by the wireless communication interface 21E.

[0075] Furthermore, the transmission / reception control unit 411 repeatedly wirelessly transmits an automatic driving execution signal to the remote monitoring device 20 when the driving assistance operation unit 62 is turned ON and buses 40A, 40B, and 40C are performing Level 4 or Level 5 driving assistance control (automatic driving control) as described later.

[0076] The call control unit 412 controls the speaker 44 and microphone 45. For example, when the wireless communication interface 41E receives a first call request from the remote monitoring device 20, the call control unit 412 enables the functions of the speaker 44 and microphone 45.

[0077] Furthermore, when the second call request operation unit 49S is operated by a crew member, the call control unit 412 causes the wireless communication I / F 41E to wirelessly transmit the second call request to the remote monitoring device 20.

[0078] Furthermore, the call control unit 412 terminates the call between the buses 40A, 40B, and 40C and the remote monitoring device 20 when the call termination operation unit 49F is operated by a crew member.

[0079] When a touch operation is performed on the transmission processing buttons 34A4, 34B4, and 34C4 of the display 24, the wireless communication I / F 21E of the control device 21 transmits event image data and sensor group detection values ​​to buses 40A, 40B, and 40C, and the display control unit 413 displays the event image data and sensor group detection values ​​on the vehicle-side display 50.

[0080] For example, as shown in Figure 5, when the transmission processing button 34A4 is touched while character information representing event 1 is displayed on the event notification unit 36 ​​of the display 24, event image data and sensor group detection values ​​associated with event 1 are transmitted to the bus 40A, and as shown in Figure 11, event image EV1 based on camera image data from the right-side external front camera 47F is displayed in the event image display area 53. This event image EV1 includes an image representing the right side of the front end of the bus 40A and an image representing the guardrail 65. Furthermore, as shown in Figure 11, the sensor group detection value display area 54 displays the sensor group detection value and time information (not shown) when event 1 occurred.

[0081] Furthermore, for example, as shown in Figure 5, when the transmission processing button 34AC is touched while text information representing event 2 is displayed on the event notification unit 36, event image data and sensor group detection values ​​related to event 2 are transmitted to the bus 40C, and as shown in Figure 12, event image EV2, which is a composite image based on image data from all the left-side exterior cameras 47S, is displayed in the event image display area 53. This event image EV2 includes an image representing the left edge of the bus 40C and an image representing other vehicles 70. At this time, as shown in Figure 12, the sensor group detection value display area 54 displays the sensor group detection value and time information (not shown) when event 2 occurred.

[0082] When the driver assistance control unit 62, provided on buses 40A, 40B, and 40C, is turned ON, the driver assistance control unit 414 performs driver assistance control using data acquired by the GNSS receiver 43, the external front camera 47F, the external rear camera 47R, and the sensor group 51. This driver assistance control is performed by operating various devices on buses 40A, 40B, and 40C, including the brake system, engine, electric motor, steering system (not shown), and turn signals. In this specification, "driver assistance control" includes driver assistance control levels 1 to 5 as defined by the SAE (Society of Automotive Engineers). When a predetermined operation is performed on the driver assistance control unit 62, the driver assistance control unit 414 determines (selects) the level of driver assistance control to be performed from levels 1 to 5.

[0083] (Mechanism of action and effect) Next, the operation and effects of this embodiment will be described.

[0084] First, the processing flow performed by the CPU 29A of the external server 29 in this embodiment will be explained using the flowchart in Figure 13. The CPU 29A repeatedly executes the processing shown in the flowchart in Figure 13 after a predetermined amount of time has elapsed.

[0085] First, in step S10 (the word "step" will be omitted hereafter), CPU 29A determines whether or not it has received event occurrence information from buses 40A, 40B, and 40C.

[0086] If CPU 29A determines "Yes" in S10, it proceeds to S11 and records the received event information in storage 29D.

[0087] When processing S11 is complete, CPU29A proceeds to S12 and determines whether or not an event occurred on any of the buses 40A, 40B, or 40C.

[0088] If CPU 29A determines "Yes" in S12, it proceeds to S13, extracts (generates) event image data, and transmits the event image data, sensor group detection values, and information regarding the importance of the event to control device 21.

[0089] When the result in S12 is determined to be No, or when the processing in S13 is completed, the CPU 29A proceeds to S14 and determines whether the predetermined deletion conditions have been met. Specifically, the CPU 29A determines whether a first predetermined time has elapsed for data with a retention period of level 1, a second predetermined time has elapsed for data with a retention period of level 2, or whether a predetermined operation has been performed by the administrator of the remote monitoring device 20 using the input device 27 for data with a retention period of level 3.

[0090] If CPU 29A determines "Yes" in S14, it proceeds to S15 and deletes the data to be deleted from storage 29D.

[0091] When the process in S15 is completed, or when the result in S10 and S14 is determined to be No, the CPU 29A temporarily terminates the process in the flowchart shown in Figure 13.

[0092] Next, the processing performed by the CPU 21A of the control device 21 will be explained using the flowchart in Figure 14. The CPU 21A repeatedly executes the processing shown in the flowchart in Figure 14 after a predetermined amount of time has elapsed.

[0093] First, in S20, the CPU 21A determines whether or not it has received event image data, sensor group detection values, and information regarding the importance of the event from the external server 29.

[0094] If CPU 21A determines "Yes" in S20, it proceeds to S21, where it displays the event image in the event image display area 53 of the display 24, and also displays the sensor group detection value and the importance of the event in the sensor group detection value display area 54.

[0095] When the processing in S21 is complete, the CPU 21A proceeds to S22 and determines whether or not a touch operation has been performed on any of the transmit buttons 34A4, 34B3, or 34C3.

[0096] If CPU 21A determines "Yes" in S22, it proceeds to S23 and sends event image data and sensor group detection values ​​to buses 40A, 40B, and 40C, which correspond to the touched transmit processing buttons 34A4, 34B3, and 34C3.

[0097] When the process in S23 is completed, or when the result in S20 and S22 is determined to be No, the CPU 21A temporarily terminates the process in the flowchart shown in Figure 14.

[0098] Next, we will explain the processes performed by CPU 41A on buses 40A, 40B, and 40C using the flowchart in Figure 15. CPU 41A repeatedly executes the processes shown in the flowchart in Figure 15 at predetermined intervals.

[0099] First, in S30, the CPU 41A determines whether or not it has received event image data and sensor group detection values.

[0100] If CPU 41A determines "Yes" in S30, it proceeds to S31, where it displays the event image in the event image display area 53 of the vehicle-side display 50 and displays the sensor group detection value in the sensor group detection value display area 54.

[0101] When the process in S31 is completed, or when the result in S30 is determined to be No, the CPU 41A temporarily terminates the process shown in the flowchart in Figure 15.

[0102] As described above, in the system 10 of this embodiment, the external server 29 uses image data and sensor group detection values ​​wirelessly transmitted from buses 40A, 40B, and 40C to determine whether or not an event has occurred on buses 40A, 40B, and 40C. Furthermore, if the external server 29 determines that an event has occurred, it extracts only the event image data, which is the image data at the time of the event, from the image data recorded in the storage 29D and transmits the event image data to the control device 21. Furthermore, this event image data is wirelessly transmitted from the control device 21 to buses 40A, 40B, and 40C. Furthermore, the vehicle-side displays 50 on buses 40A, 40B, and 40C display images based on the received event image data. Thus, in system 10, a remote monitoring device 20, separate from buses 40A, 40B, and 40C, wirelessly transmits the event image data to buses 40A, 40B, and 40C. Therefore, by adjusting the timing of transmission of event image data from the remote monitoring device 20 (control device 21) to buses 40A, 40B, and 40C, the event image can be displayed on the vehicle-side display 50 at the appropriate timing.

[0103] In other words, if the CPU 41A of buses 40A, 40B, and 40C determines whether or not an event has occurred, and immediately displays the event image on the vehicle-side display 50 when it determines that an event has occurred, there is a risk that passengers of buses 40A, 40B, and 40C who do not wish to see images of other vehicles involved in the accident or the occupants of other vehicles may unintentionally see the event image through the vehicle-side display 50. In contrast, in this embodiment, the administrator of the remote monitoring device 20 can determine the timing of transmitting the event image data to buses 40A, 40B, and 40C. Therefore, for example, if a police officer who has rushed to the scene of the accident involving buses 40A, 40B, and 40C communicates with the administrator of the remote monitoring device 20 using the audio input / output device 28 of the remote monitoring device 20 and the speakers 44 and microphones 45 of buses 40A, 40B, and 40C, and after the police officer has moved all passengers out of the buses 40A, 40B, and 40C, requests the administrator to transmit event image data and sensor group detection values, only the police officer will be able to view the event image data and sensor group detection values ​​displayed on the vehicle-side display 50.

[0104] Furthermore, the external server 29 (event determination unit 292) can determine the direction in which an event occurred as seen from buses 40A, 40B, and 40C, based on camera image data from the external front camera 47F, external rear camera 47R, and external side camera 47S, acceleration information, and steering angle information. Therefore, using the image data from the external front camera 47F, external rear camera 47R, and external side camera 47S that captured a subject in the determined direction, event image data in which the event occurred is clearly recorded can be generated. As a result, it becomes easier for anyone viewing the event image displayed on the vehicle-side display 50 to accurately understand the content of the event that occurred.

[0105] Furthermore, the event determination unit 292 can generate event image data based on composite image data generated by combining image data from multiple cameras. Therefore, the external server 29 can generate event image data that can appropriately display the events that have occurred.

[0106] Furthermore, when event occurrence information is transmitted from the external server 29 to the control device 21, a pop-up screen 38 is displayed in the camera image display areas 33A, 33B, and 33C corresponding to the buses 40A, 40B, and 40C where the event occurred. Therefore, even if the administrator of the remote monitoring device 20 is not watching the display 24, the administrator can easily notice that an event has occurred.

[0107] Furthermore, the external server 29 ranks events according to their importance. As a result, the administrator of the remote monitoring device 20, upon recognizing the importance of an event displayed on the display 24 (first display areas 33A1, 33B1, 33C1), can easily decide whether or not to transmit the event image data and sensor group detection values ​​corresponding to that event to buses 40A, 40B, and 40C. Also, for example, if the administrator recognizes through the display 24 that events have occurred simultaneously on multiple different buses 40A, 40B, and 40C, they can decide which bus to prioritize transmitting the event image data and sensor group detection values ​​to, while considering the importance of each event.

[0108] Although the system 10 according to the embodiment has been described above, these can be modified as appropriate without departing from the spirit of the present invention.

[0109] For example, the control device 21, which receives event image data and sensor group detection values ​​from an external server 29, may automatically wirelessly transmit the event image data and sensor group detection values ​​to the corresponding buses 40A, 40B, and 40C. In this case, buses 40A, 40B, and 40C may be equipped with a display operation unit for displaying the received event image data and sensor group detection values ​​on the vehicle-side display 50. The display operation unit may be a touch-operable image displayed on a touch-panel type vehicle-side display 50, or it may be a mechanical button provided on the vehicle-side display 50. In this case, the event image and sensor group detection values ​​will not be displayed on the vehicle-side display 50 simply by the buses 40A, 40B, and 40C (wireless communication I / F 41E) receiving the event image data and sensor group detection values, but will be displayed on the vehicle-side display 50 when a predetermined operation is performed on the display operation unit.

[0110] An operating terminal (e.g., a smartphone or tablet computer) that is capable of wireless communication with an external server 29 and is held by a person who has been granted specific authority by the administrator may communicate wirelessly with the external server 29 to enable the display of all image data and sensor group detection values ​​recorded in storage 29D on the operating terminal's display. In this way, for example, a police officer who has seen the event image and sensor group detection values ​​displayed on the vehicle-side display 50 of buses 40A, 40B, and 40C where an accident occurred can use their own operating terminal to check image data other than the event image data, as well as image data and sensor group detection values ​​acquired at a different time than when the event occurred. Therefore, the police officer can, for example, check the conditions inside the vehicle at the time the event occurred by looking at images taken by the in-vehicle camera 46 at the time the event occurred.

[0111] The cameras installed on buses 40A, 40B, and 40C may consist of only one of the following: an in-vehicle camera 46 and an exterior camera (exterior front camera 47F, exterior rear camera 47R, and exterior side camera 47S). Alternatively, the exterior camera may consist of at least one of the exterior front camera 47F, exterior rear camera 47R, and exterior side camera 47S.

[0112] The external front camera 47F, the external rear camera 47R, and the external side camera 47S may be one unit or multiple units other than three.

[0113] For example, buses 40A, 40B, and 40C do not necessarily need to be able to perform driver assistance control. In other words, the driver may operate buses 40A, 40B, and 40C.

[0114] Buses 40A, 40B, and 40C may be equipped with at least one of the following: an in-vehicle camera 46, an external front camera 47F, an external rear camera 47R, and an external side camera 47S. The in-vehicle camera 46 may include multiple cameras. That is, for example, buses 40A, 40B, and 40C may be equipped with five in-vehicle cameras 46.

[0115] Communication control between the remote monitoring device 20 and buses 40A, 40B, and 40C may be performed using a control method other than SIP. For example, communication control may be performed using Web Real-Time Communication. In this case, SIP may be combined with a control method other than SIP.

[0116] Multiple speakers 44 and microphones 45 may be provided on buses 40A, 40B, and 40C.

[0117] Furthermore, a bus may have multiple vehicles. For example, if a bus consists of multiple vehicles connected to each other, each vehicle may be provided with a vehicle-side display 50. In this case, the remote monitoring device 20 may display event images and sensor group detection values ​​on the vehicle-side displays 50 of all vehicles, or it may display event images and sensor group detection values ​​on the vehicle-side displays 50 of only some of the vehicles.

[0118] The present invention may also be applied to vehicles of a different type than buses 40A, 40B, and 40C (for example, passenger cars).

[0119] When the wireless communication interface 21E receives a second call request from buses 40A, 40B, or 40C, the speaker of the audio input / output device 28 may generate a predetermined sound. Also, when the wireless communication interface 21E receives a second call request from buses 40A, 40B, or 40C, the display 24 may display a message indicating the receipt of the second call request. [Explanation of Symbols]

[0120] 10. Vehicle Remote Monitoring System (System) 20 Remote monitoring device 21E Wireless communication I / F (transmission section) 24 displays 27 Input devices 29D Storage (Recording Unit) 292 Event determination unit (determination unit) (extraction unit) 40A 40B 40C Bus (vehicle) 46. ​​In-car camera (camera) 47F Front-facing camera (camera) 47R Rear exterior camera (camera) 47S Exterior Camera (Camera) 50 Vehicle-side displays 51 Sensor Groups

Claims

1. A vehicle having cameras, sensors, and a vehicle-side display that capture images of at least one of the interior and exterior of the vehicle, A remote monitoring device that is separate from the aforementioned vehicle and capable of wireless communication with the aforementioned vehicle, Equipped with, The remote monitoring device, A recording unit for image data acquired by the camera and transmitted from the vehicle, A determination unit that determines whether or not an event has occurred in the vehicle using the detection values ​​of the sensor group transmitted wirelessly from the vehicle, If it is determined that the aforementioned event has occurred, an extraction unit extracts the image data from the recording unit at the time the event occurred, A transmission unit that wirelessly transmits the extracted image data, which is event image data, to the vehicle, It has, A vehicle remote monitoring system in which the vehicle-side display shows an image based on the event image data.

2. The vehicle remote monitoring system according to claim 1, wherein the vehicle-side display displays an image based on the event image data triggered by a predetermined operation by the administrator of the remote monitoring device or an occupant of the vehicle.

3. The remote monitoring device, A display capable of displaying an image based on the aforementioned event image data, The transmission unit is equipped with an input device capable of performing an operation to wirelessly transmit the event image data to the vehicle, The vehicle remote monitoring system according to claim 2, having the following features.

4. The vehicle remote monitoring system according to claim 2 or 3, wherein the vehicle is equipped with a display operation unit capable of performing operations to display an image based on the event image data wirelessly transmitted from the transmission unit on the vehicle's display.

5. The vehicle is equipped with multiple cameras, The vehicle remote monitoring system according to claim 1 or 2, wherein the extraction unit selects at least one of the plurality of cameras as the source for acquiring the image data that will form the basis of the event image data, based on the detection values ​​of the sensor group transmitted wirelessly from the vehicle.

6. The vehicle remote monitoring system according to claim 5, wherein the extraction unit generates the event image data from synthesized image data synthesized based on the image data of a plurality of cameras.

7. The vehicle remote monitoring system according to claim 1 or claim 2, wherein when the determination unit determines that the aforementioned event has occurred, an image based on the event image data is displayed on the display provided on the remote monitoring device.

8. The vehicle remote monitoring system according to claim 1 or 2, wherein the determination unit assigns a rank to the event according to the importance of the event.

9. The vehicle remote monitoring system according to claim 1 or claim 2, wherein the vehicle-side display is capable of displaying the detected values ​​of the sensor group.

Images