Monitoring support device, monitoring support system, and program

Abstract

To provide a monitoring aiding device, a monitoring aiding system, and a program that can reduce burden on a person who monitors the traffic situation of a mobile object.SOLUTION: A monitoring aiding device comprises: a video image acquisition unit that acquires a video image captured from a mobile object; an identification unit that acquires information on an object that is a cause of stopping of the mobile body; and a video image generation unit that generates a superposed video image in which an image indicating the position of the object in a predetermined period is displayed over the video image acquired by the video image acquisition unit.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a monitoring support device, a monitoring support system, and a program.

Background Art

[0002] In recent years, the technological development of autonomous driving, in which a vehicle can run at least partially automatically (without a human driver performing driving operations), has been promoted toward practical use. Here, autonomous driving may be, for example, autonomous driving based on the levels defined by the Society of Automotive Engineers of Japan, Inc. (JSAE), a public interest incorporated association in Japan, or the Society of Automotive Engineers (SAE), a non-profit organization in the United States. On the other hand, even if autonomous driving has been developed or put into practical use to a certain level, for example, in order to cope with unforeseen situations, it is assumed that a human monitor needs to perform monitoring, for example, remotely. As something that can respond to such a need, for example, Patent Document 1 proposes a vehicle remote monitoring support system that can continue driving by remotely operating and supporting an operator belonging to a control center when driving of a vehicle becomes difficult.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a monitoring system for a moving object such as a vehicle, it is desirable to reduce the burden on a person (monitor) who monitors the traffic situation of the moving object.

[0005] An object of the present disclosure is to provide a monitoring support device, a monitoring support system, and a program that can reduce the burden on a person (monitor) who monitors the traffic situation of a moving object. [Means for solving the problem]

[0006] A monitoring support device according to one embodiment, A video acquisition unit that acquires images captured from a moving object, An identification unit that acquires information about the object causing the moving object to stop from the video acquired by the video acquisition unit, Based on one or more positions of the object acquired during a predetermined period, an image indicating the position of the object is generated, and the generated image is added to the current video acquired by the video acquisition unit. The displayed superimposed image is generated by the image generation unit, It is equipped with.

[0007] A monitoring support system according to one embodiment is: A communication unit that receives video images captured from a moving object, A video acquisition unit that acquires video received by the aforementioned communication unit, An identification unit that acquires information about the object causing the moving object to stop from the video acquired by the video acquisition unit, Based on one or more positions of the object acquired during a predetermined period, an image indicating the position of the object is generated, and the generated image is added to the current video acquired by the video acquisition unit. The displayed superimposed image is generated by the image generation unit, A display unit that displays the video generated by the aforementioned video generation unit, Includes.

[0008] A program according to one embodiment is: On the computer, A video acquisition step in which video is captured from a moving object, An identification step is to obtain information about the object that caused the moving object to stop from the video acquired in the video acquisition step, An image indicating the position of the object is generated based on one or more positions of the object acquired during a predetermined period, and the generated image is used in the current video acquired in the video acquisition step. The video generation step that generates the superimposed image that was displayed, Make it run. [Effects of the Invention]

[0009] According to one embodiment, it is possible to provide a monitoring support device, a monitoring support system, and a program that can reduce the burden on personnel (monitors) who monitor the traffic conditions of moving objects. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram schematically showing the configuration of a monitoring support system according to one embodiment. [Figure 2]It is a block diagram showing a functional configuration of a monitoring support device according to an embodiment. [Figure 3] It is a diagram showing a usage mode of a monitoring support system according to an embodiment. [Figure 4] It is a diagram showing a usage mode of a monitoring support system according to an embodiment. [Figure 5] It is a flowchart for explaining the operation of a monitoring support device according to an embodiment. [Figure 6A] It is a diagram showing an example of the operation of a monitoring support device according to an embodiment. [Figure 6B] It is a diagram showing an example of the operation of a monitoring support device according to an embodiment. [Figure 6C] It is a diagram showing an example of the operation of a monitoring support device according to an embodiment. [Figure 6D] It is a diagram showing an example of the operation of a monitoring support device according to an embodiment. [Figure 6E] It is a diagram showing an example of the operation of a monitoring support device according to an embodiment. [Figure 6F] It is a diagram showing an example of the operation of a monitoring support device according to an embodiment. [Figure 7] It is a diagram showing an example of the operation of a monitoring support device according to an embodiment. [Figure 8] It is a diagram showing an example of the operation of a monitoring support device according to an embodiment.

Embodiments for Carrying Out the Invention

[0011] In the present disclosure, the "monitoring support device" may be an electronic device that operates by electric power. Also, the "monitoring support system" may include devices (for example, the monitoring support device) that operate by electric power. Further, the "user" may be a person who uses the monitoring support device or the monitoring support system according to an embodiment. The user may also be a person who can receive the benefits by using the monitoring support device or the monitoring support system according to an embodiment.

[0012] One possible scenario in which the monitoring support device or monitoring support system according to one embodiment may be used is, for example, a facility that remotely monitors the traffic conditions of moving objects such as automobiles traveling on roads, such as a traffic monitoring center. Such a facility will be referred to simply as a "monitoring facility" below. Personnel who use the monitoring support device or monitoring support system to monitor the traffic conditions of moving objects at a monitoring facility will be referred to simply as "monitors." In other words, at a monitoring facility, monitors can monitor the traffic conditions of moving objects such as automobiles by using the monitoring support device or monitoring support system.

[0013] In the embodiments described below, a moving object such as a car being monitored by a monitor transmits images captured by an imaging device from the moving object to a monitoring support system. Such image transmission may occur continuously, for example. The images transmitted from the moving object are received by the monitoring support system and displayed on a display unit such as a screen in the monitoring support system. The monitor can monitor the traffic situation of the moving object by visually observing the images displayed on the monitoring support system.

[0014] For example, if an autonomous vehicle stops for any reason, it is assumed that the supervisor will use the monitoring support system to confirm the cause of the stoppage. After confirming that the surrounding environment of the vehicle is safe, the supervisor may remotely control the vehicle if necessary or possible. Furthermore, if the driver is able to operate the vehicle themselves, the supervisor may remotely assist the driver. It is also conceivable that a driver operating a vehicle or an autonomous vehicle may make a request to the supervisor for some kind of action, such as a request for remote control. In such cases, the supervisor will confirm that the surrounding environment of the vehicle is safe and, if necessary or possible, may remotely control the vehicle or remotely assist the driver. Such remote control or remote assistance may be performed by the supervisor themselves, or it may be assigned to another person (for example, a specialized operator) based on the supervisor's instructions.

[0015] In the situations described above, it is conceivable that a single monitor may need to monitor multiple vehicles simultaneously, for reasons such as work efficiency. In this case, the monitoring support system may display images captured from multiple vehicles side by side on a display unit, such as a screen, that is visible to a single monitor. Using such a monitoring support system, a monitor can simultaneously monitor the traffic conditions of multiple vehicles and respond sequentially to various requests from vehicles or drivers. Hereinafter, a monitoring support system and monitoring support device according to one embodiment will be described as being used in the manner described above.

[0016] A vehicle or other moving object monitored by a supervisor may be operating autonomously (e.g., Level 3 to 5), partially autonomously (e.g., Level 2), or being driven by a driver (e.g., Level 0 or 1). The vehicle or other moving object monitored by a supervisor is not limited to ordinary automobiles, but may be any vehicle, such as a bus, truck, ambulance, fire truck, or police car. Furthermore, the vehicle or other moving object monitored by a supervisor is not necessarily limited to automobiles, but may be any other moving object that can participate in road traffic.

[0017] Hereinafter, a monitoring support device and a monitoring support system according to one embodiment will be described with reference to the drawings.

[0018] Figure 1 is a diagram showing a monitoring support system, including a monitoring support device according to one embodiment, together with a mobile body such as an automobile.

[0019] As shown in Figure 1, the monitoring support system 1 may consist of a monitoring support device 10, a communication unit 120, and a display unit 130. The monitoring support system 1 may also be used to support the monitoring of a mobile object 200, such as an automobile. The monitoring support system 1 may be located in a monitoring facility, for example. The mobile object 200 may be driven by a driver, at least partially autonomously driven, or remotely controlled by, for example, a remote control device 300. The mobile object 200 may be located on a road, for example.

[0020] The mobile vehicle 200 may be, for example, an automobile. The mobile vehicle 200 may be, for example, a regular passenger car. The mobile vehicle 200 is not limited to a regular passenger car, and may be any automobile, such as a bus, truck, ambulance, fire truck, or police car. Furthermore, the mobile vehicle 200 is not necessarily limited to an automobile, and may be any other mobile vehicle capable of participating in road traffic. The following explanation assumes that the mobile vehicle 200 is a passenger car. The mobile vehicle 200 may be equipped with various power sources, such as an engine-powered vehicle, an electric vehicle, or a fuel cell vehicle.

[0021] The mobile unit 200 may be capable of autonomous driving (e.g., Level 3 to 5), partially autonomous driving (e.g., Level 2), or driver operation (e.g., Level 0 or 1). As shown in Figure 1, the mobile unit 200 may include, for example, an imaging unit 210 and a communication unit 220. The mobile unit 200 may not have some of the functional units shown in Figure 1, or it may have functional units other than those shown in Figure 1.

[0022] The imaging unit 210 can capture images. The imaging unit 210 may include an image sensor that electronically captures images, such as a digital camera. The imaging unit 210 may include an image sensor that performs photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging unit 210 is not limited to an imaging device such as a digital camera, but may be any device. The images captured by the imaging unit 210 may be still images, a series of still images, or a video. Hereinafter, images captured by the imaging unit 210, such as still images, a series of still images, and a video, will be collectively referred to as "images."

[0023] The imaging unit 210 may be installed at any location on the mobile body 200. The imaging unit 210 may be fixedly attached to the mobile body 200. Alternatively, the imaging unit 210 may be movably mounted on the mobile body 200 so that the imaging range can be changed.

[0024] In one embodiment, the imaging unit 210 may typically capture images of the area in front of the moving object 200. In particular, the imaging unit 210 may capture images that allow for the understanding of the traffic conditions (environment) of the moving object 200, mainly in front of the moving object 200. The image data captured by the imaging unit 210 may be transmitted to the communication unit 220. The imaging unit 210 may be connected to the communication unit 220 by at least one of wired and wireless connections.

[0025] The communication unit 220 of the mobile unit 200 may be a functional unit that can communicate with the communication unit 120 of the monitoring support system 1 by connecting via at least one of wired and wireless (typically wireless). The communication unit 220 may be dedicated hardware installed on the mobile unit 200, or a general-purpose product such as a smartphone or tablet terminal may be used.

[0026] The communication unit 220 has the function of an interface for wired or wireless communication. The communication method performed by the communication unit 220 in one embodiment may be a wireless communication standard. For example, the wireless communication standard may include cellular phone communication standards such as 2G, 3G, 4G, and 5G. For example, cellular phone communication standards include LTE (Long Term Evolution), W-CDMA (Wideband Code Division Multiple Access), CDMA2000, PDC (Personal Digital Cellular), GSM (Registered Trademark) (Global System for Mobile communications), and PHS (Personal Handy-phone System). For example, wireless communication standards include WiMAX (Worldwide Interoperability for Microwave Access), IEEE802.11, WiFi, Bluetooth (Registered Trademark), IrDA (Infrared Data Association), and NFC (Near Field Communication). The communication unit 220 can support one or more of the above communication standards. The communication unit 220 may be configured to include, for example, an antenna for transmitting and receiving radio waves and an appropriate RF unit. Furthermore, the communication unit 220 may be configured as an interface such as a connector for wired connection to the outside. Since the communication unit 220 can be configured using known technologies for wireless communication, a more detailed explanation of the hardware will be omitted.

[0027] The mobile unit 200 may be equipped with various sensors for detecting various statuses at different points on the mobile unit 200. These sensors may be connected to the communication unit 220 by at least one of wired and wireless connections. The mobile unit 200 may also be equipped with devices such as an ECU (engine control unit) for detecting various statuses at different points on the mobile unit 200. In this case, the ECU may be connected to the communication unit 220 by at least one of wired and wireless connections.

[0028] The mobile vehicle 200 may be equipped with various functions necessary for performing at least partial autonomous driving. Various technologies related to autonomous driving may be employed, including existing and / or future development technologies. Furthermore, the mobile vehicle 200 may be equipped with various functions that enable remote operation (remote control) by personnel other than the occupants of the mobile vehicle 200. Various technologies related to such remote operation (remote control) may be employed, including existing and / or future development technologies.

[0029] The remote control device 300 may be a controller for remotely operating (remotely controlling) the mobile unit 200. The remote control device 300 may include, for example, at least one of a joystick, slider, touchpad or touch panel, and / or various switches. For example, an operator stationed at a monitoring facility can remotely operate the mobile unit 200 using the remote control device 300.

[0030] The remote control device 300 may be connected to the communication unit 120 of the monitoring support system 1 by at least one of wired and wireless means. By being connected to the communication unit 120 and communicating with the communication unit 220, the remote control device 300 can remotely control the mobile unit 200.

[0031] Figure 2 is a block diagram showing the functional configuration of a monitoring support system 1 according to one embodiment. As shown in Figure 2, the monitoring support system 1 may be configured to include a monitoring support device 10, a communication unit 120, and a display unit 130.

[0032] The communication unit 120 has the function of an interface for communication by wire or wireless. The communication unit 120 may have the function of communicating with the communication unit 220 of the mobile unit 200 by wire or wireless. The communication unit 120 may be a functional unit based on the same or similar concept as the communication unit 220. As a functional unit that communicates with the communication unit 220 wirelessly, the communication unit 120 may be a functional unit based on the same or similar concept as known wireless communication devices such as an RF unit. Also, as a functional unit that communicates with the communication unit 220 by wire, the communication unit 120 may be a functional unit based on the same or similar concept as a device such as a connector. Since the communication unit 120 can be constructed using known technology, a more detailed explanation is omitted.

[0033] The display unit 130 may be any display device, such as a liquid crystal display (LCD), an organic electro-luminescence panel (OLED), or an inorganic electro-luminescence panel (IEL). The display unit 130 may display various types of information, such as characters, graphics, or symbols. The display unit 130 may also display various GUI objects and icon images to prompt the user to operate the monitoring support device 10 or the mobile unit 200. The display unit 130 may be a functional unit based on the same or similar concepts as the known display devices described above. Since the display unit 130 can be constructed using known technology, a more detailed explanation is omitted.

[0034] The monitoring support device 10 may include a video acquisition unit 11, an information acquisition unit 12, an identification unit 13, a video generation unit 14, and a storage unit 15, as shown in Figure 2. The monitoring support device 10 may not include some of the functional units shown in Figure 2, or it may include functional units other than those shown in Figure 2. For example, the monitoring support device 10 according to one embodiment may include a video acquisition unit 11, an identification unit 13, and a video generation unit 14.

[0035] The monitoring support device 10 according to one embodiment can be configured as various electronic devices. For example, the monitoring support device 10 may be a specially designed terminal, a general-purpose computer, a laptop computer, or a server. Furthermore, the monitoring support device 10 may be a tablet computer or a smartphone.

[0036] The monitoring support device 10 may include at least one processor, such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), to provide control and processing capabilities for performing various functions. The monitoring support device 10 may be implemented with a single processor, with several processors, or with individual processors. The processor may be implemented as a single integrated circuit (IC). The processor may be implemented as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be implemented based on various other known technologies.

[0037] In one embodiment, the monitoring support device 10 may be configured as, for example, a CPU and a program executed by the CPU. The program executed in the monitoring support device 10, and the results of processing performed in the monitoring support device 10, may be stored in, for example, a storage unit 15. The monitoring support device 10 may include memory necessary for its operation as appropriate. The monitoring support device 10 may be configured as dedicated hardware, may include software in at least part of it, or may be composed entirely of software.

[0038] The video acquisition unit 11 acquires the video captured by the imaging unit 210 of the mobile body 200. That is, the video acquisition unit 11 acquires the video captured by the mobile body 200. The video acquisition unit 11 may acquire the video captured by the imaging unit 210 via, for example, the communication unit 220 of the mobile body 200 and the communication unit 120 of the monitoring support system 1. As described above, the video acquisition unit 11 may capture video that allows for understanding the traffic situation (environment) of the mobile body 200, mainly in front of the mobile body 200. For example, the video acquisition unit 11 may capture video captured by the imaging unit 210, such as video of the mobile body 200 while it is moving or stopped. In this case, the video acquisition unit 11 may capture video of, for example, cars, motorcycles, bicycles, pedestrians, strollers, roads, buildings, and / or scenery that are in front of the mobile body 200.

[0039] The information acquisition unit 12 may acquire various types of information from various parts of the mobile unit 200. For example, the information acquisition unit 12 may acquire various types of information via the communication unit 220 of the mobile unit 200 and the communication unit 120 of the monitoring support system 1. For example, the information acquisition unit 12 may acquire various types of information about the mobile unit 200 from the ECU of the mobile unit 200. The information acquisition unit 12 may also acquire various types of information about the autonomous driving of the mobile unit 200 from electronic equipment such as a computer that controls the autonomous driving of the mobile unit 200. Furthermore, the information acquisition unit 12 may acquire various types of information about the remote control of the mobile unit 200 from electronic equipment such as a computer that controls the remote control of the mobile unit 200.

[0040] The identification unit 13 may acquire information about an object that causes the mobile body 200 to stop. The identification unit 13 identifies an object included in the video based on the video acquired by the video acquisition unit 11. As described above, the video acquired by the video acquisition unit 11 may be the video captured by the imaging unit 210 of the mobile body 200. In one embodiment, if the mobile body 200 stops during automatic driving, the identification unit 13 may identify the object that caused the mobile body 200 to stop based on the video acquired by the video acquisition unit 11. The identification unit 13 may also identify the type of object and / or the speed at which the object is moving. In this case, the identification unit 13 may identify the object that caused the mobile body 200 to stop based on information received from the mobile body 200. Alternatively, the identification unit 13 may identify the object that caused the mobile body 200 to stop based on information received from the mobile body 200 regarding the cause of the mobile body 200's stop. Furthermore, the identification unit 13 may identify an object that causes the mobile body 200 to stop based on the information acquired by the information acquisition unit 12. The identification unit may also identify an object that causes the mobile body 200 to stop its automatic operation by having the observer specify it on the video displayed on the display unit 130.

[0041] The identification unit 13 may identify an object, for example, by image recognition. The identification unit 13 may identify an object, for example, by various techniques such as pattern matching. The identification unit 13 may identify an object based on information about various objects associated with images of various objects. The information about various objects associated with images of various objects may be stored in, for example, the storage unit 15, or it may be queried from, for example, an external server via the communication unit 120. The identification unit 13 may identify an object, for example, based on AI (Artificial Intelligence) technology. The identification unit 13 may identify an object, for example, based on machine learning (and even deep learning) technology.

[0042] For example, suppose a pedestrian attempts to cross in front of the mobile vehicle 200 while it is operating autonomously. In this case, it is expected that the mobile vehicle 200 will apply the brakes and stop based on the autonomous driving control, based on the detection of the pedestrian within a predetermined distance. In this case, the identification unit 13 may identify the pedestrian who attempted to cross in front of the mobile vehicle 200 as the object that caused the mobile vehicle 200 to stop. Alternatively, suppose a bicycle attempts to cross in front of the mobile vehicle 200 while it is operating autonomously. In this case, it is expected that the mobile vehicle 200 will apply the brakes and stop based on the detection of the bicycle within a predetermined distance. In this case, the identification unit 13 may identify the bicycle that attempted to cross in front of the mobile vehicle 200 as the object that caused the mobile vehicle 200 to stop. If the identification unit 13 cannot identify the object that caused the stop, it may provide the monitor with the image captured by the imaging unit 210 of the mobile vehicle 200, allowing the monitor to identify the cause of the stop. There may be multiple causes for the stop.

[0043] When the identification unit 13 identifies an object that causes the mobile body 200 to stop, the information acquisition unit 12 may acquire information about the object identified by the electronic equipment of the mobile body 200. Alternatively, when the identification unit 13 identifies an object that causes the mobile body 200 to stop, the image acquisition unit 11 may acquire the image captured by the imaging unit 210 of the mobile body 200, and the identification unit 13 may make the identification based on that image.

[0044] The video generation unit 14 generates and outputs a predetermined video based on the video acquired by the video acquisition unit 11. In one embodiment, the video generation unit 14 may generate a superimposed video. Here, a superimposed video is a video in which images of the object that causes the moving body 200 to stop, whose appearance changes over time, are superimposed on the video acquired by the video acquisition unit 11 at predetermined intervals. The superimposed video generated by the video generation unit 14 will be described further later. By visually observing the video acquired by the video acquisition unit 11 and / or the superimposed video generated by the video generation unit 14, the monitor of the monitoring facility can monitor the moving body 200.

[0045] The video generated by the video generation unit 14 may be output to the display unit 130. The display unit 130 can display the video generated by the video generation unit 14 by receiving the video from the video generation unit 14. By visually observing the video displayed on the display unit 130, the monitor of the monitoring facility can monitor the mobile object 200 and its surroundings.

[0046] The storage unit 15 may function as a memory for storing various types of information. For example, the storage unit 15 may store programs executed in the monitoring support device 10, and the results of processes executed in at least a part of the monitoring support device 10. The storage unit 15 may also function as a database for storing various types of information. Furthermore, the storage unit 15 may function as a work memory for at least a part of the monitoring support device 10. The storage unit 15 can be made of, for example, a semiconductor memory, but is not limited to this, and can be any storage device. For example, the storage unit 15 may be a storage medium such as a memory card inserted into the monitoring support device 10 according to one embodiment. Furthermore, the storage unit 15 may be the internal memory of a CPU used in at least a part of the monitoring support device 10, or it may be connected separately to at least a part of the monitoring support device 10.

[0047] In the monitoring support system 1 shown in Figure 2, the communication unit 120 and the display unit 130 are shown as separate functional units from the monitoring support device 10. However, the monitoring support device 10 according to one embodiment may include at least one of the communication unit 120 and the display unit 130.

[0048] Next, a description of how the monitoring support system 1 according to one embodiment is used will be given. Figures 3 and 4 are diagrams illustrating how the monitoring support system 1 according to one embodiment is used. Figures 3 and 4 schematically show how a monitor WA performs monitoring of a mobile object 200, etc., using the monitoring support system 1 installed, for example, in a monitoring facility.

[0049] As shown in Figures 3 and 4, a monitor WA using the monitoring support system 1 can monitor the mobile object 200, etc., by visually inspecting the display unit 130. Monitoring the mobile object 200, etc., may include not only the mobile object itself but also its surroundings. Figures 3 and 4 show how a monitor WA visually inspects the display unit 130. The display unit 130 can, for example, display video captured from the mobile object (video transmitted from each mobile object) and video output from the monitoring support device 10. Video output from the monitoring support device 10 may include, for example, various guide displays.

[0050] As shown in Figures 3 and 4, the display unit 130 may have multiple display screens, such as screen 130A, screen 130B, screen 130C, and screen 130D. For example, screens 130A, 130B, 130C, and 130D may each display images captured from different moving objects. For example, screens 130A, 130B, 130C, and 130D may each display different images captured from the same moving object. For example, screen 130A may be used to monitor the moving object 200. Also, screen 130B may be used to monitor another moving object different from the moving object 200. Screen 130C may be used to monitor yet another moving object different from the aforementioned moving object. In Figures 3 and 4, four screens, screens 130A, 130B, 130C, and 130D, are shown as an example. However, the display unit 130 may display one or any number of screens. By using such a display unit 130, the monitor WA can monitor multiple moving objects, etc.

[0051] The screen 130A shown in Figures 3 and 4 may be used, for example, to monitor a mobile object 200. Screen 130A displays images captured by an imaging unit 210 installed on the mobile object 200. As shown in Figures 3 and 4, screen 130A displays a portion (front) of the mobile object 200 as an image of the area in front of the mobile object 200 captured by the imaging unit 210, as well as the scenery in front of the mobile object 200. In addition, screen 130A displays a fork in the road in front of the mobile object 200, where a road goes straight and a road turns right (a T-junction).

[0052] For example, suppose a monitor WA, who is monitoring multiple mobile objects using the monitoring support system 1, is notified that a mobile object 200 that was operating autonomously has made an emergency stop. In this way, the notification that the mobile object 200 has made an emergency stop may come from the mobile object 200 itself, or from the mobile object 200's autonomous driving system, for example. In addition, the fact that the mobile object 200 has made an emergency stop may be brought to the attention of the monitor WA by displaying something on the display unit 130, for example. For example, when the mobile object 200 makes an emergency stop, the screen 130A may flash brightly for a moment, the screen 130A may be surrounded by a red frame, or an object to draw attention may be displayed somewhere on the screen 130A.

[0053] When the supervisor WA recognizes that the mobile vehicle 200 has made an emergency stop, they visually check the screen 130A to confirm the situation around the mobile vehicle 200. As shown in Figure 3, the display unit 130 shows a pedestrian P in front of the emergency-stopped mobile vehicle 200. By confirming this situation, the supervisor WA can understand that the cause of the emergency stop of the mobile vehicle 200 is the pedestrian P. For example, suppose pedestrian P was walking across the road. In this case, the supervisor WA can instruct the mobile vehicle 200 to stop until pedestrian P has finished crossing the road. For example, the supervisor WA may instruct the control unit of the mobile vehicle 200's automatic driving, or the driver of the mobile vehicle 200. Alternatively, the supervisor WA may take over the automatic driving of the mobile vehicle 200 via remote control and remotely operate the mobile vehicle 200 after pedestrian P has finished crossing the road.

[0054] While waiting for pedestrian P to finish crossing the road, the observer WA can monitor other moving objects besides the moving object 200. For example, observer WA can visually check the status of other moving objects besides the moving object 200 by looking at screen 130B or screen 130C. After some time has passed in this manner, suppose observer WA returns to checking the situation around the moving object 200. At this time, suppose pedestrian P has finished crossing the road. In this case, when observer WA visually checks screen 130A, as shown in Figure 4, pedestrian P is no longer displayed on screen 130A.

[0055] Because the observer WA was paying attention to other situations, it is conceivable that even after some time has passed and they look at screen 130A shown in Figure 4 again, they may not immediately recognize (or remember) that the mobile vehicle 200 had made an emergency stop due to the pedestrian P. In other words, even after looking at screen 130A shown in Figure 4, observer WA cannot immediately grasp whether it is safe for the mobile vehicle 200 to resume operation.

[0056] Furthermore, even if the monitoring WA visually inspects screen 130A shown in Figure 4, it is conceivable that they may not be able to determine whether the mobile vehicle 200 was going straight or turning right before it made an emergency stop. In other words, even if the monitoring WA visually inspects screen 130A shown in Figure 4, they may not be able to immediately grasp the originally planned course, for example, when taking over the autonomous driving of the mobile vehicle 200 via remote control.

[0057] Therefore, the monitoring support device 10 according to one embodiment provides support measures to reduce the burden on personnel (monitors) who monitor the traffic conditions of moving objects, from the viewpoint described above. Specifically, the monitoring support device 10 according to one embodiment allows monitors to easily understand the circumstances that caused a moving object to stop when it stops. In addition, the monitoring support device 10 according to one embodiment allows monitors to easily understand the planned path of a moving object before it stopped when it stops.

[0058] The operation of the monitoring support device 10 according to one embodiment will be described below. Figure 5 is a flowchart illustrating the operation of the monitoring support device 10 according to one embodiment. Figure 5 may typically show the operation of the monitoring support device 10, assuming that it is performed when the mobile body 200 is traveling on a road.

[0059] The operation shown in Figure 5 may be assumed to begin when the mobile body 200 is driving autonomously, as shown in Figure 1. At the start of the operation shown in Figure 5, the monitoring support device 10 is in a state where it can communicate with the mobile body 200 (its communication unit 220) via the communication unit 120. At the start of the operation shown in Figure 5, the imaging unit 210 of the mobile body 200 is assumed to be capturing an image of the area in front of the mobile body 200, and the image acquisition unit 11 is assumed to be acquiring the image captured by the imaging unit 210. At the start of the operation shown in Figure 5, the information acquisition unit 12 may be assumed to be acquiring various information about the mobile body 200.

[0060] When the operation shown in Figure 5 begins, the monitoring support device 10 determines whether the mobile body 200 has stopped (step S11). As a typical example, the mobile body 200 may stop if it detects an object, such as a pedestrian P, at a predetermined distance in front of it, as shown in Figure 6A. The monitoring support device 10 may determine whether the mobile body 200 has stopped based on the information acquired by the information acquisition unit 12. For example, the monitoring support device 10 may determine whether the mobile body 200 has stopped based on information acquired by the information acquisition unit 12 from the ECU of the mobile body 200. Alternatively, the monitoring support device 10 may determine whether the mobile body 200 has stopped based on information acquired by the information acquisition unit 12 from various functional units related to the automatic driving of the mobile body 200.

[0061] If the identification unit 13 determines in step S11 that the mobile body 200 has stopped, it identifies the object that caused the mobile body 200 to stop (step S12). In step S12, the identification unit 13 may identify the object that caused the mobile body 200 to stop based on the video acquired by the video acquisition unit 11. For example, in the situation shown in Figure 6A, the identification unit 13 may identify pedestrian P as the object that caused the mobile body 200 to stop. In step S12, if a function unit related to the automatic driving of the mobile body 200 identifies the object that caused the mobile body 200 to stop, the information acquisition unit 12 may acquire the result of this identification from the mobile body 200. Also, in step S12, once the object that caused the mobile body 200 to stop has been identified, the monitoring support device 10 may store the location of that object in the storage unit 15. Here, the position of the object may be the position in the video acquired by the video acquisition unit 11 where the object that causes the moving body 200 to stop exists.

[0062] In step S12, once the object is identified, the video generation unit 14 generates a video displaying an object indicating the location of the object that caused the mobile body 200 to stop (step S13). In step S13, the object indicating the location of the object that caused the mobile body 200 to stop may be any object shown to alert the observer WA that the mobile body 200 stopped because of an object located at that location. In other words, by visually inspecting this object, the observer WA can easily understand that the mobile body 200 stopped because of an object located at that object's location. Thus, in one embodiment, when the identification unit 13 identifies an object that caused the mobile body 200 to stop, the video generation unit 14 may generate a video displaying an object indicating the location of that object.

[0063] When the monitoring support device 10 outputs an image displaying the object generated in step S13, the display unit 130 may display a screen such as the one shown in screen 130A of Figure 6B. In screen 130A of Figure 6B, an object OB1 is displayed that indicates the position of the object (pedestrian P) that caused the moving body 200 to stop. This object OB1 may have any shape, any color, and any display manner. In order to draw the attention of the monitor WA to the cause of the moving body 200's stoppage, object OB1 may be shown by surrounding the pedestrian P with a frame such as a red rectangle.

[0064] Following step S13, the monitoring support device 10 determines whether or not there is still an object that could cause the mobile body 200 to stop in the video acquired by the video acquisition unit 11 (step S14). In step S14, for example, the identification unit 13 may determine whether or not it has still identified an object that could cause the mobile body 200 to stop in the video acquired by the video acquisition unit 11.

[0065] If the object still exists in step S14, the monitoring support device 10 stores the position of the object in, for example, the storage unit 15 at predetermined intervals (step S15). Here, the position of the object may be the position in the video acquired by the video acquisition unit 11 where the object that causes the moving body 200 to stop exists. In step S15, the predetermined interval may be any interval, for example, every second. For example, the predetermined interval may be an interval long enough to grasp the movement (change in position) of the object that causes the moving body 200 to stop. The predetermined interval may be determined to a different value depending on, for example, the type of object and / or the speed at which the object moves. For example, if the object is moving at high speed, the predetermined interval may be shorter than that for an object moving at low speed. For example, if the object identified by the identification unit is a fast-moving object (e.g., a vehicle, motorcycle, etc.), the predetermined interval may be shorter than that for a slow-moving object (pedestrian).

[0066] If the object's position is stored in step S15, the monitoring support device 10 may return to step S13 and repeat the operation. If, at the time of performing the operation in step S13 after going through step S15, the position of the object causing the mobile body 200 to stop has moved, the video generation unit 14 may generate a video displaying the object showing its latest position at that time.

[0067] In this way, through the loop from step S13 to step S15, the monitoring support device 10 can output an image like the screen 130A shown in Figures 6B to 6E. As shown in Figures 6B to 6E, in screen 130A, pedestrian P is walking across the road from right to left. At this time, the monitoring support device 10 displays object OB1 on screen 130A, following the movement of pedestrian P, in order to indicate the position of the moving pedestrian P (the object that causes the moving body 200 to stop). In this way, the image generation unit 14 may generate an image in which object OB1 follows the movement of the object that causes the moving body 200 to stop.

[0068] By visually observing object OB1 (and the video of pedestrian P) on screen 130A shown in Figures 6B to 6E, the observer WA can easily determine that the object causing the moving object 200 to stop is still present.

[0069] Figure 6F shows the situation at the next point in time following the situation shown in Figures 6B to 6E. Figure 6F shows that after pedestrian P has moved from Figure 6B to Figure 6E, pedestrian P has moved again, and therefore pedestrian P is no longer present in the video acquired by the video acquisition unit 11.

[0070] As described above, for example, if a monitor WA attempts to resume monitoring mobile body 200 after dealing with another mobile body different from mobile body 200, even if they visually view screen 130A as shown in Figure 6F, they cannot easily identify the object that caused mobile body 200 to stop. Also, for example, if a monitor WA first visually views screen 130A as shown in Figure 6F when starting mobile body 200, they cannot easily grasp the current status of mobile body 200. Furthermore, if a monitor WA takes over the operation of mobile body 200 and remotely controls it, even if they visually view screen 130A as shown in Figure 6F, they cannot easily determine whether mobile body 200 was scheduled to move forward or turn right before stopping.

[0071] Therefore, in one embodiment, the monitoring support device 10 performs the operation in step S16 if it is determined in step S14 that the object no longer exists. In step S14, if the object that caused the mobile body 200 to stop is not captured in the video acquired by the video acquisition unit 11, the monitoring support device 10 may determine that the object no longer exists. Also, in step S14, if the identification unit 13 no longer identifies the object that caused the mobile body 200 to stop in the video acquired by the video acquisition unit 11, it may determine that the object no longer exists.

[0072] In step S16, the video acquisition unit 11 may generate a video displaying an object in the current video that indicates the position of the object that caused the mobile body 200 to stop. Here, the position of the object that caused the mobile body 200 to stop may be, for example, the position of the object that was first recognized in step S13. That is, the object indicating the position of the object in step S16 may be an object that indicates the position where the object that caused the mobile body 200 to stop was first identified.

[0073] When the monitoring support device 10 outputs an image displaying the object generated in step S16, the display unit 130 may display, for example, the object OB2 shown in screen 130A of Figure 7. This object OB2 may have any shape, any color, and any display mode. Furthermore, object OB2 may be an object with a different display mode from object OB1 (Figures 6B to 6E) mentioned in step S13. Object OB2 may be an object that indicates the original location of the object that caused the moving body 200 to stop, while also indicating that the object no longer exists on the screen. To draw the attention of the monitor WA that the cause of the moving body 200's stoppage no longer exists, object OB2 may be shown, for example, by surrounding the pedestrian P with a blue or green rectangle or other frame.

[0074] Thus, the video generation unit 14 may generate a video in which object OB2, which indicates the position of the object that caused the moving body 200 to stop, is displayed in the current video. Alternatively, the video generation unit 14 may perform the above operation when the identification unit 13, which had identified the object that caused the moving body 200 to stop, stops identifying that object, based on the video acquired by the video acquisition unit 11.

[0075] If an image displaying object OB2 is generated in step S16, the image generation unit 14 performs the operation in step S17. In step S17, the image generation unit 14 changes the display mode of the object and generates an image showing the position of the object at predetermined time intervals from the time the identification unit 13 identified the object that caused the moving body 200 to stop until the present (step S17). The image generation unit 14 may generate an image showing the position of the object at predetermined time intervals from the time the identification unit 13 identified the object that caused the moving body 200 to stop until a predetermined period has elapsed. The image generation unit 14 may also generate an image showing the position of the object at predetermined time intervals from the time the identification unit 13 identified the object that caused the moving body 200 to stop until it is no longer present on the screen. The image generation unit 14 may also generate an image showing the position of the object at predetermined time intervals from the time the object identified by the identification unit 13 as causing the moving body 200 to stop until a predetermined period has elapsed. The video generation unit 14 may generate video showing the position of an object identified by the identification unit 13 as the cause of stopping the mobile body 200, at predetermined intervals during a predetermined period of time when the object is present on the screen. Here, the position of the object at predetermined intervals may be, for example, the position of the object stored in the storage unit 15 in step S15. That is, in step S17, the video generation unit 14 generates video showing the position of the object at predetermined intervals, as shown in Figures 6B to 6E. Here, if the video of the object that causes the mobile body 200 to stop is overlaid on video where the object no longer exists, it may become difficult for the observer WA to distinguish whether the video of the object is real or virtual. Therefore, in step S17, the video generation unit 14 may perform appropriate image processing, such as making the object semi-transparent, before generating video showing the position of the object at predetermined intervals. For example, the video generation unit 14 may overlay images with a different display mode than the images of the object (pedestrian P) shown in Figures 6B to 6E at each position of the object as shown in Figures 6B to 6E.

[0076] For example, the video generation unit 14 may display the video of pedestrian P in a semi-transparent manner, as shown in the video Q of pedestrian P in Figure 7. Alternatively, the video generation unit 14 may display the video in a different manner from the video Q of pedestrian P shown in Figure 7, for example, by showing only the outer frame of the video of pedestrian P, or by graying out the video of pedestrian P.

[0077] Thus, the video generation unit 14 may generate a superimposed video by overlaying an image of the object causing the mobile body 200 to stop, with a modified display, onto the video acquired by the video acquisition unit 11 at predetermined intervals. Alternatively, the video generation unit 14 may generate a superimposed video by overlaying, for example, a semi-transparent image of the object causing the mobile body 200 to stop onto the current video acquired by the video acquisition unit 11. The video generation unit 14 may also generate such a superimposed video when the identification unit 13, which had been identifying the object causing the mobile body 200 to stop based on the video acquired by the video acquisition unit 11, ceases to identify the object.

[0078] By visually inspecting screen 130A as shown in Figure 7, the supervisor WA can easily understand that the object that caused the mobile unit 200 to stop was originally located at the position of object OB2. Furthermore, by visually inspecting object OB1 as shown in Figures 6B to 6E, the supervisor WA can easily understand that the object that caused the mobile unit 200 to stop is still present on screen 130A. In other words, by visually inspecting object OB1 as shown in Figures 6B to 6E, the supervisor WA can easily understand that it is dangerous to restart the operation of the mobile unit 200 at this time.

[0079] On the other hand, by visually inspecting object OB2 shown in Figure 7, rather than object OB1 shown in Figures 6B to 6E, the observer WA can easily understand that the object that caused the mobile vehicle 200 to stop is no longer present on screen 130A. Furthermore, by visually inspecting the pedestrian image Q shown in Figure 7, the observer WA can easily understand how the object that is no longer present on screen 130A came to be absent from screen 130A. In this way, by visually inspecting screen 130A as shown in Figure 7, the observer WA can easily understand that it is safe for the mobile vehicle 200 to resume operation.

[0080] In one embodiment, the video generation unit 14 may perform the operation of step S18 after the operation of step 17. That is, in step S18, the video generation unit 14 may generate a video in which an object indicating the originally planned path is displayed.

[0081] For example, if the mobile vehicle 200 is being driven autonomously, the information acquisition unit 12 can obtain information about the planned route before the mobile vehicle 200 stops in step S11. Similarly, if the mobile vehicle 200 is driving according to the car navigation system, the information acquisition unit 12 can also obtain information about the planned route before the mobile vehicle 200 stops in step S11.

[0082] Therefore, if it is determined in step S14 that the object no longer exists, the video generation unit 14 may display an object indicating the path that was planned before the time the moving body 200 stopped (step S18).

[0083] Figure 8 shows the screen 130A shown in Figure 7, with an object OB3 further displayed, indicating the planned path of the mobile body 200 from the moment it stopped. As shown in Figure 8, object OB3 may be displayed in a manner that makes it easy to understand the path the mobile body 200 should take. Object OB3, as an object indicating a path, may be of any size, shape, color, and display manner. By visually inspecting object OB3 on screen 130A shown in Figure 7, the supervisor WA can easily understand the path to take when resuming operation of the stopped mobile body 200.

[0084] Thus, the video generation unit 14 may generate a video showing object OB3, which indicates the planned path of the moving body 200. The video generation unit 14 may also perform the above operation when the identification unit 13, which had identified an object causing the moving body 200 to stop, ceases to identify that object based on the video acquired by the video acquisition unit 11.

[0085] As described above, the monitoring support device 10 according to one embodiment allows the monitor to easily understand the circumstances that caused the moving object to stop when it stops. Furthermore, the monitoring support device 10 according to one embodiment allows the monitor to easily understand the planned path of the moving object before it stopped when it stops. Therefore, the monitoring support device 10 according to one embodiment can reduce the burden on personnel who monitor the traffic conditions of moving objects.

[0086] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art will find it easy to make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are included within the scope of this disclosure. For example, the functions included in each component or step can be rearranged in a logically consistent manner, and multiple components or steps can be combined into one or divided. While embodiments relating to this disclosure have been described primarily in terms of apparatus, embodiments relating to this disclosure can also be realized as methods including steps performed by each component of the apparatus. Embodiments relating to this disclosure can also be realized as methods, programs, or storage media recording programs executed by a processor in the apparatus. These should also be understood to be included within the scope of this disclosure.

[0087] Furthermore, the embodiments described above may be implemented, for example, as a control method for electronic equipment such as the monitoring support device described above. Moreover, the embodiments described above may be implemented, for example, as a system including the monitoring support device described above. For example, the embodiments described above may be implemented as a monitoring support system including the monitoring support device 10, a communication unit 120, and a display unit 130. In this case, the display unit 130 may display the image generated by the image generation unit 14 of the monitoring support device 10. The monitoring support system may also further include an imaging unit 210 that captures images, either in place of or together with the communication unit 120. Furthermore, the embodiments described above may be implemented, for example, as a program executed by the monitoring support device or monitoring support system described above. Moreover, the embodiments described above may be implemented, for example, as a recording medium that records the program executed by the monitoring support device or monitoring support system described above, i.e., a computer-readable recording medium.

[0088] When the above-described embodiment is implemented as a program, the program is: On the computer, A video acquisition step in which video is captured from a moving object, An identification step to obtain information about the object that causes the moving body to stop, A video generation step that generates a superimposed video in which an image showing the position of the object over a predetermined period of time is displayed on the video acquired in the video acquisition step, This can be used to execute the following: [Explanation of symbols]

[0089] 1. Monitoring and Support System 10 Monitoring support equipment 11. Video Acquisition Unit 12 Information acquisition department 13 Identification Unit 14. Video Generation Unit 15 Storage section 120 Communications Department 130 Display section 200 Mobile Units 210 Imaging Unit 220 Communications Department 300 Remote Control Devices

Claims

1. A video acquisition unit that acquires images captured from a moving object, An identification unit that acquires information about the object causing the moving object to stop from the video acquired by the video acquisition unit, A video generation unit generates an image indicating the position of an object based on one or more positions of the object acquired over a predetermined period, and displays the generated image on the current video acquired by the video acquisition unit to generate a superimposed image. A monitoring and support device equipped with the following features.

2. The superimposed image is an image obtained by the image acquisition unit, on which an image showing the position of the object from the time the identification unit identifies the object until the predetermined period has elapsed is displayed.

3. The monitoring support device according to claim 1, wherein the video generation unit generates an image showing the position of the object until the object is no longer included in the video acquired by the video acquisition unit.

4. The monitoring support device according to claim 1, wherein the image generation unit generates an image indicating the position of the object based on the position of the object acquired at predetermined time intervals during the predetermined period.

5. The monitoring support device according to claim 4, wherein the predetermined time is determined according to the type of object or the movement speed of the object.

6. The monitoring support device according to claim 1, wherein the image generation unit generates a semi-transparent image of the object as an image indicating the position of the object.

7. The mobile body further comprises an information acquisition unit that acquires information about the planned path of the mobile body from the mobile body, When the identification unit, which had been identifying the object based on the image acquired by the image acquisition unit, ceases to identify the object, The monitoring support device according to claim 1, wherein the video generation unit generates a video showing an object indicating the planned path of the moving object.

8. A communication unit that receives video images captured from a moving object, A video acquisition unit that acquires video received by the aforementioned communication unit, An identification unit that acquires information about the object causing the moving object to stop from the video acquired by the video acquisition unit, A video generation unit generates an image indicating the position of an object based on one or more positions of the object acquired over a predetermined period, and displays the generated image on the current video acquired by the video acquisition unit to generate a superimposed image. A display unit that displays the video generated by the aforementioned video generation unit, A monitoring and support system, including a monitoring support system.

9. On the computer, A video acquisition step in which video is captured from a moving object, An identification step is to obtain information about the object that caused the moving object to stop from the video acquired in the video acquisition step, A video generation step that generates an image indicating the position of an object based on one or more positions of the object acquired over a predetermined period, and displays the generated image on the current video acquired in the video acquisition step to generate a superimposed image, A program that executes something.

Images