Remote monitoring control device and remote monitoring system
The remote monitoring system uses omnidirectional cameras and a head-mounted display to align the operator's view with the driver's perspective, addressing the limitations of fixed-camera systems by providing a dynamic field-of-view image and essential vehicle information, thereby improving situational awareness and safety.
Patent Information
- Application Number
- JP2023090170
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-12
AI Technical Summary
Conventional remote monitoring systems for railway vehicles provide images from a fixed camera direction, limiting the operator's view to a specific direction, which differs from the driver's field of view, making it difficult to understand the surrounding situation accurately.
A remote monitoring system that includes omnidirectional cameras installed on the train, a head-mounted display for the operator, and a control device that selects and generates images based on the operator's line-of-sight direction, providing a field-of-view image that matches the driver's perspective, along with vehicle information and abnormality notifications.
Enables the operator to view the surroundings in alignment with their line of sight, including both forward and rear views, and receive critical vehicle information and alerts, enhancing situational awareness and safety.
Smart Images

Figure 2024172446000001_ABST
Abstract
Description
[Technical field]
[0001] An embodiment of the present invention relates to a remote monitoring control device and a remote monitoring system. [Background technology]
[0002] Conventionally, there is a remote monitoring system for railway vehicles that remotely monitors the surrounding conditions of trains such as railways. In conventional remote monitoring systems, a video transmission device transmits video captured by a camera with a fixed shooting direction so as to capture the inside of the track in the direction of travel of the train to a remote monitoring room, and a remote monitoring control device displays the video transmitted from the video transmission device on a display device installed in the monitoring room. In this way, an operator in the monitoring room monitors the state of the inside of the track in the direction of travel of the train.
[0003] However, in conventional remote monitoring systems, the shooting direction of the camera is fixed, so the image displayed on the display device in the monitoring room is an image from a specific direction, and the operator in the monitoring room cannot see images from any direction other than the specified direction. A driver in the driver's seat of an actual train can ensure safety and perform driving operations according to the situation by visually checking not only the direction in which the train is moving but also the surrounding conditions of the train. In other words, conventional remote monitoring systems have a problem in that the image that the operator in the monitoring room can see in the remote monitoring room differs from the image that the driver can see in the field of view of the driver who is actually riding on the train, so it is sometimes difficult for the operator in the monitoring room to grasp the situation around the train. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2002-234440 A Summary of the Invention [Problem to be solved by the invention]
[0005] The present invention has been made in consideration of the above problems, and has an object to provide a remote monitoring control device and a remote monitoring system that make it easy to grasp the surrounding situation. [Means for solving the problem]
[0006] According to an embodiment, the remote monitoring and control device includes a first wireless communication unit, a second wireless communication unit, and a control unit. The first wireless communication unit receives video data captured by an omnidirectional camera installed on a train. The second wireless communication unit communicates with a head mounted display. The control unit determines the line of sight direction of an operator wearing the head mounted display, and displays on the head mounted display a field of view image generated based on the line of sight direction of the operator from the video data captured by the omnidirectional camera received by the first wireless communication unit. [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a remote monitoring system including a remote monitoring and controlling device according to an embodiment. [Diagram 2] FIG. 2 is a block diagram for explaining an example of the configuration of an in-vehicle system in the remote monitoring system according to the embodiment. [Diagram 3] FIG. 3 is a block diagram showing a configuration example of a remote monitoring control device, a head mounted display, and a pseudo cab in the remote monitoring system according to the embodiment. [Figure 4] FIG. 4 is a flowchart for explaining an example of the operation of the remote monitoring and control device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Hereinafter, an embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of a remote monitoring system 1 including a remote monitoring and control device 13 according to an embodiment. As shown in FIG. 1, the remote monitoring system 1 includes an in-vehicle system 2, a remote monitoring control device 13, a head mounted display (HMD) 14, and a pseudo cab 15.
[0009] The in-car system 2 is a system composed of various devices installed in the cars that make up the train. The in-car system 2 is composed of various devices including a wireless communication device 21 (see FIG. 2) that communicates with a remote monitoring and control device 13. The in-car system 2 also has multiple omnidirectional cameras 11A, 11B installed on the train. In the configuration example shown in FIG. 1, the omnidirectional camera 11A is installed at the front of the train, and the omnidirectional camera 11B is installed at the rear of the train. Here, the direction moving forward from the front is the traveling direction of the train, and the opposite direction is the reverse direction.
[0010] Omnidirectional cameras 11A and 11B are cameras that capture omnidirectional images from their installation positions. Omnidirectional cameras 11A and 11B are installed at positions where they can capture images of the surroundings outside the vehicle. In this embodiment, omnidirectional camera 11A installed at the front of the train is referred to as the first omnidirectional camera. First omnidirectional camera 11A captures omnidirectional images centered on the front of the train, including the front of the train. Omnidirectional camera 11B installed at the rear of the train is referred to as the second omnidirectional camera. Second omnidirectional camera 11B captures omnidirectional images centered on the rear of the train, including the rear of the train.
[0011] The in-car system 2 processes video data captured by the omnidirectional cameras 11A, 11B using a video processing device 22 (see FIG. 2) installed inside the car, and then transmits the processed video data to the ground-side remote monitoring and control device 13 via the wireless communication device 21. The in-car system 2 also transmits vehicle information including various information indicating the status of the cars constituting the train to the remote monitoring and control device 13. Furthermore, the in-car system 2 acquires information indicating commands related to train operation input by an operator at a pseudo cab 15 connected to the remote monitoring and control device 13 from the remote monitoring and control device 13, and supplies the information to a main controller 24 (see FIG. 2) that outputs commands related to train operation.
[0012] The remote monitoring and control device 13 is installed on the ground away from the train. The remote monitoring and control device 13 communicates via wireless communication with the in-car system 2 installed in the cars that make up the train. The head-mounted display 14 is worn by an operator on the ground away from the train. The head-mounted display 14 communicates via wireless communication with the remote monitoring and control device 13. The pseudo cab 15 is a device that allows the operator to perform operations related to the operation of the train. The pseudo cab 15 is provided so as to be able to communicate with the remote monitoring and control device 13 via wired or wireless communication. In this embodiment, the remote monitoring and control device 13, the head-mounted display (HMD) 14, and the pseudo cab 15 are provided in a monitoring room on the ground where an operator is stationed.
[0013] The remote monitoring control device 13 acquires video data captured by the omnidirectional cameras 11A and 11B from the in-car system 2 of the train via wireless communication. The remote monitoring control device 13 generates display data to be displayed on the head-mounted display 14 based on the video data acquired from the in-car system 2, and supplies the display data to the head-mounted display 14. The remote monitoring control device 13 selects an image captured by one of the two omnidirectional cameras based on the train's traveling direction, and further generates an image to be the operator's field of view image based on the line of sight direction of the operator wearing the head-mounted display 14.
[0014] The head mounted display 14 displays an image based on the display data supplied from the remote monitoring and control device 13. The head mounted display 14 displays an image so that the image viewed by the operator who wears the display becomes the image in the field of view of the operator. The head mounted display 14 also has sensors such as an acceleration sensor, and outputs information detected by the sensors to the remote monitoring and control device as information (gaze information) indicating the movement of the operator's head or gaze. The head mounted display 14 may also have a function of switching between a transparent display mode and a non-transparent display mode. For example, when the head mounted display 14 is in the transparent display mode, it can display a display image and various information in a state in which the operator wearing the display device can directly view the pseudo cab 15, etc.
[0015] Furthermore, the remote monitoring control device 13 receives, in addition to the video data captured by the two omnidirectional cameras 11A and 11B, vehicle information such as various indicator light information and brake pressure, and main control information such as the notch of the main controller, from the in-vehicle system 2. The remote monitoring control device 13 may display not only the display data generated from the video data, but also the vehicle information, main control information, and the like on the head mounted display 14.
[0016] Furthermore, the remote monitoring control device 13 detects abnormalities around the train by analyzing the video data captured by the omnidirectional cameras 11A and 11B obtained from the in-car system 2. For example, the remote monitoring control device 13 may detect obstacles on the track from the video data captured by the omnidirectional cameras 11A and 11B, or detect abnormal conditions on the platform before and after passing through a station. When an abnormality is detected around the train, the remote monitoring control device 13 may display a guide or the like on the head-mounted display 14 to alert the train driver to the abnormal condition.
[0017] Furthermore, the remote monitoring control device 13 switches the omnidirectional camera that captures the image to be displayed on the head mounted display 14 depending on the traveling direction of the train. For example, when the train is moving forward, the remote monitoring control device 13 sets the image captured by the first omnidirectional camera 11A installed at the front as the display data to be displayed on the head mounted display 14. When the train is moving backward, the remote monitoring control device 13 sets the image captured by the second omnidirectional camera 11B installed at the rear as the display data to be displayed on the head mounted display 14.
[0018] The in-car system 2 may select one of the omnidirectional cameras 11A, 11B according to the traveling direction of the train, and transmit the video data captured by the selected omnidirectional camera to the remote monitoring control device 13. Here, the remote monitoring control device 13 receives the video data captured by each omnidirectional camera, and selects video data according to the traveling direction of the train, mainly from the information of the main controller acquired from the in-car system 2. This allows the remote monitoring control device 13 to instantly display an image of the rearward direction (rear) of the train on the head-mounted display 14, even if the train reverses due to overrunning at a stop station.
[0019] The pseudo cab 15 is a device that imitates an actual cab. The pseudo cab 15 is equipped with an operation unit that allows an operator in a monitoring room to input main control instructions to the train, and a display unit that displays vehicle information, etc. Information (main control information) indicating main control instructions (powering or brake notch, forward or reverse driving operations) inputted at the operation unit of the pseudo cab 15 is transmitted to the in-car system 2 (train main controller) via the remote monitoring and control device 13. In addition, the display unit of the pseudo cab 15 displays vehicle information such as various indicator light information and brake pressure in the train obtained via the remote monitoring and control device 13.
[0020] Next, a configuration of the in-vehicle system 2 in the remote monitoring system 1 according to the embodiment will be described. FIG. 2 is a block diagram showing an example of the configuration of the in-vehicle system 2 in the remote monitoring system 1 according to the embodiment. As shown in FIG. 2, the in-vehicle system 2 includes a first omnidirectional camera 11A, a second omnidirectional camera 11B, a wireless communication device 21, a video processing device 22, a vehicle information device 23, a main controller 24, and the like.
[0021] The omnidirectional camera (first omnidirectional camera) 11A captures omnidirectional video centered on the front of the train in which the in-car system 2 is installed. The omnidirectional camera (second omnidirectional camera) 11B captures omnidirectional video centered on the rear of the train in which the in-car system 2 is installed. The first omnidirectional camera 11A and the second omnidirectional camera 11B are connected to an image processing device 22 installed in the vehicle. The images from the first omnidirectional camera 11A and the second omnidirectional camera 11B are transmitted to the image processing device 22.
[0022] The video processing device 22 is a device that processes the video captured by the first omnidirectional camera 11A and the second omnidirectional camera 11B. The video processing device 22 is connected to the first omnidirectional camera 11A, the second omnidirectional camera 11B, and the wireless communication device 21. The video processing device 22 acquires data of the video captured by the first omnidirectional camera 11A and the video captured by the second omnidirectional camera 11B. The video processing device 22 performs a predetermined image processing on the acquired video data. For example, the video processing device 22 performs a process of encoding the video data acquired from the first omnidirectional cameras 11A and 11B for wireless communication. The video processing device 22 transmits the video data that has been subjected to image processing such as encoding to the remote monitoring control device 13 via the wireless communication device 21.
[0023] The wireless communication device 21 includes a communication interface for wireless communication with the remote monitoring and control device 13. The wireless communication device 21 transmits video data processed by the video processing device 22 to the remote monitoring and control device 13 via wireless communication. The wireless communication device 21 is also connected to a vehicle information device and a main controller. The wireless communication device 21 transmits vehicle information output from the vehicle information device 23 to the remote monitoring and control device 13 via wireless communication. The wireless communication device 21 also receives main control information corresponding to the driving operation input by the operator at the pseudo cab 15 from the remote monitoring and control device 13, and supplies the received main control information to the main controller 24.
[0024] The vehicle information device 23 manages various vehicle information related to the vehicle, such as speed, notch, and malfunction information. The vehicle information device 23 is connected to the wireless communication device 21 in the in-vehicle system 2, and transmits the vehicle information to the remote monitoring and control device 13 via the wireless communication device 21.
[0025] The master controller 24 outputs control commands corresponding to the operation of power running or brake notch, forward or reverse to the drive control device of the vehicle. In the in-vehicle system 2 according to this embodiment, the master controller 24 not only outputs control commands corresponding to the driving operation inputted at the driver's seat installed in the vehicle, but also has a function of outputting control commands corresponding to master control information received from the remote monitoring control device 13.
[0026] Next, a configuration of the remote monitoring control device 13 in the remote monitoring system 1 according to the embodiment will be described. FIG. 3 is a block diagram showing a configuration example of the remote monitoring control device 13, the head mounted display (HMD) 14, and the pseudo cab 15 in the remote monitoring system 1 according to the embodiment. As shown in FIG. 3, the remote monitoring and control device 13 includes a control unit 31, a wireless communication unit 32, and a wireless communication unit 33.
[0027] The control unit 31 is configured as a computer equipped with a processor such as a CPU, a memory, and various interfaces. The processor is, for example, a CPU. The memory is configured as a storage device such as a nonvolatile memory such as a ROM, a volatile memory such as a RAM, and a rewritable nonvolatile memory such as an NVM. The control unit 31 realizes various processing functions by the processor executing programs stored in the memory.
[0028] The wireless communication unit 32 is a communication interface (first wireless communication unit) that performs wireless communication with the wireless communication device 21 of the in-vehicle system 2. The wireless communication unit 32 may be any unit capable of data communication with the in-vehicle system 2, and may perform data communication with the in-vehicle system 2 via a separately provided communication facility. The wireless communication unit 32 receives video data captured by the omnidirectional cameras 11A and 11B from the in-vehicle system 2. In this embodiment, the wireless communication unit 32 receives video data that has been subjected to image processing such as encoding by the video processing device of the in-vehicle system 2. The wireless communication unit 32 also receives vehicle information and the like from the vehicle information device 23 that is sent from the wireless communication device 21 of the in-vehicle system 2.
[0029] The wireless communication unit 33 is a communication interface (second wireless communication unit) for wireless communication with the head mounted display 14. The wireless communication unit 33 may include a communication interface corresponding to a wireless communication method equipped in the head mounted display 14. The wireless communication unit 33 transmits data to be displayed on the head mounted display 14, which is output from the control unit 31, to the head mounted display 14. The wireless communication unit 33 also receives information (gaze detection information) for determining (estimating) the line of sight direction of the operator detected by various sensors installed in the head mounted display 14 from the head mounted display 14, and supplies the received gaze detection information to the control unit 31.
[0030] The communication interface 34 is an interface (second communication interface) for connecting to the pseudo cab 15. The communication interface 34 may be an interface that communicatively connects to the pseudo cab 15 via a wire, or may be an interface that communicatively connects to the pseudo cab 15 wirelessly.
[0031] In the exemplary configuration shown in FIG. 3, the control unit 31 has processing functions such as a video decoding unit 35, a line-of-sight determination unit 36, a video analysis unit 37, and a display video generation unit 38. The video decoding unit 35 decodes the encoded video data acquired from the in-vehicle system 2. That is, the video decoding unit 35 performs a process of restoring the video data captured by the omnidirectional cameras 11A and 11B from the data received by the wireless communication unit 33.
[0032] The line-of-sight determination unit 36 determines the line-of-sight direction of the operator wearing the head mounted display 14 based on line-of-sight detection information acquired from the head mounted display 14, and supplies information indicating the operator's line-of-sight direction to the display image generation unit 38. Note that the head mounted display 14 may determine the operator's line-of-sight direction based on various sensor information. In this case, the line-of-sight determination unit 36 may acquire information indicating the line-of-sight direction from the head mounted display 14 via the wireless communication unit 33, and supply the acquired information indicating the line-of-sight direction to the display image generation unit 38.
[0033] The video analysis unit 37 detects abnormalities around the train based on the video data captured by the omnidirectional cameras 11A and 11B. For example, when the train is moving forward, the video analysis unit 37 can not only detect an obstacle in front of the train as an abnormal state from the video data (first video data) captured by the first omnidirectional camera 11A, but also detect an abnormal state occurring behind the train from the video data (second video data) captured by the second omnidirectional camera 11B. This allows the video analysis unit 37 to detect, for example, a person or object that has fallen off a platform of a station that the train has passed after passing through the station as an abnormal state. Furthermore, the video analysis unit 37 can detect not only obstacles on the train's track, but also weather conditions around the train from the video captured by the omnidirectional cameras, and detect objects approaching the train's track from outside the track.
[0034] The display image generating unit 38 generates a display image to be displayed on the head mounted display 14 from the image data of the omnidirectional cameras 11A and 11B. The display image generating unit 38 selects either the image data of the first omnidirectional camera 11A or the image data of the second omnidirectional camera 11B as the image data to be displayed on the head mounted display 14 according to the traveling direction of the train. In addition, the display image generating unit 38 generates a display image (field of view image) that is an image of the field of view of the operator wearing the head mounted display 14 from the image data based on the line of sight direction of the operator wearing the head mounted display 14 determined by the line of sight determination unit 36.
[0035] Furthermore, the display image generating unit 38 causes the head mounted display 14 to display vehicle information acquired from the in-car system 2 together with a display image generated based on the image data from the omnidirectional cameras 11A and 11B. Furthermore, the display image generating unit 38 causes the head mounted display 14 to display guidance to attract the operator's attention when the image analyzing unit 37 detects an abnormal condition inside the train (car) and around the train.
[0036] Next, the configuration of the head mounted display 14 in the remote monitoring system 1 according to the embodiment will be described. As shown in FIG. 3, a head mounted display (HMD) 14 in a remote monitoring system 1 according to the embodiment includes a control unit 40, a wireless communication unit 41, a display unit 42, a line of sight detection unit 43, and the like.
[0037] The control unit 40 is composed of a computer equipped with a processor, memory, and various interfaces. The processor is an arithmetic processing unit that executes programs such as a CPU. The memory is composed of various storage devices such as a ROM, RAM, and NVM. The control unit 40 realizes control of each part and various processing functions by the processor executing the programs stored in the memory.
[0038] The wireless communication unit 41 is a communication interface for wireless communication with the remote monitoring control device 13. The wireless communication unit 41 may be, for example, a communication method that enables communication with the remote monitoring control device 13, and may be, for example, Bluetooth (registered trademark). The wireless communication unit 41 receives display image data from the remote monitoring control device 13, and sends out to the remote monitoring control device 13 gaze detection information detected by various sensors serving as the gaze detection unit 43.
[0039] The display unit 42 displays an image that can be viewed by an operator wearing the device on his / her head. The display unit 42 displays the display image received from the remote monitoring control device 13 via the wireless communication unit 41 as an image in the operator's field of view.
[0040] The gaze detection unit 43 has a motion sensor that detects the movement of the operator's head, a sensor that detects the direction of the gaze, etc. The gaze detection unit 43 outputs gaze detection information indicating the gaze of the operator to be sent to the remote monitoring control device 13 by the wireless communication unit 41. The gaze detection unit 43 may also determine the gaze direction of the operator based on signals detected by various sensors, and output the determination result of the gaze direction as gaze detection information.
[0041] Next, the configuration of the pseudo cab 15 in the remote monitoring system 1 according to the embodiment will be described. As shown in FIG. 3, the pseudo cab 15 in the remote monitoring system 1 according to the embodiment includes a control unit 51, a display unit 52, an operation unit 53, a communication interface 54, and the like. The control unit 50 is composed of a computer equipped with a processor, memory, and various interfaces. The processor is an arithmetic processing unit that executes programs such as a CPU. The memory is composed of various storage devices such as a ROM, RAM, and NVM. The control unit 40 realizes control of each unit and various processing functions by the processor executing the programs stored in the memory.
[0042] The display unit 52 displays vehicle information, etc. For example, the display unit 52 is configured with a display that displays various vehicle information such as various indicator light information and brake pressure. The display unit 52 may also be configured with a display that imitates indicator lights and instruments provided in the driver's cab of an actual vehicle.
[0043] The operation unit 53 is composed of an operation device for inputting driving operations for the vehicle. The operator performs driving operations such as power running or brake notch, forward driving or reverse driving operations using the operation unit 53. The control unit 50 functions as a main control setting unit that transmits main control information corresponding to the driving operations using the operation unit 53 to the in-vehicle system 2 (main controller of the vehicle) via the remote monitoring and control device 13.
[0044] The communication interface 54 is an interface (first communication interface) for connecting to the remote monitoring control device 13. The communication interface 54 may be an interface for connecting to the remote monitoring control device 13 via a wire, or may be an interface for connecting to the remote monitoring control device 13 wirelessly.
[0045] Next, the operation of the remote monitoring control device 13 in the remote monitoring system 1 according to the embodiment will be described. FIG. 4 is a flowchart for explaining an example of the operation of the remote monitoring control device 13 in the remote monitoring system 1 according to the embodiment. The in-vehicle system 2 encodes video data captured by omnidirectional cameras 11A, 11B installed on the train using a video processing device, and transmits the encoded video data to a remote monitoring and control device 13 on the ground side via a wireless communication device 21.
[0046] Here, the in-vehicle system 2 encodes both the video data captured by the first omnidirectional camera 11A and the video data captured by the second omnidirectional camera 11B and transmits the encoded data to the remote monitoring control device 13 at a predetermined cycle. However, the in-vehicle system 2 may select either the video data captured by the first omnidirectional camera 11A or the video data captured by the second omnidirectional camera 11B according to the traveling direction of the train, encode the selected video data, and transmit the encoded data to the remote monitoring control device 13 at a predetermined cycle.
[0047] The remote monitoring control device 13 receives the video data captured by the omnidirectional cameras 11A and 11B, which has been encoded by the wireless communication unit 32 (step ST11). The control unit 31 of the remote monitoring control device 13 decodes the encoded video data received by the wireless communication unit 32 using the video decoding unit 35 (step ST12).
[0048] The control unit 31 identifies the traveling direction of the train based on the vehicle information from the in-car system 2, and selects either the video data captured by the first omnidirectional camera 11A or the video data captured by the second omnidirectional camera 11B as the display image to be displayed on the head-mounted display 14 according to the traveling direction (step ST13). For example, when the train is moving forward, the control unit 31 selects the video data captured by the first omnidirectional camera 11A as the display image, and when the train is moving backward, the control unit 31 selects the video data captured by the second omnidirectional camera 11B as the display image.
[0049] Furthermore, the remote monitoring and control device 13 acquires gaze detection information for identifying the gaze direction of the operator wearing the head mounted display 14 from the head mounted display 14 via the wireless communication unit 33. Upon acquiring the gaze detection information from the head mounted display 14, the control unit 31 determines (estimates) the gaze direction of the operator based on the gaze detection information acquired by the gaze determination unit 36 (step ST14). Here, the control unit 31 may acquire information indicating the gaze direction of the operator who is the wearer from the head mounted display 14.
[0050] When the control unit 31 identifies the operator's line of sight direction using the line of sight determination unit 36, the display image generation unit 38 generates a field of view image that is assumed to be reflected in the operator's field of view based on the operator's line of sight direction (step S15). For example, the display image generation unit 38 generates the field of view image by cutting out an image that falls within a predetermined field of view angle based on the operator's line of sight direction from the image of the selected omnidirectional camera 11A or 11B. In this way, the display image generation unit 38 can generate a virtual image that falls within the field of view of the operator when it is assumed that the operator is in the driver's seat or at the installation position of the omnidirectional camera as the field of view image.
[0051] The control unit 31 generates a field of view image using the display image generation unit 38, and receives vehicle information of the train from the in-car system 2 via the wireless communication unit 32. The control unit 31 superimposes the vehicle information of the train on the field of view image generated from the image data by the display image generation unit 38 (step S15). As a result, the control unit 31 displays the vehicle information together with the field of view image on the head mounted display 14. For example, the control unit 31 extracts specific information to be displayed from the vehicle information acquired from the in-car system 2, and displays the extracted information on the head mounted display 14 by superimposing it on the field of view image.
[0052] Furthermore, the control unit 31 executes an abnormality detection process by the video analysis unit 37. For example, the control unit 31 detects the presence or absence of an abnormal state around the train from the video data. The abnormality detection process may detect an obstacle in the track as an abnormal state, may detect the intrusion or approach of a person or object into the track, or may detect an abnormality in the environment around the track, such as weather. Note that the abnormality detection process may detect an abnormality not only from the video data displayed as the field of view image, but also from the video data that is not displayed on the head mounted display 14. For example, the control unit 31 may detect an abnormality on the track or around the track immediately after the train passes (on the opposite side to the direction of travel) while displaying an image in the direction of travel on the head mounted display 14 as the field of view image.
[0053] If no abnormality is detected by the abnormality detection process (step ST17, NO), the control unit 31 transmits to the head-mounted display 14 a display image in which vehicle information is superimposed on a field of view image generated from the video data captured by the omnidirectional camera 11A or 11B via the wireless communication unit 33 (step S19).
[0054] When an abnormality is detected by the abnormality detection process by the video analysis unit 37 (step ST17, YES), the control unit 31 superimposes a notification indicating an abnormal state (abnormality notification) on the field of view image (step ST18). The abnormality notification may be information that notifies the user of the detected abnormal state, or information that notifies the user of a warning about the abnormal state. When an abnormality is detected, the control unit 31 transmits to the head mounted display 14, via the wireless communication unit 33, a display image in which the vehicle information and an abnormality notification indicating the abnormality detected by the abnormality detection process are superimposed on the field of view image (step ST19). As a result, the control unit 31 causes the head mounted display 14 to display the vehicle information and abnormality detection together with the field of view image.
[0055] In the above-described operation example, the remote monitoring control device 13 supplies the display image in which the vehicle information and the abnormality notification are superimposed on the field of view image to the head mounted display 14, but the remote monitoring control device 13 may transmit the field of view image generated from the image of the omnidirectional camera, the vehicle information, and the abnormality notification to the head mounted display 14. In this case, the control unit 40 of the head mounted display 14 may superimpose the vehicle information and the abnormality notification on the field of view image and display them on the display unit 42.
[0056] As described above, the remote monitoring system according to the embodiment transmits image data captured by the omnidirectional cameras installed at the front and rear of the train to the remote monitoring control device via wireless communication. The remote monitoring control device determines the line of sight of the operator wearing the head-mounted display according to the output of a sensor that detects the movement of the operator, converts the images captured by the omnidirectional cameras into a virtual reality field of view image based on the line of sight of the operator, and displays it on the head-mounted display.
[0057] This allows the operator to view images on the head-mounted display in a field of view that corresponds to the movement of the line of sight, rather than being limited to images in a specific direction, and allows the operator to check images of the area around the train that are displayed in response to not only the direction of travel but also their own movements.
[0058] In addition, the remote monitoring and control device according to the embodiment selects an omnidirectional camera according to the direction of travel of the train and generates a field of view image from the image captured by the selected omnidirectional camera. This allows the operator to easily check not only the image around the front of the train, but also the image around the rear of the train when the train is reversing.
[0059] In addition, the remote monitoring and control device displays various vehicle information (speed, notch, malfunction information, etc.) acquired from the in-car system on the head-mounted display together with the field of view image. This allows the operator to easily check not only the field of view image but also various vehicle information of the train in question while wearing the head-mounted display according to the line of sight direction.
[0060] In addition, when the remote monitoring and control device detects an abnormal condition (obstacle, etc.) from the video data of the omnidirectional camera, it displays an abnormality notification to call attention on the head mounted display. This allows the operator to check not only the field of view image according to the line of sight direction, but also warnings about abnormal conditions detected on the train or around the train while wearing the head mounted display.
[0061] Furthermore, the remote monitoring system has a pseudo cab where an operator wearing a head mounted display 14 drives the vehicle. The pseudo cab transmits main control information indicating control commands such as a powering notch command and a braking notch command based on the driving operation by the operator to the remote monitoring control device. The remote monitoring control device transmits the main control information from the pseudo cab to a main controller 24 in the in-vehicle system 2 by wireless communication. This allows the operator to drive the vehicle at the pseudo cab installed in a remote location while viewing an image of the actual vehicle's surroundings in his or her own field of vision on the head mounted display.
[0062] Moreover, the remote monitoring system 1 according to the above-described embodiment can also be modified and implemented as follows. The omnidirectional camera 11A may be installed in a position where it can capture various indicator lights, instruments, and switches installed in the cab of the actual vehicle. The in-vehicle system 2 may also be configured to include a third omnidirectional camera installed to capture devices (indicator lights, instruments, and switches) installed in the cab of the actual vehicle. When an omnidirectional camera that captures the cab of the actual vehicle is provided, the control unit 31 of the remote monitoring control device 13 may extract various displays and setting states in the cab of the actual vehicle from the image captured by the omnidirectional camera and display them on the head-mounted display 14. This allows an operator wearing the head-mounted display 14 to visually confirm the information displayed in the cab of the actual vehicle remotely.
[0063] The control unit 31 of the remote monitoring control device 13 may detect a possible malfunction in the in-vehicle system 2 or the like as an abnormal state by analyzing the image captured by an omnidirectional camera capable of capturing the devices (indicator lights, gauges, switches, etc.) installed in the cab of the actual vehicle by the image analysis unit 37. For example, the image analysis unit 37 detects information displayed by the devices installed in the cab of the actual vehicle (information indicated by the devices installed in the vehicle) from the image captured by the omnidirectional camera. When the information indicated by the devices installed in the vehicle detected from the image data differs from the vehicle information acquired from the in-vehicle system 2, the control unit 31 may determine that a malfunction (abnormal state) may occur in the in-vehicle system 2 or the like, and may display a warning or an alert on the head-mounted display 14. This enables the remote monitoring control device 13 to notify the operator that a malfunction may occur in the in-vehicle system 2 or the like.
[0064] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]
[0065] 1...remote monitoring system, 2...in-vehicle system, 11A...omnidirectional camera (first omnidirectional camera), 11B...omnidirectional camera (second omnidirectional camera), 13...remote monitoring control device, 14...head-mounted display (HMD), 15...pseudo driver's cab, 21...wireless communication device, 22...image processing device, 23...vehicle information device, 24...main controller, 31...control unit, 32...wireless communication unit, 33...wireless communication unit, 34...communication interface, 35...image decoding unit, 38...display image generation unit, 37...image analysis unit, 36...gaze determination unit, 40...control unit, 41...wireless communication unit, 42...display unit, 43...gaze detection unit, 51...control unit, 52...display unit, 53...operation unit, 54...communication interface.
Claims
1. a first wireless communication unit that receives video data captured by an omnidirectional camera installed on the train; A second wireless communication unit that communicates with the head mounted display; a control unit that determines a line of sight direction of an operator wearing the head mounted display, and displays on the head mounted display a field of view image generated based on the line of sight direction of the operator from video data captured by the omnidirectional camera and received via the first wireless communication unit; A remote monitoring and control device comprising:
2. The second wireless communication unit receives an output from a sensor that detects a movement of the operator from the head mounted display, the control unit determines a line of sight direction of the operator based on an output from the sensor received from the head mounted display via the second wireless communication unit. The remote monitoring and control device according to claim 1 .
3. The first wireless communication unit receives first video data captured by a first omnidirectional camera installed at the front of the train and second video data captured by a second omnidirectional camera installed at the rear of the train, The control unit generates the field of view image from the first image data or the second image data selected according to the traveling direction of the train. The remote monitoring and control device according to claim 1 .
4. The first wireless communication unit receives vehicle information from an in-car system installed in the train, The control unit causes the head mounted display to display the vehicle information together with the field of view image. The remote monitoring and control device according to claim 1 .
5. When an abnormal condition around the train is detected from the video data captured by the omnidirectional camera and received by the first wireless communication unit, the control unit displays a notification indicating the abnormal condition on the head mounted display. The remote monitoring and control device according to any one of claims 1 to 4.
6. A remote monitoring system having an in-vehicle system and a remote viewing control device, The in-vehicle system includes: An omnidirectional camera installed on the train, A wireless communication device that transmits the image data captured by the omnidirectional camera to a remote monitoring control device installed on the ground, The remote monitoring and control device includes: a first wireless communication unit that receives video data captured by the omnidirectional camera from the in-vehicle system; A second wireless communication unit that communicates with the head mounted display; a control unit that determines a line of sight direction of an operator wearing the head mounted display, and displays a field of view image on the head mounted display, the field of view image being generated based on the line of sight direction of the operator and based on the video data captured by the omnidirectional camera and received via the first wireless communication unit; A remote monitoring system comprising:
7. The second wireless communication unit receives an output from a sensor that detects a movement of the operator from the head mounted display, the control unit determines a line of sight direction of the operator based on an output from the sensor received from the head mounted display via the second wireless communication unit. The remote monitoring system according to claim 6.
8. The first wireless communication unit receives first video data captured by a first omnidirectional camera installed at the front of the train and second video data captured by a second omnidirectional camera installed at the rear of the train, The control unit generates the field of view image from the first image data or the second image data selected according to the traveling direction of the train. The remote monitoring system according to claim 6.
9. The first wireless communication unit receives vehicle information from an in-car system installed in the train, The control unit causes the head mounted display to display the vehicle information together with the field of view image. The remote monitoring system according to claim 6.
10. When an abnormal condition around the train is detected from the video data captured by the omnidirectional camera and received by the first wireless communication unit, the control unit displays a notification indicating the abnormal condition on the head mounted display. A remote monitoring system according to any one of claims 6 to 9.
11. Further, a remote monitoring system having a pseudo cab, The pseudo cab includes: a first communication interface for communicating with the remote monitoring and control device; An operation unit to which the driving operation by the operator is input, The remote monitoring and control device includes: a second communication interface for communicating with the pseudo cab; The control unit transmits a control command for the vehicles constituting the train in response to the driving operation inputted by the operation unit of the pseudo cab to the in-car system including a main controller of the train via the first wireless communication unit. The remote monitoring system according to claim 6.
Citation Information
Patent Citations
Track inside abnormal condition emergency communication system for railway, station service remote monitoring system, monitoring system having image delivery function
JP2002234440A
