Remote monitoring method, remote monitoring terminal, and remote monitoring program

Abstract

A remote monitoring method according to the present disclosure comprises: a first step for acquiring, from a towing vehicle which includes at least one sensor including a camera, sensor data which is output from the at least one sensor and which includes a first image that shows a surrounding environment of the towing vehicle and that is captured by the camera; a second step for identifying, on the basis of the sensor data, a state of a vehicle being towed by the towing vehicle; a third step for specifying, on the basis of the identified state, a first area that requires focus; and a fourth step for outputting a second image in which a first portion corresponding to the first area in the first image is highlighted.

Description

Remote monitoring method, remote monitoring terminal, and remote monitoring program

[0001] The present disclosure relates to a remote monitoring method, a remote monitoring terminal, and a remote monitoring program.

[0002] 2. Description of the Related Art As a technique for operating a vehicle consisting of a towing vehicle and a towed vehicle coupled to the towing vehicle, there is a technique for generating an overhead image of the vehicle (see, for example, Patent Document 1).

[0003] Special Publication No. 2020-534209

[0004] There are cases where a vehicle consisting of the above-described towing vehicle and a towed vehicle coupled to the towing vehicle is operated in a remote location.

[0005] The present disclosure aims to provide a remote monitoring method, a remote monitoring terminal, and a remote monitoring program that can reduce the burden placed on an operator in monitoring the conditions around a vehicle.

[0006] The remote monitoring method disclosed herein includes a first step of acquiring sensor data output by one or more sensors, including a camera, from a towing vehicle equipped with one or more sensors, the sensor data including a first image capturing the surrounding environment of the towing vehicle captured by the camera; a second step of identifying the condition of a towed vehicle being towed by the towing vehicle based on the sensor data; a third step of identifying a first area requiring attention based on the identified condition; and a fourth step of outputting a second image in which a first portion of the first image corresponding to the first area is highlighted.

[0007] According to the present disclosure, it is possible to provide a remote monitoring method, a remote monitoring terminal, and a remote monitoring program that can reduce the burden placed on an operator to monitor the conditions around a vehicle.

[0008] FIG. 1 is a diagram showing the configuration of a remote monitoring system according to the first embodiment. FIG. 2 is a diagram showing the configuration of a vehicle monitored by the remote monitoring system according to the first embodiment. FIG. 3 is a diagram showing the configuration of a vehicle control device according to the first embodiment. FIG. 4 is a diagram for explaining a gaze area according to the first embodiment. FIG. 5 is a diagram showing a method for highlighting a portion showing the gaze area by the remote monitoring server according to the first embodiment. FIG. 6 is a diagram showing the configuration of a remote monitoring server according to the first embodiment. FIG. 7 is a diagram showing the configuration of a remote monitoring terminal according to the first embodiment. FIG. 8 is a flowchart showing a remote monitoring operation according to the first embodiment. FIG. 9 is a flowchart showing an operation for identifying the status of a queue of towed vehicles according to the first embodiment. FIG. 10 is a diagram for explaining an operation for detecting the side of a towed vehicle according to the first embodiment. FIG. 11 is a diagram for explaining an operation for identifying the status of a queue of towed vehicles according to the second embodiment. FIG. 12 is a flowchart showing an operation for identifying the status of a queue of towed vehicles according to the third embodiment. FIG. 13 is a diagram for explaining an emphasis method according to Modification 1. FIG. 14 is a diagram for explaining an emphasis method according to Modification 2. Fig. 15 is a diagram for explaining an emphasis method according to Modification 3. Fig. 16 is a diagram for explaining an emphasis method according to Modification 4. Fig. 17 is a diagram for explaining an emphasis method according to Modification 5. Fig. 18 is a diagram for explaining a gaze area according to Modification 7. Fig. 19 is a diagram for explaining a gaze area according to Modification 8. Fig. 20 is a diagram showing an example of a screen according to the sixth embodiment, showing an area not shown in an image captured by a camera. Fig. 21 is a diagram showing an example when an obstacle is in the blind spot of the camera.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings as appropriate. However, more detailed description than necessary may be omitted. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0010] First Embodiment FIG. 1 is a diagram showing the configuration of a remote monitoring system 1000 according to a first embodiment.

[0011] The remote monitoring system 1000 is a system that supports an operator OP who is not a passenger in the vehicle to monitor the surrounding conditions of one or more vehicles VH operated in a remote location. Hereinafter, monitoring the surrounding conditions of the vehicle VH will be referred to as "monitoring the vehicle VH."

[0012] The vehicle VH is operated in a remote location. For example, the vehicle VH is operated in an airport, a warehouse, or the like, and an operator OP can monitor the vehicle VH from a location far away from the vehicle VH using the remote monitoring system 1000. Note that the location where the vehicle VH is operated is not limited to these.

[0013] The vehicle VH is composed of a towing vehicle TT and one or more towed vehicles TL connected to the towing vehicle TT. The towing vehicle TT has a power source, while each towed vehicle TL does not have a power source. The towing vehicle TT tows one or more towing vehicles TL using power from its own power source. The power source may be an internal combustion engine or an electric motor.

[0014] The towing vehicle TT may be driven by a driver, or may be driven by remote operation by an operator OP, or may be configured to be capable of automatic driving.

[0015] 1 shows vehicle VH1 equipped with four towed vehicles TL, vehicle VH2 equipped with four towed vehicles TL, and vehicle VH3 equipped with one towed vehicle TL as examples of vehicles VH monitored by remote monitoring system 1000. An operator OP can monitor these vehicles VH1 to VH3 by himself using a remote monitoring terminal 2, which will be described later. Note that any number of vehicles VH can be monitored by one remote monitoring terminal 2.

[0016] The remote monitoring system 1000 includes a remote monitoring server 1 and a remote monitoring terminal 2. Each vehicle VH and the remote monitoring server 1 are connected via a network 3 to enable mutual data communication. The remote monitoring server 1 and the remote monitoring terminal 2 are connected via the network 3 to enable mutual data communication.

[0017] The remote monitoring server 1 acquires images of the surrounding environment of each vehicle VH from each vehicle VH. The remote monitoring server 1 then performs arithmetic processing on the acquired images and outputs the processed images to the remote monitoring terminal 2. The remote monitoring terminal 2 displays the images received from the remote monitoring server 1 on its own display device (a display device 26, described below). The images received from the remote monitoring server 1 and displayed on the display device 26 of the remote monitoring terminal 2 are referred to as monitoring screens.

[0018] The remote monitoring terminal 2 is an information processing terminal used by an operator OP. The operator OP can monitor each vehicle VH by visually checking an image displayed on a display device 26 provided in the remote monitoring terminal 2.

[0019] The number of remote monitoring terminals 2 included in the remote monitoring system 1000 is not limited to one. The remote monitoring system 1000 may include multiple remote monitoring terminals 2, and monitoring of multiple vehicles VH may be shared among the multiple remote monitoring terminals 2. One remote monitoring terminal 2 may display a monitoring screen related to one vehicle VH, or may display multiple monitoring screens related to different vehicles VH. When one remote monitoring terminal 2 displays multiple monitoring screens, the remote monitoring server 1 may display the multiple monitoring screens side by side on the display device 26, or may switch between them depending on the time.

[0020] 2 is a diagram showing the configuration of a vehicle VH monitored by the remote monitoring system 1000 according to the first embodiment. In describing the orientation and positional relationship, the direction facing forward when the vehicle VH is moving forward is referred to as "forward," and the direction opposite to the forward direction is referred to as "rear." The direction turned 90 degrees to the left from the forward direction is referred to as "left," and the direction turned 90 degrees to the right from the forward direction is referred to as "right."

[0021] The towing vehicle TT, which is disposed at the front of the vehicle VH, is equipped with a plurality of cameras 50. In the example shown in Fig. 2, the towing vehicle TT is equipped with a front camera 50a, a right camera 50b, a left camera 50c, and a rear camera 50d as examples of the plurality of cameras 50.

[0022] Each camera 50 is an imaging device that incorporates an image sensor, such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor). Each camera 50 can output images at a predetermined frame rate. Each camera 50 has a wide-angle lens or a fisheye lens. Therefore, each camera 50 can capture an image of a wide area toward which its optical axis is directed. Furthermore, each camera 50 is directed toward the outside of the towing vehicle TT, allowing each camera 50 to capture an image of the surrounding environment of the towing vehicle TT.

[0023] The front camera 50a is mounted on the front of the towing vehicle TT (for example, on the front grille). The optical axis of the front camera 50a is directed forward. Therefore, the front camera 50a can capture images of a wide area in front of the vehicle VH.

[0024] The right camera 50b is mounted on the right side of the towing vehicle TT (for example, on the right door mirror). The optical axis of the right camera 50b is directed to the right. Therefore, the right camera 50b can capture images of a wide area to the right of the towing vehicle TT.

[0025] The left camera 50c is mounted on the left side of the towing vehicle TT (for example, on the left door mirror). The optical axis of the left camera 50c is directed leftward. Therefore, the left camera 50c can capture images of a wide area to the left of the towing vehicle TT.

[0026] The rear camera 50d is provided at the rear (for example, at the rear ceiling) of the towing vehicle TT. The rear camera 50d faces rearward. Therefore, the rear camera 50d can capture images of a wide area behind the towing vehicle TT.

[0027] The locations at which the cameras 50 are installed are not limited to the above example. The number of cameras 50 installed on the towing vehicle TT and the locations of the cameras 50 are arbitrary as long as they are capable of capturing images of the environment surrounding the towing vehicle TT.

[0028] The towing vehicle TT is provided with a vehicle control device 40. The vehicle control device 40 controls the towing vehicle TT. As part of the control of the towing vehicle TT, the vehicle control device 40 transmits images of the surrounding environment captured by the four cameras 50 to the remote monitoring server 1.

[0029] FIG. 3 is a diagram showing the configuration of the vehicle control device 40 according to the first embodiment.

[0030] The vehicles control device 40 is connected to the cameras 50a to 50d, a distance measurement sensor 51, a position sensor 52, a steering angle sensor 53, and a wheel speed sensor 54. The vehicles control device 40, the cameras 50a to 50d, the distance measurement sensor 51, the position sensor 52, the steering angle sensor 53, and the wheel speed sensor 54 are provided on the towing vehicle TT. The vehicles control device 40, the cameras 50a to 50d, the distance measurement sensor 51, the position sensor 52, the steering angle sensor 53, and the wheel speed sensor 54 are driven by power supplied from the power source of the towing vehicle TT.

[0031] Some or all of the cameras 50a to 50d, the distance measurement sensor 51, the position sensor 52, the steering angle sensor 53, and the wheel speed sensor 54 may be connected to the vehicle control device 40 via a network such as a CAN (Controller Area Network).

[0032] The distance measurement sensor 51 is a sensor that detects an obstacle and measures the distance to the obstacle when an obstacle exists around the towing vehicle TT. The distance measurement sensor 51 is, for example, a sonar or a stereo camera.

[0033] The position sensor 52 is a sensor that detects position information of the towing vehicle TT. As the position sensor 52, for example, a sensor that performs positioning using signals from positioning satellites of the Global Navigation Satellite System (GNSS) can be applied.

[0034] The steering angle sensor 53 is a sensor that detects the steering angle.

[0035] The wheel speed sensor 54 is a sensor that detects the amount of rotation or the number of rotations per unit time of the wheels of the towing vehicle TT as wheel speed information.

[0036] The cameras 50a to 50d, the distance measurement sensor 51, the position sensor 52, the steering angle sensor 53, and the wheel speed sensor 54 are examples of one or more sensors.

[0037] The vehicle control device 40 includes a CPU (Central Processing Unit) 41, a RAM (Random Access Memory) 42, a ROM (Read Only Memory) 43, a network interface 44, an I / O (Input / Output) interface 45, and a bus 46. The CPU 41, the RAM 42, the ROM 43, the network interface 44, and the I / O interface 45 are electrically connected to the bus 46.

[0038] The I / O interface 45 is an interface for the vehicle control device 40 to communicate data with external devices. In this example, the external devices are the cameras 50a to 50d, the distance measurement sensor 51, the position sensor 52, the steering angle sensor 53, and the wheel speed sensor 54.

[0039] The network interface 44 is an adapter that enables the vehicle control device 40 to communicate with the remote monitoring server 1 via the network 3. When the vehicle control device 40 communicates wirelessly with the remote monitoring server 1, the network interface 44 is a wireless communication adapter.

[0040] The ROM 43 stores computer programs and parameters necessary for executing the computer programs. The RAM 42 temporarily stores various data used in the calculations performed by the CPU 41. The CPU 41 is a processor capable of executing computer programs.

[0041] In the vehicle control device 40, a program for controlling the towing vehicle TT is stored in the ROM 43. The CPU 41 loads the program from the ROM 43 into the RAM 42 at a predetermined timing, such as at startup. The CPU 41 then controls the towing vehicle TT based on the program in the RAM 42.

[0042] The towing vehicle TT is not limited to being equipped with one vehicle control device 40. The towing vehicle TT may be equipped with multiple vehicle control devices 40, each with different functions. For example, the function of acquiring various sensor data such as images captured by each camera 50, the function of transmitting the various sensor data to the remote monitoring server 1, and the function of controlling the traveling of the towing vehicle TT may each be equipped in a different vehicle control device 40.

[0043] The remote monitoring server 1 generates a monitoring screen that displays images captured by the cameras 50a to 50d, and outputs the monitoring screen to the remote monitoring terminal 2. At this time, the remote monitoring server 1 identifies an area that requires attention, and on the monitoring screen, emphasizes the portion of the image displayed on the display device 26 that shows the area that requires attention. Hereinafter, the area that requires attention will be referred to as the "attention area."

[0044] 4 is a diagram for explaining the gaze area in the first embodiment, in which the gaze area will be explained using the vehicle VH1 as an example.

[0045] As the towing vehicle TT continues to move forward while steering left, the four towed vehicles TL follow the towing vehicle TT, causing the line of towed vehicles TL to curve sharply to the left. This state of the line of towed vehicles TL is referred to as a left turn. If an obstacle is found in area A1, which is the area on the turning direction side of the two areas on either side of the towed vehicle TL when the line of towed vehicles TL is turning left, there is a risk that the vehicle VH1 will be caught up in the obstacle. Therefore, an area where an obstacle may be caught up, such as area A1 in Figure 4, is identified as the focus area.

[0046] The state in which the line of towed vehicles TL is curved sharply to the right is referred to as a right turn state. The state in which the line of towed vehicles TL is in a straight line is referred to as a straight-ahead state. The state in which the line of towed vehicles TL is in an S-shape is referred to as an S-turn state.

[0047] FIG. 5 is a diagram showing a method for emphasizing a portion showing a gaze area by the remote monitoring server 1 according to the first embodiment.

[0048] Normally, that is, when there is no gaze area, the remote monitoring server 1 outputs the monitoring screen D1.

[0049] The monitoring screen D1 includes a front image field F1, a right image field F2, a left image field F3, and a rear image field F4. The front image field F1 is a field in which an image captured by the front camera 50a is displayed. The right image field F2 is a field in which an image captured by the right camera 50b is displayed. The left image field F3 is a field in which an image captured by the left camera 50c is displayed. The rear image field F4 is a field in which an image captured by the rear camera 50d is displayed. The remote monitoring server 1 may display the images captured by each camera 50 directly in the corresponding field. The remote monitoring server 1 may also display the images captured by each camera 50 in the corresponding field after performing predetermined image processing, such as cropping.

[0050] The monitoring screen D1a is an example of a screen that is output when a focus area is specified. Here, an area A1 where an obstacle may be caught is located to the left of the vehicle VH1 (see FIG. 4), and the area A1 is specified as the focus area.

[0051] Area A1 appears in the image captured by left camera 50c. Therefore, the remote monitoring server 1 highlights left image field F3, which displays the image captured by left camera 50c. In the example shown in Fig. 5, the left image field F3 is surrounded by a thick frame, thereby emphasizing the image displayed in left image field F3.

[0052] The group of images captured by cameras 50a-50d and sent from towing vehicle TT to remote monitoring server 1 is an example of a first image. The portion of the group of images captured by cameras 50a-50d that shows the gaze area (in the case of FIG. 5, the image captured by left camera 50c) is an example of a first portion. The monitoring screen D1a in which the image captured by left camera 50c, which is the portion that shows the gaze area, is highlighted is an example of a second image.

[0053] Depending on the setting of the angle of view of the rear camera 50d, part or all of the area A1 may appear, for example, in the left portion of the image captured by the rear camera 50d. In such a case, the remote monitoring server 1 may emphasize not only the image displayed in the left image field F3 but also the image displayed in the rear image field F4. Alternatively, if part or all of the area A1 appears in the left portion of the image captured by the rear camera 50d, the remote monitoring server 1 may emphasize not only the image displayed in the left image field F3 but also the left portion of the image displayed in the rear image field F4. In other words, if the gaze area spans multiple images, the remote monitoring server 1 may highlight each of the multiple images or highlight the portion of each of the multiple images in which the gaze area appears.

[0054] FIG. 6 is a diagram showing the configuration of the remote monitoring server 1 according to the first embodiment.

[0055] The remote monitoring server 1 includes a CPU 11, a RAM 12, a ROM 13, a network interface 14, a storage device 15, and a bus 16. The CPU 11, the RAM 12, the ROM 13, the network interface 14, and the storage device 15 are electrically connected to the bus 16.

[0056] The network interface 14 is an adapter that enables the remote monitoring server 1 to communicate with each vehicle VH and the remote monitoring terminal 2 via the network 3 .

[0057] The ROM 13 stores computer programs and parameters necessary for executing the computer programs. The CPU 11 is a processor capable of executing computer programs. The RAM 12 temporarily stores various data used in the calculations of the CPU 11. The storage device 15 stores data that should be saved in a non-volatile manner. For example, the storage device 15 stores operation history.

[0058] In this embodiment, a remote monitoring program 100 is stored in the ROM 13. The CPU 11 loads the remote monitoring program 100 from the ROM 13 into the RAM 12 at a predetermined timing, such as at startup. The CPU 11 then implements the remote monitoring operation (i.e., the remote monitoring method) according to the first embodiment based on the remote monitoring program 100 stored in the RAM 12. The operation of the CPU 11 described below is executed by the CPU 11 based on the remote monitoring program 100.

[0059] The remote monitoring program 100 may be stored in the storage device 15 instead of the ROM 13 .

[0060] FIG. 7 is a diagram showing the configuration of the remote monitoring terminal 2 according to the first embodiment.

[0061] The remote monitoring terminal 2 includes a CPU 21, a RAM 22, a ROM 23, a network interface 24, an input device 25, a display device 26, and a bus 27. The CPU 21, the RAM 22, the ROM 23, the network interface 24, the input device 25, and the display device 26 are electrically connected to the bus 27.

[0062] The network interface 24 is an adapter for the remote monitoring terminal 2 to communicate with the remote monitoring server 1 via the network 3 .

[0063] The input device 25 is an HMI (Human Machine Interface) for inputting information. The input device 25 is, for example, a touch panel, a keyboard, or a pointing device. The operator OP can input instructions to the remote monitoring terminal 2 via the input device 25.

[0064] The display device 26 is an HMI that outputs information in a visible manner. The display device 26 is, for example, an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display. The display device 26 displays a monitoring screen (e.g., see monitoring screens D1 and D1a in FIG. 5 ) received from the remote monitoring server 1.

[0065] Next, the operation of the remote monitoring according to the first embodiment will be described.

[0066] 8 is a flowchart showing the operation of remote monitoring according to the first embodiment. Note that the series of operations shown in this figure is repeatedly executed at predetermined short time intervals (for example, the frame rate of the camera 50).

[0067] The CPU 11 acquires images captured by the cameras 50a to 50d (S101), and then identifies the state of the queue of towed vehicles TL (S102).

[0068] FIG. 9 is a flowchart showing the operation of identifying the state of the queue of towed vehicles TL according to the first embodiment.

[0069] The CPU 11 performs side detection of the towed vehicle TL (S201). The CPU 11 detects the side of the towed vehicle TL from the image captured by the right camera 50b and the image captured by the left camera 50c among the images captured by the cameras 50a to 50d.

[0070] For example, consider a case where a line of towed vehicles TL is turning left, as shown in Figure 10. In such a case, the side of the towed vehicle TL may be captured in range R1 of the imaging range of the left camera 50c. In the example shown in Figure 10, the side P1 of the third towed vehicle TL from the lead vehicle (towing vehicle TT), the side P2 of the fourth towed vehicle TL from the lead vehicle (towing vehicle TT), and the side P3 of the fifth towed vehicle TL from the lead vehicle are captured in the image captured by the left camera 50c.

[0071] Conversely, when the line of towed vehicles TL is turning right, the side of the towed vehicles TL is captured in the image captured by the right camera 50b.

[0072] When the line of towed vehicles TL is moving straight, the side of the towed vehicles TL is not visible in either the image captured by the right camera 50b or the image captured by the left camera 50c.

[0073] When a line of towed vehicles TL is making an S-shaped turn, the sides of the towed vehicles TL may appear in the images captured by the right camera 50b or the left camera 50c, but the number of towed vehicles TL whose sides appear in these images is smaller than when the line of towed vehicles TL is making a right turn or a left turn.

[0074] Therefore, the CPU 11 detects the side of the towed vehicle TL for each image and identifies the state of the line of towed vehicles TL based on the detection results. Note that the algorithm for detecting the side of the towed vehicle TL is not limited to a specific algorithm. The CPU 11 may detect the side of the towed vehicle TL by, for example, image recognition.

[0075] First, the CPU 11 determines whether the side of the towed vehicle TL is detected in any of the images (S202). If the side of the towed vehicle TL is not detected in any of the images (S202: No), the CPU 11 determines that the line of towed vehicles TL is moving straight ahead (S203), and the operation of identifying the state of the line of towed vehicles TL ends.

[0076] If the side of the towed vehicle TL is detected from any of the images (S202: Yes), it is determined whether the image in which the side of the towed vehicle TL is detected shows the sides of a set number of or more towed vehicles TL (S204).

[0077] As described above, in an S-turn state, the number of towed vehicles TL whose sides are visible in the image is smaller than in a left or right turn state. Therefore, a threshold value for distinguishing between an S-turn state and a left or right turn state is set in advance as a set number of vehicles.

[0078] If the side faces of the towed vehicles TL are not visible in the image in which the sides of the towed vehicles TL are detected, which is equal to or greater than the set number (S204: No), the CPU 11 determines that the line of towed vehicles TL is in an S-shaped turn (S205), and the operation of identifying the state of the line of towed vehicles TL ends.

[0079] If the image in which the side of the towed vehicle TL is detected shows the side of more than the set number of towed vehicles TL (S204: Yes), the CPU 11 determines whether the image in which the side of the towed vehicle TL is detected is an image captured by the right camera 50b (S206).

[0080] If the image in which the side of the towed vehicle TL is detected is not an image captured by the right camera 50b (S206: No), the CPU 11 determines that the line of towed vehicles TL is turning left (S207), and the operation of identifying the state of the line of towed vehicles TL ends.

[0081] If the image in which the side of the towed vehicle TL is detected is an image captured by the right camera 50b (S206: Yes), the CPU 11 determines that the line of towed vehicles TL is turning right (S208), and the operation of identifying the state of the line of towed vehicles TL ends.

[0082] Returning to the explanation of Fig. 8, after identifying the state of the queue of towed vehicles TL, the CPU 11 identifies a gaze area (S103).

[0083] For example, if the line of towed vehicles TL is turning left or right, the CPU 11 identifies the area on the side of the line of towed vehicles TL that is closer to the turning direction as the gaze area. In the example shown in Figure 4, the line of towed vehicles TL is turning left, so area A1 on the left side of the line of towed vehicles TL is identified as the gaze area.

[0084] If the line of towed vehicles TL is moving straight or making an S-turn, the CPU 11 determines that the gaze area has not been specified and performs the subsequent processing.

[0085] Next, the CPU 11 determines whether or not a gaze area has been identified (S104). If a gaze area has not been identified (S104: No), the CPU 11 outputs a normal monitoring screen (S105). The normal monitoring screen is, for example, the monitoring screen D1 in FIG. 5.

[0086] If the gaze area is identified (S104: Yes), the CPU 11 outputs a monitoring screen in which the portion showing the gaze area is highlighted (S106). The monitoring screen in which the portion showing the gaze area is highlighted is, for example, the monitoring screen D1a in FIG. 5.

[0087] After the process of S105 or S106, one remote monitoring operation is completed. As described above, the series of operations shown in Fig. 8 is repeatedly executed at short time intervals. Therefore, the operator OP can recognize the multiple monitoring screens displayed at short time intervals on the display device 26 as a moving image.

[0088] 8, the remote monitoring server 1 repeatedly executes the series of operations shown in Fig. 8 to highlight the portion of the gaze area that is displayed from the time the line of towed vehicles TL starts turning left or right until the line of towed vehicles TL starts going straight or making an S-turn. The fact that the line of towed vehicles TL starts going straight or making an S-turn after starting to turn left or right may be detected by a method different from the method shown in Fig. 9.

[0089] For example, the CPU 11 of the remote monitoring server 1 sequentially acquires the steering angle of the towing vehicle TT output from the steering angle sensor 53. The CPU 11 of the remote monitoring server 1 determines, based on the steering angle of the towing vehicle TT, whether the towing vehicle TT has traveled straight for a predetermined distance or a predetermined period of time after the line of towed vehicles TL has turned left or right. The CPU 11 of the remote monitoring server 1 determines the predetermined distance and the predetermined period of time based on the number of towed vehicles TL being towed by the towing vehicle TT, the length of the line of towed vehicles TL, or the steering angle of the towing vehicle TT when the line of towed vehicles TL has turned left or right. If it is determined that the towing vehicle TT has traveled straight for the predetermined distance or the predetermined period of time, the CPU 11 of the remote monitoring server 1 stops highlighting the portion of the gaze area.

[0090] If the CPU 11 of the remote monitoring server 1 is configured to detect whether the line of towed vehicles TL is moving straight or making an S-turn based on whether the towing vehicle TT has been moving straight for a predetermined distance or period of time, the CPU 11 of the remote monitoring server 1 may display on the monitoring screen, while highlighting the portion showing the gaze area, the remaining time until the gaze area is exhausted or the distance to be traveled until the gaze area is exhausted. This allows the operator OP to know how much longer he or she needs to keep an eye on the line.

[0091] Furthermore, in the example shown in FIG. 8 , when the line of towed vehicles TL is making an S-shaped turn, it is determined that a gaze area has not been identified. The CPU 11 of the remote monitoring server 1 may determine that a gaze area has been identified and highlight the image when the line of towed vehicles TL is making an S-shaped turn. Furthermore, the method of detecting whether the line of towed vehicles TL is making an S-shaped turn is not limited to the example shown in FIG. 9 . The CPU 11 of the remote monitoring server 1 may determine that the line of towed vehicles TL is making an S-shaped turn when the side of either towed vehicle TL is captured in both the image captured by the right camera 50b and the image captured by the left camera 50c. When the line of towed vehicles TL is making an S-shaped turn, the CPU 11 of the remote monitoring server 1 may determine that the gaze area is captured in both the image captured by the right camera 50b and the image captured by the left camera 50c and highlight them.

[0092] As described above, according to the first embodiment, the remote monitoring server 1 acquires images of the surrounding environment of the towing vehicle TT captured by the cameras 50a-50d of the towing vehicle TT, which is equipped with one or more sensors, including the cameras 50a-50d (see, for example, S101 in FIG. 8 ). The remote monitoring server 1 then identifies the queue of towed vehicles TL based on the acquired images (see, for example, S102 in FIG. 8 and FIG. 9 ). The remote monitoring server 1 then identifies a focus area based on the queue of towed vehicles TL (see, for example, S103 in FIG. 8 ). The remote monitoring server 1 then outputs a monitoring screen that highlights the portion of the acquired image that corresponds to the focus area (see, for example, S106 in FIG. 8 ).

[0093] Therefore, the operator OP can recognize at a glance the areas that require his / her attention on the monitoring screen on which various images are displayed, without having to look over every corner of the monitoring screen, which reduces the burden on the operator OP of monitoring the situation around the vehicle VH.

[0094] Furthermore, according to the first embodiment, the remote monitoring server 1 detects the sides of the towed vehicle TL from the images captured by the right camera 50b and the left camera 50c (see, for example, S201 in FIG. 9).The remote monitoring server 1 then determines the state of the queue of towed vehicles TL based on the results of the detection of the sides of the towed vehicles TL (see, for example, S202 to S208 in FIG. 9).

[0095] Possible methods for determining the status of the line of towed vehicles TL include installing a new sensor in the towed vehicle TL or flying a drone or the like to capture images of the vehicles VH. However, because the towed vehicle TL does not have a power source, installing a sensor in the towed vehicle TL is cost-prohibitive. Furthermore, flying a drone requires adding a configuration for controlling the drone, which also inevitably increases costs. In the first embodiment, it is possible to determine the status of the line of towed vehicles TL without installing a new sensor in the towed vehicle TL or flying a drone or the like. Therefore, remote monitoring can be achieved at low cost.

[0096] Furthermore, according to the first embodiment, when the towed vehicle TL is in a turning state, the remote monitoring server 1 identifies the area on the side of the towed vehicle TL closest to the turning direction as the observation area (see, for example, Figure 4).

[0097] Therefore, the operator OP can easily recognize the part showing the area where an obstacle may be caught in the turn.

[0098] In the example described in the first embodiment, the CPU 11 of the remote monitoring server 1 detected the side of the towed vehicle TL from the images captured by the right camera 50b and the left camera 50c, and determined the state of the line of towed vehicles TL based on the detection results. The CPU 11 of the remote monitoring server 1 may also determine the state of the line of towed vehicles TL based on images captured by the rear camera 50d. Specifically, the CPU 11 of the remote monitoring server 1 acquires images captured by the rear camera 50d. The rear camera 50d is attached to the center of the towed vehicle TT in the width direction. Furthermore, the rear camera 50d is mounted on the towed vehicle TT with its optical axis facing directly behind the towed vehicle TT. When the line of towed vehicles TL is traveling straight, the image of each towed vehicle TL captured by the rear camera 50d is located on the center line of the image captured by the rear camera 50d. When the line of towed vehicles TL is turning right, the image of each towed vehicle TL in the image captured by the rear camera 50d is positioned to the right of the center line of the image captured by the rear camera 50d. When the line of towed vehicles TL is turning left, the image of each towed vehicle TL in the image captured by the rear camera 50d is positioned to the left of the center line of the image captured by the rear camera 50d. Therefore, the CPU 11 of the remote monitoring server 1 determines the state of the line of towed vehicles TL based on the deviation of the image of each towed vehicle TL in the image captured by the rear camera 50d from the center line of the image captured by the rear camera 50d.

[0099] In this way, the CPU 11 of the remote monitoring server 1 detects the towed vehicle TL from images captured by the camera 50 (right camera 50b, left camera 50c, or rear camera 50d) installed on the towing vehicle TT, and determines the state of the line of towed vehicles TL based on the detection results of the towed vehicles TL.

[0100] Second Embodiment In the first embodiment, the remote monitoring server 1 identifies the state of the queue of towed vehicles TL based on the images captured by the cameras 50a to 50d. The operation of identifying the state of the queue of towed vehicles TL is not limited to this.

[0101] In the second embodiment, as another example of the operation for identifying the line state of the towed vehicles TL, an operation using position information detected by the position sensor 52 will be described. Note that in the second embodiment, differences from the first embodiment will be described. Explanations of the same matters as in the first embodiment will be omitted.

[0102] In the second embodiment, the CPU 11 of the remote monitoring server 1 acquires, for example, images captured by the cameras 50a to 50d as well as position information of the towing vehicle TT detected by the position sensor 52. The CPU 11 then sequentially stores the acquired position information in the RAM 12 or the storage device 15, etc., thereby obtaining a history of the position information.

[0103] The history of position information can be regarded as the travel path of the towing vehicle TT. Since the towed vehicle TL is towed by the towing vehicle TT, the shape of the line of towed vehicles TL can be estimated from the travel path of the towing vehicle TT.

[0104] The CPU 11 identifies the state of the queue of towed vehicles TL based on the history of acquired position information.

[0105] FIG. 11 is a diagram for explaining the operation of identifying the state of a train of towed vehicles TL according to the second embodiment.

[0106] Each of the multiple points LH is a position indicated by the position information. From these multiple points LH, it can be seen that the towing vehicle TT is steering left as it moves forward. The four towed vehicles TL are moving along the trajectory connecting the multiple points LH, making a large left turn, i.e., a left turn.

[0107] In the case shown in FIG. 11, the CPU 11 determines that the line of towed vehicles TL is turning left based on the multiple points LH.

[0108] As described above, according to the second embodiment, the remote monitoring server 1 identifies the queue state of the towed vehicle TL based on the history of the position information of the towing vehicle TT detected by the position sensor 52.

[0109] Therefore, similar to the first embodiment, it is possible to identify the status of the line of towed vehicles TL without installing new sensors on the towed vehicles TL or flying drones, etc. Therefore, it is possible to realize remote monitoring at low cost.

[0110] Third Embodiment In the third embodiment, as yet another example of the operation for identifying the line state of the towed vehicles TL, an operation using the steering angle of the towing vehicle TT detected by the steering angle sensor 53 will be described. In the third embodiment, differences from the first embodiment will be described. Explanation of the same aspects as in the first embodiment will be omitted.

[0111] In the third embodiment, the CPU 11 of the remote monitoring server 1 acquires, for example, images captured by the cameras 50a to 50d as well as the steering angle of the towing vehicle TT detected by the steering angle sensor 53. Then, the CPU 11 identifies the state of the queue of towed vehicles TL based on the steering angle.

[0112] FIG. 12 is a flowchart showing the operation of identifying the state of the queue of towed vehicles TL according to the third embodiment.

[0113] When the towing vehicle TT is traveling straight, the CPU 11 acquires the distance that the towing vehicle TT has continuously traveled straight. The distance that the towing vehicle TT has continuously traveled straight is referred to as the straight-line distance of the towing vehicle TT.

[0114] The CPU 11 stores the straight-ahead distance of the towing vehicle TT, for example, in the RAM 12. Then, while the towing vehicle TT is traveling straight, the CPU 11 calculates the distance traveled at each predetermined control cycle and increments the straight-ahead distance of the towing vehicle TT by the calculated distance traveled. In this way, the CPU 11 obtains the straight-ahead distance of the towing vehicle TT in approximately real time.

[0115] The method for calculating the distance traveled for each control cycle is not limited to a specific method. For example, the CPU 11 calculates the distance traveled by the towing vehicle TT for each control cycle by using, for example, the history of position information of the towing vehicle TT detected by the position sensor 52 or wheel speed information from the wheel speed sensor 54.

[0116] In the operation for identifying the queue state of the towed vehicle TL, the CPU 11 first resets the stored straight-line distance of the towing vehicle TT (S301).Then, the CPU 11 determines whether the towing vehicle TT is traveling straight based on the steering angle detected by the steering angle sensor 53 (S302).

[0117] If the towing vehicle TT is traveling straight (S302: Yes), the CPU 11 updates the straight-line distance of the towing vehicle TT, i.e., increments it by the distance traveled (S303).Then, based on the straight-line distance of the towing vehicle TT, the CPU 11 determines whether the towing vehicle TT has traveled straight for the length of the line of towed vehicles TL (S304).

[0118] If the towing vehicle TT travels straight for the length of the line of towed vehicles TL (S304: Yes), the CPU 11 determines that the state of the line of towed vehicles TL is a straight line (S305), and the operation of determining the state of the line of towed vehicles TL ends.

[0119] If the towing vehicle TT does not travel straight for the length of the line of towed vehicles TL (S304: No), the control transitions to S302.

[0120] If the towing vehicle TT is not traveling straight (S302: No), the CPU 11 determines whether a reverse turn has occurred based on the steering angle detected by the steering angle sensor 53 (S306). A reverse turn is when the vehicle first turns in one direction and then turns in the other direction. The CPU 11 stores, for example, the immediately preceding state as a history in the RAM 12, and determines whether a reverse turn has occurred based on the immediately preceding state and the detected value of the current steering angle.

[0121] If a reverse turn has occurred (S306: Yes), the CPU 11 determines that the line of towed vehicles TL is in an S-turn state (S307), and the control transitions to S301.

[0122] If there is no reverse turn (S306: No), the CPU 11 determines whether the towing vehicle TT is turning left based on the steering angle detected by the steering angle sensor 53 (S308).

[0123] If the towing vehicle TT is turning left (S308: Yes), the CPU 11 determines that the line of towed vehicles TL is turning left (S309), and the control proceeds to S301. In S309, the CPU 11 stores, for example, in the RAM 12, a history indicating that the line of towed vehicles TL is turning left.

[0124] If the towing vehicle TT is not turning left (S308: No), the CPU 11 determines that the line of towed vehicles TL is turning right (S310), and the control proceeds to S301. In S310, the CPU 11 stores, for example, in the RAM 12, a history indicating that the line of towed vehicles TL is turning left.

[0125] As described above, according to the third embodiment, the remote monitoring server 1 identifies the state of the queue of towed vehicles TL based on the steering angle of the towing vehicle TT detected by the steering angle sensor 53 .

[0126] Therefore, similar to the first and second embodiments, it is possible to identify the status of the line of towed vehicles TL without providing new sensors to the towed vehicles TL or flying drones, etc. Therefore, it is possible to realize remote monitoring at low cost.

[0127] In the first to third embodiments, the remote monitoring server 1 may be configured to allow, for example, the operator OP to input the number of towed vehicles TL towed by each towing vehicle TT. The remote monitoring server 1 may use the input number to specify the status of the line of towing vehicles TL. Alternatively, the remote monitoring server 1 may be configured to obtain the number of towed vehicles TL towed by each towing vehicle TT from an external system. For example, if there is a management server in which the number of towed vehicles TL towed by each towing vehicle TT is registered, the remote monitoring server 1 may obtain the number of towed vehicles TL towed by each towing vehicle TT from the management server. Alternatively, if an infrastructure camera counts the number of towed vehicles TL towed by each towing vehicle TT and the counting results are recorded in the infrastructure camera or a management server managing the infrastructure camera, the remote monitoring server 1 may obtain the number of towed vehicles TL towed by each towing vehicle TT from the management server.

[0128] Furthermore, in the first embodiment, if the number of towed vehicles TL towed by the towing vehicle TT and the position of each towed vehicle TL from the front are known in advance, the CPU 11 of the remote monitoring server 1 may perform image recognition processing on images captured by the camera 50 (e.g., the right camera 50b, the left camera 50c, or the rear camera 50d) to identify the towed vehicle TL appearing in the image. If the CPU 11 of the remote monitoring server 1 is configured in this manner, the CPU 11 of the remote monitoring server 1 may display on the monitoring screen the remaining time until the gaze area is exhausted or the distance to be traveled until the gaze area is exhausted, based on the image from the camera 50 and the identification result of the towed vehicle TL appearing in the image. Furthermore, if a towed vehicle TL is appearing in the image displayed on the monitoring screen, the CPU 11 of the remote monitoring server 1 may display on the monitoring screen the position of the towed vehicle TL from the front.

[0129] Furthermore, if the CPU 11 of the remote monitoring server 1 is configured to detect the state of the line of towed vehicles TL based on the steering angle of the towing vehicle TT detected by the steering angle sensor 53 as shown in the third embodiment, the CPU 11 of the remote monitoring server 1 may display the state of the line of towed vehicles TL on the monitoring screen. The state of the line of towed vehicles TL may be displayed on the monitoring screen using text information such as straight-ahead status, right-turn status, left-turn status, or S-turn status, or an image object having a shape corresponding to the shape of the line of towed vehicles TL may be displayed on the monitoring screen. Furthermore, the CPU 11 of the remote monitoring server 1 may display on the monitoring screen the position of the towed vehicle TL in the line of towed vehicles TL that is captured in the image captured by the camera 50 displayed on the monitoring screen. This allows the operator OP to recognize, for example, where in the line of towed vehicles TL the towed vehicle TL is captured in the image captured by the camera 50 displayed on the monitoring screen, even if the line of towed vehicles TL is very long.

[0130] Furthermore, the CPU 11 of the remote monitoring server 1 may limit the speed of the towing vehicle TT if the entire line of towed vehicles TL does not fit within the image captured by the camera 50 displayed on the monitoring screen. Alternatively, the CPU 11 of the remote monitoring server 1 may display a notification on the monitoring image urging the user to limit the speed of the towing vehicle TT. This makes it possible to increase safety by limiting the vehicle speed when remote monitoring is not possible.

[0131] Furthermore, if the CPU 11 of the remote monitoring server 1 determines that the line of towed vehicles TL has become a straight-ahead state while the speed of the towing vehicle TT is limited, the CPU 11 of the remote monitoring server 1 may cancel the speed limit on the towing vehicle TT. Alternatively, the CPU 11 of the remote monitoring server 1 may display a notification on the monitoring image urging the user to cancel the speed limit on the towing vehicle TT.

[0132] As the line of towed vehicles TL approaches a straight-line state, the possibility of an obstacle being caught in the line decreases. Therefore, the CPU 11 of the remote monitoring server 1 may de-emphasize the portion of the line of towed vehicles TL that shows the gaze area before the line of towed vehicles TL becomes completely straight-lined. More specifically, the CPU 11 of the remote monitoring server 1 may continue to highlight the portion of the line of towed vehicles TL that shows the gaze area until a portion of the line of towed vehicles TL, i.e., a predetermined length at the beginning of the line of towed vehicles TL (referred to as the set length), becomes straight-lined, and then de-emphasize the portion of the line of towed vehicles TL that shows the gaze area when that portion of the line of towed vehicles TL becomes straight-lined. The set length may be dynamically changeable.

[0133] For example, if the operator OP is under a heavy load, such as if the operator OP is monitoring a large number of vehicles VH, monitoring the gaze area until the line of towed vehicles TL is moving completely straight would be a waste of man-hours and inefficient. Therefore, as described above, the CPU 11 of the remote monitoring server 1 can cancel the emphasis on the portion of the gaze area displayed before the line of towed vehicles TL is moving completely straight, thereby reducing the load on the operator OP.

[0134] Alternatively, the CPU 11 of the remote monitoring server 1 may estimate the load of the operator OP and, depending on the estimated load of the operator OP, switch between emphasizing the portion of the gaze area until the line of towed vehicles TL becomes completely straight, and canceling the emphasis on the portion of the gaze area before the line of towed vehicles TL becomes completely straight. For example, if the load of the operator OP is less than a predetermined judgment threshold, the CPU 11 of the remote monitoring server 1 may emphasize the portion of the gaze area until the line of towed vehicles TL becomes completely straight. If the load of the operator OP is greater than the judgment threshold, the CPU 11 of the remote monitoring server 1 cancels the emphasis on the portion of the gaze area before the line of towed vehicles TL becomes completely straight. The numerical information representing the load of the operator OP is not limited to specific numerical information. In one example, the CPU 11 of the remote monitoring server 1 may use the number of vehicles VH monitored by the operator OP as numerical information representing the load of the operator OP.

[0135] Furthermore, the CPU 11 of the remote monitoring server 1 may switch between emphasizing the portion of the gaze area until the line of towed vehicles TL becomes completely straight-moving and deemphasizing the portion of the gaze area before the line of towed vehicles TL becomes completely straight-moving, depending on the surrounding environment of the vehicle VH. For example, the CPU 11 of the remote monitoring server 1 estimates the risk of the vehicle VH colliding with an obstacle. The CPU 11 of the remote monitoring server 1 estimates the risk of the vehicle VH colliding with an obstacle based on any information, including the detection results of obstacles around the vehicle VH based on images captured by the distance measurement sensor 51 and the camera 50, information about the area in which the vehicle VH is traveling (such as the width of the road), and the like. If the risk of the vehicle VH colliding with an obstacle is higher than a predetermined judgment threshold, the CPU 11 of the remote monitoring server 1 emphasizes the portion of the gaze area until the line of towed vehicles TL becomes completely straight-moving. When the risk of the vehicle VH colliding with an obstacle is lower than the judgment threshold, the CPU 11 of the remote monitoring server 1 cancels the emphasis on the part showing the gaze area before the state of the line of towed vehicles TL becomes a completely straight-ahead state.

[0136] In the fourth embodiment, as variations of the method for highlighting the portion of the monitoring screen in which the gaze area is displayed, modified examples 1 to 6 will be described. The fourth embodiment can be applied to any of the first to third embodiments.

[0137] (Modification 1) FIG. 13 is a diagram for explaining an emphasis method according to Modification 1. In FIG.

[0138] The remote monitoring server 1 normally outputs the monitoring screen D1. When the gaze area is identified, the remote monitoring server 1 enlarges the portion of the image showing the gaze area.

[0139] 13 shows a monitoring screen D1b as an example of a monitoring screen when the gaze area is captured in an image captured by the left camera 50c. In the monitoring screen D1b, the width of the left image field F3 is enlarged compared to the monitoring screen D1, so that the image captured by the left camera 50c is displayed larger. The display position of the front image field F1 is shifted to the right by the amount of the enlargement of the width of the left image field F3, and the width of the right image field F2 is reduced.

[0140] 13, the left image field F3 is enlarged, so that the image capturing the gaze area captured by the left camera 50c is displayed in a larger size. The method for enlarging the image capturing the gaze area is not limited to this.

[0141] For example, when the remote monitoring server 1 crops the images captured by each camera 50 and displays them in the corresponding fields, it may expand the cropping range and reduce the cropped image so that the cropped image fits into the corresponding field. This reduces the scale of the display on the monitoring screen, but makes it possible to display a larger image that shows the area of ​​interest.

[0142] In this way, the remote monitoring server 1 may enlarge the portion showing the gaze area.

[0143] Furthermore, the remote monitoring server 1 may reduce the size of the portion that does not show the gaze area (for example, the right image field F2 in the example of FIG. 13).

[0144] The remote monitoring server 1 may also be configured not to display on the monitoring screen any portion that does not show the gaze area.

[0145] (Modification 2) FIG. 14 is a diagram for explaining an emphasis method according to Modification 2. In FIG.

[0146] In the second modification, the image showing the gaze area is enlarged by enlarging the corresponding field, but the position and size of the field displaying the image not showing the gaze area remain unchanged from the normal monitoring screen D1.

[0147] 14 shows a monitoring screen D1c as an example of a monitoring screen when the gaze area is captured in an image captured by the left camera 50c. In the monitoring screen D1c, the width of the left image field F3 is enlarged compared to the normal monitoring screen D1, thereby displaying the image captured by the left camera 50c at a larger size. However, the display positions and sizes of the front image field F1 and the right image field F2 remain unchanged. Therefore, a portion of the enlarged left image field F3 is displayed superimposed on the front image field F1.

[0148] (Variation 3) In Variation 3, multiple monitoring screens for monitoring different vehicles VH are displayed on the display device 26. That is, the CPU 11 of the remote monitoring server 1 acquires images captured by the cameras 50a-50d for each of the multiple towing vehicles TT (see, for example, S101 in FIG. 8 ), identifies the status of the line of towed vehicles TL (see, for example, S102 in FIG. 8 ), and identifies the area of ​​interest (see, for example, S103 in FIG. 8 ). The CPU 11 of the remote monitoring server 1 then outputs a monitoring screen for each of the multiple towing vehicles TT. The CPU 11 of the remote monitoring server 1 may output the multiple monitoring screens so that the multiple monitoring screens are displayed side by side, or may output the multiple monitoring screens so that the multiple monitoring screens are switched between displays.

[0149] 15 is a diagram for explaining the highlighting method according to Modification 3. In the example of this figure, a plurality of monitoring screens are displayed side by side.

[0150] Normally, that is, when there is no gaze area, multiple monitoring screens D2 for monitoring different vehicles VH are displayed side by side. Here, monitoring screens D2-1 and D2-2 are displayed side by side. Each of monitoring screens D2-1 and D2-2 has the same configuration as monitoring screen D1 shown in FIGS. 5, 13, and 14.

[0151] When a gaze area appears to the left of the vehicle VH being monitored by the monitoring screen D2-1 and the gaze area is identified, the monitoring screen D2a-1 is displayed in place of the monitoring screen D2-1. The monitoring screen D2a-1 is an example of a monitoring screen when an area A1 where an obstacle may be caught has appeared to the left of the corresponding vehicle VH1 (see FIG. 4) and the area A1 has been identified as the gaze area.

[0152] According to the monitoring screen D2a-1, the width of the left image field F3 is enlarged compared to the monitoring screen D2-1. The sizes of the front image field F1 and the right image field F2 remain unchanged. Due to the enlargement of the width of the left image field F3, the width of the monitoring screen D2a-1 is enlarged compared to the monitoring screen D2-1.

[0153] In this way, the remote monitoring server 1 may output a monitoring screen for a towing vehicle TT for which a gaze area has been identified larger than a monitoring screen for a towing vehicle TT for which a gaze area has not been identified, so that these monitoring screens are displayed side by side.

[0154] When multiple monitoring screens are switched and displayed, the remote monitoring server 1 may make the display time of the monitoring screen for a towing vehicle TT whose gaze area has been identified longer than the display time of the monitoring screen for a towing vehicle TT whose gaze area has not been identified.

[0155] Even when multiple monitoring screens are output, the remote monitoring server 1 may, for each monitoring screen, emphasize the portion showing the gaze area without changing the size of the monitoring screen, as in the first embodiment or variants 1 and 2.

[0156] (Modification 4) FIG. 16 is a diagram for explaining an emphasis method according to Modification 4. In FIG.

[0157] The remote monitoring server 1 outputs a gaze area display screen D10 in addition to the monitoring screen. The gaze area display screen D10 is a screen on which the portion of the gaze area is displayed only when the gaze area is specified.

[0158] When a gaze area is not specified, the monitoring screen and a gaze area display screen D10 on which no image is displayed are displayed.

[0159] When the gaze area is specified, the portion of the monitoring screen in which the gaze area is displayed is highlighted. Furthermore, the portion in which the gaze area is displayed is also displayed on the gaze area display screen D10.

[0160] 16, the monitoring screen D1d is a monitoring screen when an area A1 where an obstacle may be caught in the vehicle VH1 is located to the left of the vehicle VH1 (see FIG. 4), and the area A1 is identified as the focus area. The highlighted portion on the monitoring screen D1d is also displayed on the focus area display screen D10.

[0161] In this way, the monitoring screen may be provided with a dedicated screen on which the portion showing the gaze area is displayed.

[0162] (Modification 5) FIG. 17 is a diagram for explaining an emphasis method according to Modification 5. In FIG.

[0163] In the example shown in Figure 17, four monitoring screens D3a-1, D3-2, D3b-3, and D3-4 and a gaze area display screen D11 that displays the portion of the gaze area only when the gaze area is identified are displayed.

[0164] On the monitoring screen D3a-1, the image displayed in the left image field F3 is highlighted. The image displayed in the left image field F3 on the monitoring screen D3a-1 is also displayed as image G1 on the gaze area display screen D11.

[0165] On the monitoring screen D3b-3, the image displayed in the right image field F2 is highlighted. The image displayed in the right image field F2 of this monitoring screen D3b-3 is also displayed as image G2 on the gaze area display screen D11.

[0166] In this way, even when multiple monitoring screens for monitoring different vehicles VH are displayed on the display device 26, a screen showing the portion of the gaze area may be displayed simultaneously only when the gaze area is identified.

[0167] According to Modifications 3 and 5, multiple monitoring screens for monitoring different vehicles VH are displayed on the display device 26. When a gaze area is identified, the portion showing the gaze area is highlighted. When a gaze area is identified, the CPU 11 of the remote monitoring server 1 may highlight the monitoring screen for the towing vehicle TT for which the gaze area has been identified, instead of or in addition to the portion showing the gaze area. For example, if, among the multiple vehicles VH being monitored by a single remote monitoring terminal 2, there is a vehicle VH in the line of towed vehicles TL that is turning left or right, the CPU 11 of the remote monitoring server 1 may highlight the monitoring screen for the vehicle VH in the line of towed vehicles TL that is turning left or right.

[0168] (Variation 6) In the first to third embodiments and variations 1 to 5, the portion showing the gaze area is emphasized in units of images displayed in fields F1 to F4. The CPU 11 of the remote monitoring server 1 may emphasize a partial region of each of the images displayed in fields F1 to F4.

[0169] 4 is identified as the gaze area, the CPU 11 of the remote monitoring server 1 may emphasize the portion of the image displayed in the left image field F3 that shows area A1, and may not emphasize the portion of the image displayed in the left image field F3 that shows an area different from area A1. In other words, when the gaze area is identified, the CPU 11 of the remote monitoring server 1 may emphasize only the portion that shows the gaze area.

[0170] The CPU 11 of the remote monitoring server 1 may emphasize only the portion corresponding to the gaze area in the image captured by the cameras 50a to 50d, or may emphasize the image including the portion corresponding to the gaze area, as long as it emphasizes the portion corresponding to the gaze area.

[0171] Fifth Embodiment In the fifth embodiment, as variations of the area identified as the gaze area, Modifications 7 and 8 will be described. The fifth embodiment can be applied to any of the first to fourth embodiments.

[0172] 18 is a diagram illustrating a gaze area according to Modification 7. As shown in the example, the CPU 11 of the remote monitoring server 1 may specify, as the gaze area, an area A2 whose distance from the towing vehicle TT or the towed vehicle TL is equal to or less than a first threshold value.

[0173] (Variation 8) In Variation 8, the CPU 11 of the remote monitoring server 1 acquires detection data from the distance measurement sensor 51 from the vehicle control device 40 of the towing vehicle TT. If the distance measurement sensor 51 detects the presence of an obstacle and the distance between the towing vehicle TT or towed vehicle TL and the obstacle is equal to or less than the second threshold value, the CPU 11 identifies the area between the towing vehicle TT or towed vehicle TL and the obstacle as the focus area.

[0174] 19 is a diagram illustrating the gaze area according to Modification 8. As shown in this diagram, when the towing vehicle TT approaches the obstacle 300 and the distance between the towing vehicle TT and the obstacle 300 is equal to or less than the second threshold value, the CPU 11 of the remote monitoring server 1 identifies an area A3 between the towing vehicle TT and the obstacle 300 as the gaze area.

[0175] When the towing vehicle TT continues to move forward and the distance from not only the towing vehicle TT but also the towed vehicle TL to the obstacle 300 becomes equal to or less than the second threshold value, an area A3a sandwiched between the towing vehicle TT or the towed vehicle TL and the obstacle is identified as the observation area.

[0176] In this way, if the distance measurement sensor 51 detects the presence of an obstacle and the distance between the towing vehicle TT or towed vehicle TL and the obstacle is equal to or less than the second threshold value, the CPU 11 of the remote monitoring server 1 may specify the area between the towing vehicle TT or towed vehicle TL and the obstacle as the gaze area. When configured as described above, the CPU 11 of the remote monitoring server 1 will continue to output a monitoring screen that highlights the portion of the gaze area for a time period corresponding to the length of the towed vehicle TL, i.e., the total length of the line of towed vehicles TL being towed by the towing vehicle TT.

[0177] When the CPU 11 of the remote monitoring server 1 detects another obstacle (hereinafter referred to as a second obstacle), the CPU 11 may determine whether to display the second obstacle based on whether the second obstacle is present within the gaze area (the gaze area of ​​Modification 7 or any of the above-described embodiments). For example, if the second obstacle is present within the gaze area, the CPU 11 of the remote monitoring server 1 displays the second obstacle with or without emphasis. If the second obstacle is not present within the gaze area, the CPU 11 of the remote monitoring server 1 does not display the second obstacle.

[0178] The CPU 11 of the remote monitoring server 1 may detect an obstacle and the distance between the towed vehicle TL and the obstacle based on an image captured by the camera 50 of the towing vehicle TT, instead of or in addition to the detection data from the distance measurement sensor 51. Furthermore, if the distance between the obstacle and the towed vehicle TL, detected based on the image captured by the camera 50 of the towing vehicle TT, is equal to or less than a second threshold value, instead of or in addition to the detection data from the distance measurement sensor 51, the CPU 11 of the remote monitoring server 1 may notify the operator OP or send an instruction to the towing vehicle TL to stop the towing vehicle TL.

[0179] The towed vehicle TL is often not equipped with a sensor that can be used to detect obstacles, such as a distance sensor or a camera. In such cases, even if an obstacle is detected by the towing vehicle TT's sensors and then becomes undetectable, the towed vehicle TL may still collide with the obstacle. Therefore, even if an obstacle is detected by the towing vehicle TT's sensors and then becomes undetectable, the CPU 11 of the remote monitoring server 1 may prohibit steering toward the obstacle from the time the obstacle becomes undetectable until the towed vehicle TL has traveled the length of the train.

[0180] If the CPU 11 of the remote monitoring server 1 is configured to detect obstacles based on images captured by the cameras 50 of the towing vehicle TT, there is a possibility that the obstacle will enter the blind spot of the cameras 50 as the towing vehicle TT travels. For example, if the vehicle VH travels near an obstacle 301 as shown in FIG. 21 , the obstacle 301 will not fit completely within the field of view R2 of the right camera 50b, and an area A5 between the obstacle 301 and the towed vehicle TL will be in the blind spot of the right camera 50b. It is also difficult to capture an image of area A5 with the rear camera 50d. Therefore, the CPU 11 of the remote monitoring server 1 cannot display an image of area A5. The CPU 11 of the remote monitoring server 1 may notify the operator OP if a detected obstacle enters a blind spot that is not captured by any of the cameras 50 installed on the towing vehicle TT as the vehicle VH travels. Alternatively, instead of notifying the operator OP, or in addition to notifying the operator OP, the CPU 11 of the remote monitoring server 1 may stop the tow vehicle TT, temporarily prohibit the operator OP from steering in the direction of the obstacle, limit the driving speed of the vehicle VH, make the vehicle VH make a wide turn, or notify the operator of an area that has entered the blind spot (area A5 in the example shown in Figure 21).

[0181] Furthermore, the CPU 11 of the remote monitoring server 1 may be configured to change the driving route depending on the length of the line of towed vehicles TL or the number of towed vehicles TL towed by the towing vehicle TT. For example, the longer the line of towed vehicles TL or the more towed vehicles TL towed by the towing vehicle TT, the CPU 11 of the remote monitoring server 1 may determine a driving route such that the distance to an obstacle increases.

[0182] Sixth Embodiment In a sixth embodiment, the CPU 11 of the remote monitoring server 1 outputs, in addition to the monitoring screen, a screen showing areas not shown in the images captured by the cameras 50a to 50d.

[0183] 20 is a diagram showing an example of a screen according to the sixth embodiment, which shows an area not captured in the images captured by the cameras 50a to 50d. Area A4 shown in this diagram is in a blind spot for all of the cameras 50a to 50d, and is not captured in any of the images captured by the cameras 50a to 50d. Therefore, the CPU 11 of the remote monitoring server 1 outputs the image shown in this diagram, highlighting area A4.

[0184] The sixth embodiment can be applied to any of the first to fifth embodiments.

[0185] The remote monitoring program 100, which is a computer program executed by the remote monitoring server 1 of the first to sixth embodiments, is pre-stored in the ROM 13 of the remote monitoring server 1. The remote monitoring program 100 may be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD (Compact Disc)-ROM (Read Only Memory), a flexible disk (FD), a CD-R (Recordable), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, or an SD (Secure Digital) card.

[0186] Furthermore, the remote monitoring program 100 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0187] A part or all of the processing executed by the CPU 11 in accordance with the remote monitoring program 100 may be realized by a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0188] Although several embodiments of the present disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents.

[0189] REFERENCE SIGNS LIST 1 Remote monitoring server 2 Remote monitoring terminal 3 Network 11 CPU 12 RAM 13 ROM 14 Network interface 15 Storage device 16 Bus 21 CPU 22 RAM 23 ROM 24 Network interface 25 Input device 26 Display device 27 Bus 40 Vehicle control device 41 CPU 42 RAM 43 ROM 44 Network interface 45 I / O interface 46 Bus 50 Camera 50a Front camera 50b Right camera 50c Left camera 50d Rear camera 51 Distance sensor 52 Position sensor 53 Steering angle sensor 54 Wheel speed sensor 100 Remote monitoring program 300, 301 Obstacle 1000 Remote monitoring system TT Towing vehicle TL Towed vehicle

Claims

1. A remote monitoring method comprising: a first step of acquiring sensor data output by one or more sensors including a camera from a towing vehicle equipped with one or more sensors including a camera, the sensor data including a first image capturing the surrounding environment of the towing vehicle captured by the camera; a second step of identifying a state of a towed vehicle being towed by the towing vehicle based on the sensor data; a third step of identifying a first area requiring attention based on the identified state; and a fourth step of outputting a second image in which a first portion of the first image corresponding to the first area is highlighted.

2. A remote monitoring method as described in claim 1, wherein the second step includes a step of detecting the towed vehicle from the first image, and a step of identifying the state of the towed vehicle based on the detection result of the towed vehicle.

3. The remote monitoring method described in claim 2, wherein the cameras include a first camera that captures images of the surrounding environment to the left of the towing vehicle and a second camera that captures images of the surrounding environment to the right of the towing vehicle, and the second step includes a step of detecting the sides of the towed vehicle from images captured by the first camera and images captured by the second camera, and a step of identifying the condition based on the side detection results.

4. The remote monitoring method of claim 2, wherein the camera includes a third camera that images the surrounding environment behind the towing vehicle, and the second step includes a step of identifying the condition based on a deviation of an image of the towed vehicle from a center line of an image captured by the third camera.

5. The remote monitoring method according to claim 1, wherein the one or more sensors include a position sensor, and the second step includes a step of identifying the state based on a history of position information of the towing vehicle detected by the position sensor.

6. The remote monitoring method according to claim 1, wherein the one or more sensors include a steering angle sensor, and the second step includes a step of identifying the state based on a steering angle detected by the steering angle sensor.

7. The remote monitoring method according to claim 1, wherein the third step includes a step of identifying an area on one of the two sides of the towed vehicle in the turning direction as the first area when the towed vehicle is in a turning state.

8. The remote monitoring method according to claim 1, further comprising the step of identifying an area whose distance from the towing vehicle or the towed vehicle is equal to or less than a first threshold as the first area.

9. The remote monitoring method described in claim 1, wherein the third step includes a step of identifying an area between the towing vehicle or the towed vehicle and the obstacle as the first area when the distance between the towing vehicle or the towed vehicle and the obstacle is equal to or less than a second threshold value.

10. The remote monitoring method according to claim 9, wherein said fourth step further comprises a step of continuing to output said second image for a time period corresponding to the length of said towed vehicle.

11. A remote monitoring method as described in any one of claims 1 to 10, wherein highlighting the first portion comprises: putting a frame around the first portion; enlarging the first portion; making a second portion of the first image that is different from the first portion smaller; or displaying the first portion without displaying the second portion.

12. A remote monitoring method as described in any one of claims 1 to 10, further comprising the steps of performing the first step, the second step, and the third step for each of a plurality of towing vehicles, wherein the fourth step includes the steps of generating the second image for a first towing vehicle of the plurality of towing vehicles for which the first area has been identified, and generating a third image not including a highlighted portion for a second towing vehicle of the plurality of towing vehicles for which the first area has not been identified, and outputting the second image and the third image.

13. The remote monitoring method of claim 12, wherein the fourth step includes a step of making the second image larger than the third image and outputting the second image and the third image so that the second image and the third image, made larger than the third image, are displayed side by side.

14. The remote monitoring method according to claim 12, wherein the fourth step includes a step of switching between the second image and the third image and outputting the second image and making the display time of the second image longer than the display time of the third image.

15. A remote monitoring method as claimed in any one of claims 1 to 10, further comprising a fifth step of outputting a fourth image showing a second area not shown in the first image.

16. A remote monitoring terminal comprising: an interface for receiving a second image generated by emphasizing a first portion of a first image showing the surrounding environment of a towing vehicle captured by a camera equipped in the towing vehicle, the second image corresponding to a first area requiring attention in the first image; and a display device for displaying the second image.

17. A remote monitoring program that causes a computer to execute the following steps: a first step of acquiring sensor data output by one or more sensors including a camera from a towing vehicle equipped with one or more sensors, the sensor data including a first image capturing the surrounding environment of the towing vehicle captured by the camera; a second step of identifying the condition of a towed vehicle being towed by the towing vehicle based on the sensor data; a third step of identifying a first area requiring attention based on the identified condition; and a fourth step of outputting a second image in which a first portion of the first image corresponding to the first area is highlighted.

Images