Remote system

Abstract

To provide a remote system in which fixation of a video displayed on a display device of a remote terminal is detected and appropriate measures against the fixation of the video can be taken.SOLUTION: A remote system 101 of the present disclosure includes a vehicle 10 equipped with a camera 11 and a remote terminal 20 having a display device 22. The vehicle 10 and the remote terminal 20 are connected via a communication network 30. The vehicle 10 is configured to execute: generating a synthesized video in which a moving object video changing over time is synthesized with a camera video captured by the camera 11, and transmitting the synthesized video to the remote terminal 20. The remote terminal 20 displays the synthesized video on the display device 22, detects fixation of a video displayed on the display device 22 based on a change in the moving object video included in the synthesized video, and outputs an abnormality signal in response to detection of the fixation of the video.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] , ,

[0006] , ,

[0005] , , , ,

[0001] The present disclosure relates to a remote system in which a vehicle and a remote terminal are connected via a communication network.

Background Art

[0002] Japanese Unexamined Patent Application Publication No. 2020-180466 discloses a monitoring system that enables visual recognition of changes in images between the present and the past by transparently synthesizing the current image acquired by a camera and the image recorded in the past.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] [[ID=3S]]Here, consider applying the above monitoring system to a vehicle remote system. A vehicle remote system is a system that transmits an image captured by an in-vehicle camera to a remote terminal via a communication network and displays the image on a display device of the remote terminal.

[0005] In the remote system, a failure may occur in which the image displayed on the display device is fixed, that is, a moving image becomes a still image. When the image is fixed, since the current image and the past image are the same, it becomes meaningless to compare the two. That is, considering the image fixation that may occur in the monitoring system, it is difficult to apply the above monitoring system to the remote system. In addition, image fixation often occurs due to interruption of the communication network. Therefore, even if the method is to compare the current image and the past image in the vehicle and transmit the result to the remote terminal, it is still difficult to apply the above monitoring system to the remote system.

[0006] This disclosure has been made in view of the above-mentioned issues. One objective of this disclosure is to enable a vehicle remote system to detect when an image displayed on a remote terminal's display device is stuck and to take appropriate action against the image stuck. [Means for solving the problem]

[0007] This disclosure provides a remote system for achieving the above objective. The remote system of this disclosure comprises a vehicle equipped with a camera and a remote terminal having a display device. The vehicle and the remote terminal are connected via a communication network. The vehicle is configured to generate a composite image by combining camera footage captured by the camera with motion footage that changes over time, and to transmit the composite image to the remote terminal. The remote terminal is configured to display the composite image on a display device, to detect image fixation on the display device based on changes in motion footage included in the composite image, and to output an abnormal signal upon detection of image fixation. [Effects of the Invention]

[0008] According to the remote system of this disclosure, the display device of the remote terminal displays a composite image, which is a combination of the camera image captured by the camera and a moving image that changes over time. If the composite image displayed on the display device becomes fixed, the moving image, which should be changing over time, will also stop changing. Therefore, the fact that the image is fixed can be detected based on the changes in the moving image. According to the remote system of this disclosure, an abnormal signal is output in response to the detection of image fixation, so it is possible to take appropriate action against image fixation. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing the configuration of the remote system according to the first embodiment. [Figure 2] This figure shows a specific example of a method for generating a composite image by combining camera footage with moving imagery. [Figure 3]This figure shows a specific example of a method for detecting image fixation based on changes in moving images. [Figure 4] This figure shows an example of an application of the remote system according to the first embodiment. [Figure 5] This is a diagram showing the configuration of the remote system according to the second embodiment. [Figure 6] Figure 6A shows the camera configuration according to the third embodiment, Figure 6B shows the camera configuration according to the fourth embodiment, and Figure 6C shows the remote terminal configuration according to the fifth embodiment. [Modes for carrying out the invention]

[0010] Embodiments of the remote system of this disclosure will be described below with reference to the figures. The remote system includes a remote monitoring system, a remote support system, and a remote driving system. The remote monitoring system is configured to allow a remote operator to monitor video transmitted from an in-vehicle camera. The video to be monitored includes video of the vehicle's surroundings, including the direction of travel, and video of the interior of the vehicle showing the occupants. Vehicles to which the remote monitoring system applies include manually driven vehicles operated by a driver on board, and autonomous vehicles operated by an autonomous driving system. The remote support system is configured to allow an autonomous vehicle, when it encounters a situation where it is unsure of what to do, to request assistance from a remote operator and act according to instructions from the remote operator. The remote driving system is configured to allow a remote operator to drive the vehicle remotely. The remote system of this disclosure is applicable to any of the remote monitoring system, remote support system, and remote driving system.

[0011] 1. First Embodiment 1-1. Remote System Configuration Figure 1 shows the configuration of a remote system 101 according to the first embodiment of this disclosure. The remote system 101 is a system in which a vehicle 10 and a remote terminal 20 are connected via a communication network 30. The vehicle 10 is configured to connect to the communication network 30 via mobile communication such as LTE or 5G. The remote terminal 20 is an operating terminal operated by a remote operator. If the remote system 101 is a remote driving system or a remote support system, the remote terminal 20 is also referred to as a remote cockpit. The communication network 30 may include one or more servers. In other words, communication between the vehicle 10 and the remote terminal 20 may be conducted directly or via a server.

[0012] Vehicle 10 is equipped with a camera 11, a motion image generator 12, a video synthesizer 13, and a video encoder 14. Camera 11 is an external camera that is pointed outwards from vehicle 10 and captures the area around vehicle 10, including the direction of travel. However, if vehicle 10 is equipped with an in-vehicle camera that is pointed inwards to capture the occupants inside the vehicle, the in-vehicle camera may also be included in camera 11. In other words, camera 11 represents the various cameras mounted on vehicle 10.

[0013] The motion image generator 12, the video synthesizer 13, and the video encoder 14 may be physically constructed circuits or programs running on an in-vehicle computer. The motion image generator 12 is configured to generate motion images that change over time. The video synthesizer 13 is configured to combine the motion images generated by the motion image generator 12 with camera images captured by the camera 11. The video encoder 14 is configured to encode the composite image, which is the camera image with the motion images combined, and to transmit the encoded composite image to the remote terminal 20. The transmission of video from the vehicle 10 to the remote terminal 20 may be continuous or limited to a predetermined period of time.

[0014] The remote terminal 20 includes a video decoder 21, a display device 22, a light sensor 23, and a change detector 24. The video decoder 21 is configured to decode the composite video transmitted from the vehicle 10 and display the decoded composite video on the display device 22. The display device 22 may have multiple screens depending on the number of cameras 11, or it may display multiple videos side by side on a single screen. The light sensor 23 is attached to the screen of the display device 22 and is configured to extract as a signal changes in the moving image included in the composite video displayed on the display device 22. The change detector 24 is configured to detect the fixation of the image displayed on the display device 22 from the signal output from the light sensor 23. The change detector 24 outputs an abnormal signal upon detecting image fixation. The video decoder 21 and the change detector 24 may be physically configured circuits or programs that run on a computer.

[0015] 1-2. Method for generating composite images and method for detecting image fixation Next, the method for generating composite images in the remote system 101 and the method for detecting image fixation will be explained in detail using diagrams.

[0016] Figure 2 shows a specific example of a method for generating a composite image 43. As shown in Figure 2, a motion image 42 is composited with a camera image 41 to generate a composite image 43. The motion image 42 is an image of a moving object that undergoes visual changes over time. Visual changes include, for example, changes in brightness, color, and shape. Changes over time include, for example, changes at a constant frequency and changes at random timings. A typical motion image 42 is an image of a moving object whose brightness changes at a constant frequency, for example, an image of a lamp blinking at a constant frequency. However, the frequency of the changes in the motion image 42 is smaller than the frame rate of the camera image 41.

[0017] The moving image 42 is adjusted to a sufficiently small size with respect to the camera image 41 so as not to reduce the amount of information obtained from the camera image 41. Also, the position where the composite image 43 is composited with respect to the camera image 41 is fixed. When the composite image 43 is displayed on the display device 22, the moving image 42 is composited with the camera image 41 such that the moving image 42 is located at a position far from the center of the screen of the display device 22, typically at the corner of the screen. By setting the position of the moving image 42 to the corner of the screen, it is possible to suppress the inhibition of concentration on the image of the remote driver (camera image).

[0018] FIG. 3 is a diagram showing a method for detecting image sticking based on the change of the moving image 42 displayed on the display device 22. The composite image 43 including the moving image 42 is displayed on the display device 22. The camera images constituting the composite image 43 change according to the change over time of the object photographed by the camera 11. On the other hand, the moving image 42 changes regularly at a constant frequency. T1, T2, T3, T4, T5, T6, T7, and T8 shown in FIG. 3 are the times when the blinking of the lamp as the moving image 42 should switch, and the intervals between the times are constant.

[0019] In the example shown in FIG. 3, the lamp is on (bright) at time T1, off (dark) at time T2, on at time T3, off at time T4, on at time T5, and off at time T6. If the lamp switches between bright and dark in such a cycle, the lamp should be on at time T7. However, in the example shown in FIG. 3, the lamp is off at time T7 and also off at time T8. This means that the moving image 42 has not switched, that is, the composite image 43 including the moving image 42 is stuck. It can be determined that the image is stuck after time T7. Therefore, an abnormal signal is output at the timing after time T7 when the sticking of the image is detected. By outputting an abnormal signal in response to the detection of image sticking, it becomes possible to take appropriate measures against image sticking.

[0020] 1-3. Application Examples Next, an application example in which an emergency stop function is added to the remote system 101 described above will be explained using Figure 4. However, in Figure 4, elements that are the same as those shown in Figure 1 in terms of elements and functions are given the same reference numerals, and the explanations of those elements are omitted or simplified.

[0021] In the application example shown in Figure 4, the abnormal signal output from the change detector 24 is used as a stop signal to automatically stop the vehicle 10. Alternatively, the abnormal signal output from the change detector 24 is used as a stop signal to illuminate the warning light 25. In the latter case, when the remote operator 50 sees the warning light 25 illuminate, they press the emergency stop button 26, which outputs a stop signal. In other words, the remote system 101 may be configured to output a stop signal automatically, or it may be configured to output a stop signal in response to a manual stop operation by the remote operator 50.

[0022] The stop signal is transmitted to the vehicle controller 15 mounted on the vehicle 10 via the communication network 30. Because the bit information of the stop signal is small, it is possible to transmit the stop signal even when video transmission is interrupted, as long as the video freeze is not due to a communication network interruption. Upon receiving the stop signal, the vehicle controller 15 activates the brakes 16 to prevent the vehicle 10 from starting, or if the vehicle is moving, activates the brakes 16 to decelerate and bring the vehicle 10 to a stop. At the same time, the vehicle controller 15 controls the powertrain 18 to suppress the driving force, and after the vehicle has stopped, puts the transmission 17 into park to maintain the stopped state.

[0023] 2. Second Embodiment Figure 5 shows the configuration of the remote system 102 according to the second embodiment of this disclosure. However, in Figure 5, elements that are the same as those shown in Figure 1 in terms of elements and functions are denoted by the same reference numerals, and the descriptions of those elements are omitted or simplified.

[0024] In the remote system 102, a speed signal output from a speed sensor (not shown) is input to the motion image generator 12 and the image synthesizer 13. The motion image generator 12 determines whether the vehicle 10 is stopped from the speed signal and generates motion images only when the vehicle 10 is stopped. The image synthesizer 13 determines whether the vehicle 10 is stopped from the speed signal and synthesizes the motion images with the camera images only when the vehicle 10 is stopped. In other words, in the remote system 102, the synthesized image with motion images is generated only when the vehicle 10 is stopped.

[0025] In the remote system 102, a speed signal is transmitted from the vehicle 10 to the remote terminal 20 along with the video. The video transmitted from the vehicle 10 to the remote terminal 20 is a composite video created by combining moving images when the vehicle 10 is stationary, but is a camera image when the vehicle 10 is moving. Therefore, the display device 22 displays the composite video only when the vehicle 10 is stationary, and only the camera image when the vehicle 10 is moving. The speed signal is input to the change detector 24. The change detector 24 determines when the vehicle 10 is stationary from the speed signal and detects image fixation from the signal output from the light sensor 23 only when the vehicle 10 is stationary.

[0026] The problem of image freezing is particularly noticeable when the vehicle 10 is stationary, as the view from camera 11 does not change significantly. By activating the image freezing detection function only when the vehicle 10 is stationary, the generation of moving images while the vehicle is in motion is eliminated, thereby reducing the processing load on the vehicle 10. Furthermore, since moving images that change, such as flashing, are not displayed on the display device 22 while the vehicle is in motion, the remote operator 50's ability to concentrate on the image (camera image) is not hindered.

[0027] 3. Other Embodiments In the first and second embodiments, motion images are composited with camera images downstream of camera 11. However, as described in the third and fourth embodiments below, the compositing of motion images can also be performed inside camera 11.

[0028] The third embodiment employs an optical method for synthesizing motion images inside the camera 11. Figure 6A shows the configuration of the camera 11 according to the third embodiment. As shown in Figure 6A, a prism 110 is placed in front of the image sensor 111 inside the camera 11. The prism 110 receives light from the camera 11's external field of view, as well as light from a motion object 112 located inside the camera 11. The motion object 112 is, for example, a lamp that blinks at a constant frequency. The light from the motion object 112 is projected from the prism 110 onto the image sensor 111 along with the light from the external field of view. As a result, a composite image with the image of the motion object 112 is obtained inside the camera 11. The composite image output from the camera 11 is encoded by a video encoder 14 and transmitted to a remote terminal 20 via a communication network 30. With this configuration, not only the communication network 30, but also the image sensor 111 and signal processing within the camera 11 can be collectively monitored for their operational status. In the configuration shown in Figure 6A, a half-mirror may be installed instead of the prism 110 as a means for projecting light from the moving object 112 onto the image sensor 111.

[0029] The fourth embodiment employs an electrical method for synthesizing motion images inside the camera 11. Figure 6B shows the configuration of the camera 11 according to the fourth embodiment. As shown in Figure 6B, a readout signal processor 113 is located behind the image sensor 111 inside the camera 11. The readout signal processor 113 receives the readout signal read from the image sensor 111 along with the motion signal from the motion signal output circuit 114 provided inside the camera 11. The motion signal output circuit 114 outputs a motion signal that changes over time, for example, a signal that flashes at a constant frequency, mimicking a lamp. The readout signal processor 113 superimposes the motion signal onto the readout signal read from the image sensor 111. As a result, a composite image with motion images is obtained inside the camera 11. The composite image output from the camera 11 is encoded by the video encoder 14 and transmitted to the remote terminal 20 via the communication network 30.

[0030] In the first and second embodiments, a light sensor 23 is attached to the screen of the display device 22 to detect changes in the moving image. However, as in the fifth embodiment described below, it is also possible to detect changes in the moving image directly from the image itself.

[0031] Figure 6C is a diagram showing the configuration of the remote terminal 20 according to the fifth embodiment. As shown in Figure 6C, the composite video decoded by the video decoder 21 is input to the video extractor 27 in parallel with the display device 22. The video extractor 27 is configured to extract motion images included in the composite video from the composite video. The motion images extracted by the video extractor 27 are input to the change detector 24. If there is no change in the extracted motion images, it can be determined that the composite video including the motion images is fixed. The change detector 24 is configured to detect the fixation of the image from the signal output from the video extractor 27. Upon detecting the fixation of the image, the change detector 24 outputs an abnormal signal. In the configuration shown in Figure 6C, instead of the composite video decoded by the video decoder 21, the image obtained by removing motion images from the composite video in the video extractor 27 may be input to the display device 22. The configuration of the remote terminal 20 according to the fifth embodiment can be combined with the third embodiment, and it can also be combined with the fourth embodiment. [Explanation of Symbols]

[0032] 10 vehicles 11 Cameras 20 Remote terminals 22 Display device 30 Communication Networks 101, 102 Remote Systems

Claims

1. A vehicle equipped with a camera, A remote terminal comprising a display device and an optical sensor, The vehicle and the remote terminal are connected via a communication network. The aforementioned vehicle shall only operate when the aforementioned vehicle is stopped. The process involves generating a composite image by combining the camera image captured by the aforementioned camera with a moving image that changes over time, The system is configured to transmit the synthesized video to the remote terminal, The aforementioned remote terminal is Displaying the composite image on the display device, The optical sensor outputs a signal based on the change in the motion image included in the image displayed on the display device, Based on the signal output from the optical sensor, the fixation of the image displayed on the display device is detected. The system is configured to output an abnormal signal upon detection of the aforementioned sticking. A remote system characterized by the following features.

2. In the remote system described in claim 1, The abnormal signal is used as a stop signal to warn the remote operator operating the remote terminal to stop the vehicle. A remote system characterized by the following features.

3. In the remote system described in claim 1, The aforementioned abnormal signal is used as a stop signal to automatically stop the vehicle. A remote system characterized by the following features.

4. In the remote system according to any one of claims 1 to 3, The vehicle is configured to generate the composite image by projecting a time-varying moving object onto the image sensor of the camera, or by superimposing a time-varying moving object signal onto the readout signal from the image sensor of the camera. A remote system characterized by the following features.

Images