Remote support system and remote support method

Abstract

To secure an accuracy of a remote support of a mobile body using an infrastructure camera.SOLUTION: A remote support system is a system for performing a remote support of an object mobile body. The remote support system acquires a first video image photographed by a first infrastructure camera installed in an object area where the mobile body is moved, via a communication. The remote support system presents the first video image to the remote supporter performing the remote support of the mobile body. In the case where an abnormality of the first video image occurs, the remote support system cancels the abnormality of the first video image, or executes an abnormal correspondence processing that indicates a replacement video image as a replacement of the first video image to the remote supporter.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] This disclosure relates to a technology for remotely supporting a moving object using an infrastructure camera.

Background Art

[0002] Patent Document 1 discloses a technology for remotely operating a vehicle.

[0003] Non-Patent Document 1 discloses an image recognition technology using ResNet (Deep Residual Net).

[0004] Non-Patent Document 2 discloses a technology (EnlightenGAN) for converting a low-illumination image into a normal-light image using deep learning. Thereby, for example, an image captured in a scene such as at night or against the light can be corrected to an appropriate brightness.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Documents

[0006]

Non-Patent Document 1

Non-Patent Document 2

[0007] This section considers remote support (remote monitoring, remote assistance, remote operation) of mobile devices by remote supporters. Generally, cameras mounted on the mobile device capture the surrounding environment, and this footage is presented to the remote supporter. The remote supporter views the presented footage, recognizes the environment around the mobile device, and provides remote support for the mobile device.

[0008] In addition to cameras mounted on mobile devices, infrastructure cameras could also be used for remote support. In this case, the infrastructure cameras would capture images of the mobile device and its surroundings, and the images showing the mobile device would be presented to the remote supporter. This is expected to further improve the accuracy of remote support. However, if an anomaly occurs in the images presented to the remote supporter, the improvement in the accuracy of remote support may not be fully realized.

[0009] One of the purposes of this disclosure is to provide a technology that can ensure the accuracy of remote support for mobile objects using infrastructure cameras. [Means for solving the problem]

[0010] The first aspect relates to remote support systems for providing remote support to mobile devices. The remote support system comprises one or more processors. One or more processors acquire, via communication, a first video image captured by a first infrastructure camera installed in the target area where a mobile object is moving. One or more processors present the first image to a remote supporter who provides remote support for a mobile object. If an anomaly occurs in the first video, one or more processors will perform anomaly response processing to resolve the anomaly in the first video, or to present an alternative video to the remote supporter.

[0011] The second aspect relates to remote support methods performed by computers for providing remote support to mobile objects. Remote support methods are: The first video footage, captured by a first infrastructure camera installed in the target area where a mobile object is moving, is acquired via communication. The first video will be shown to the remote supporter who provides remote support for the mobile device, If an anomaly occurs in the first video, an anomaly response process will be performed to resolve the anomaly in the first video, or to present an alternative video to the remote supporter. Includes. [Effects of the Invention]

[0012] According to this disclosure, a first video image captured by a first infrastructure camera is presented to a remote supporter. If an anomaly occurs in the first video image, an anomaly response process is executed. The anomaly response process either resolves the anomaly in the first video image or presents an alternative video image to the remote supporter. The remote supporter can then continue providing remote support by referring to the first video image or the alternative video image after the anomaly has been resolved. This ensures the accuracy of the remote support. [Brief explanation of the drawing]

[0013] [Figure 1] This is a schematic diagram showing an example configuration of a remote support system according to an embodiment. [Figure 2]It is a conceptual diagram for explaining the abnormality determination process in the remote support system according to the embodiment. [Figure 3] It is a conceptual diagram for explaining the abnormality handling process in the remote support system according to the embodiment. [Figure 4] It is a conceptual diagram for explaining the process related to the alternative video presentation process according to the embodiment. [Figure 5] It is a conceptual diagram for explaining the process related to the video recovery process according to the embodiment. [Figure 6] It is a conceptual diagram for explaining the process related to the camera recovery process according to the embodiment. [Figure 7] It is a flowchart summarizing the processes related to the abnormality determination process and the abnormality handling process in the remote support system according to the embodiment. [Figure 8] It is a block diagram showing a configuration example of a vehicle according to the embodiment. [Figure 9] It is a block diagram showing a configuration example of a remote support terminal according to the embodiment. [Figure 10] It is a block diagram showing a configuration example of a management device according to the embodiment.

Mode for Carrying Out the Invention

[0014] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0015] 1. Remote Support System Let's consider remote support for mobile objects. Remote support is a concept that includes remote monitoring, remote assistance, and remote driving. Examples of mobile objects include vehicles and robots. Vehicles may be autonomous vehicles or vehicles driven by a driver. Examples of robots include logistics robots. As an example, in the following explanation, we will consider the case where the mobile object being remotely supported is a vehicle. When generalizing, replace "vehicle" with "mobile object" in the following explanation.

[0016] Figure 1 is a schematic diagram showing an example configuration of the remote support system 1 according to this embodiment. The remote support system 1 includes a vehicle 100, a remote support terminal 200, and a management device 300. The vehicle 100 is the target of remote support by the remote supporter X. The remote support terminal 200 is a terminal device operated by the remote supporter X when providing remote support to the vehicle 100. The remote support terminal 200 can also be called a remote cockpit. The management device 300 manages the remote support system 1. Typically, the management device 300 is a management server on the cloud. The management device 300 may be composed of multiple servers that perform distributed processing.

[0017] The vehicle 100, the remote support terminal 200, and the management device 300 can communicate with each other via a communication network. The vehicle 100 and the remote support terminal 200 can communicate with each other via the management device 300. Alternatively, the vehicle 100 and the remote support terminal 200 may communicate directly without going through the management device 300.

[0018] The driver or autonomous driving system of vehicle 100 requests remote support as needed. For example, if vehicle 100 encounters a situation where autonomous driving is difficult, the autonomous driving system requests remote support. Vehicle 100 sends a remote support request to the management device 300. The remote support request may be a Request for Information (RFI) or a Request for Operation (RFO). In response to the remote support request, the management device 300 assigns a remote supporter X from among several candidates to vehicle 100, which is the target of remote support. The management device 300 manages the assignment relationship between vehicle 100 and remote supporter X, and provides information about this assignment relationship to vehicle 100 and the remote support terminal 200. Based on the assignment relationship information, vehicle 100 and the remote support terminal 200 establish communication. After communication is established, vehicle 100 and the remote support terminal 200 may communicate directly without going through the management device 300.

[0019] Vehicle 100 is equipped with various sensors, including an on-board camera C. The on-board camera C photographs the area around vehicle 100 and acquires video (images) showing the surrounding conditions. Vehicle information VCL is information obtained from the various sensors, including video obtained from the on-board camera C. Vehicle 100 transmits vehicle information VCL to the remote support terminal 200.

[0020] The remote support terminal 200 receives vehicle information VCL transmitted from vehicle 100. The remote support terminal 200 presents the vehicle information VCL to the remote supporter X. Specifically, the remote support terminal 200 is equipped with a display device and displays images, etc., on the display device. The remote supporter X looks at the displayed information, recognizes the situation around vehicle 100, and provides remote support for vehicle 100. The remote support information SUP is information related to the remote support provided by the remote supporter X. For example, the remote support information SUP includes instructions or operation amounts entered by the remote supporter X. The remote support terminal 200 transmits the remote support information SUP to vehicle 100 as needed.

[0021] Vehicle 100 receives remote support information SUP transmitted from remote support terminal 200. Vehicle 100 performs vehicle driving control according to the received remote support information SUP.

[0022] In this embodiment, the remote support system 1 further includes one or more infrastructure cameras 400. The infrastructure cameras 400 are installed in the target area where the vehicle 100 is moving. The target area is not particularly limited, but examples include a city, a parking lot, a factory site, etc. The infrastructure cameras 400 photograph the target area and acquire video IMGs showing the situation of the target area. In particular, the infrastructure cameras 400 are installed in a position and orientation that allows them to photograph the vehicle 100 traveling in the target area. Therefore, the video IMGs captured by the infrastructure cameras 400 may include the vehicle 100 that is the target of remote support. The infrastructure cameras 400 transmit the video IMGs to the management device 300.

[0023] The management device 300 communicates with the infrastructure camera 400 and collects and manages the video IMG captured by the infrastructure camera 400. The management device 300 also transmits the video IMG captured by the infrastructure camera 400 to the remote support terminal 200.

[0024] The remote support terminal 200 acquires video images (IMG) captured by the infrastructure camera 400 in addition to the video images captured by the in-vehicle camera C, and presents these images to the remote supporter X. The video images (IMG) captured by the infrastructure camera 400 may include images of the vehicle 100 that is the target of the remote support. By presenting such video images (IMG) to the remote supporter X, it is expected that the accuracy of the remote support and the convenience for the remote supporter X will be further improved.

[0025] As one application example, consider a scenario where an autonomous vehicle is driving autonomously within a factory premises. For example, an autonomous vehicle assembled in an assembly plant automatically drives from the assembly plant to the yard. One or more infrastructure cameras 400 are installed along the road from the assembly plant to the yard. By using these infrastructure cameras 400, the autonomous vehicle can be remotely monitored as it drives autonomously. Furthermore, if the autonomous vehicle encounters a situation where it is difficult to drive autonomously, it can be remotely driven using the infrastructure cameras 400. It is also conceivable to send staff to the site to take over the autonomous vehicle by manual driving when it encounters a situation where it is difficult to drive autonomously, but this is time-consuming and labor-intensive. Remote driving using the infrastructure cameras 400 is more convenient and also saves time and effort.

[0026] 2. Anomaly detection and anomaly response processes As described above, the video image (IMG) captured by the infrastructure camera 400 is presented to the remote supporter X. However, if an anomaly occurs in the video image (IMG) presented to the remote supporter X, the effect of improving the accuracy of remote support cannot be fully obtained. Therefore, the remote support system 1 according to this embodiment is configured to perform the "anomaly detection process" and "anomaly response process" described below.

[0027] 2-1. Anomaly detection process Figure 2 is a conceptual diagram illustrating the anomaly detection process in the remote support system 1. The remote support system 1 includes an anomaly detection unit 10. Here, we consider the case where the first video image IMG-1, captured by the first infrastructure camera 400-1, is presented to the remote supporter X. The first video image IMG-1 may show the target vehicle 100T, which is the subject of remote support. Alternatively, the target vehicle 100T may not currently be shown in the first video image IMG-1, but may be shown in the first video image IMG-1 in the future. The anomaly detection unit 10 determines whether or not an anomaly has occurred in the first video image IMG-1 presented to the remote supporter X. Various causes are possible for an anomaly in the first video image IMG-1 presented to the remote supporter X.

[0028] 2-1-1. First abnormality caused by communication quality The primary cause of the abnormality in the first video image IMG-1 is a deterioration in the communication quality in the communication path of the first video image IMG-1. The first infrastructure camera 400-1 transmits the first video image IMG-1 to the management device 300, and the management device 300 transmits the first video image IMG-1 to the remote support terminal 200. The communication quality (e.g., communication speed, communication delay) in such a communication path may deteriorate. For example, if the communication speed deteriorates, the first video image IMG-1 presented to the remote supporter X may be delayed or become stuck. As another example, if communication is interrupted, the first video image IMG-1 will not reach the remote support terminal 200 and will not be presented to the remote supporter X. Thus, when the communication quality in the communication path of the first video image IMG-1 deteriorates, the quality of the first video image IMG-1 presented to the remote supporter X also deteriorates. An abnormality in the first video image IMG-1 caused by such deterioration in the communication quality in the communication path of the first video image IMG-1 will be referred to as the "first abnormality" below.

[0029] The anomaly detection unit 10 receives the first video IMG-1 and monitors the communication quality in the communication path (e.g., communication speed, communication delay) based on the reception status. The anomaly detection unit 10 then determines whether or not a first anomaly has occurred based on the communication quality in the communication path. For example, if the communication quality in the communication path falls below a first threshold, the anomaly detection unit 10 determines that a first anomaly has occurred.

[0030] 2-1-2. Second abnormality caused by congestion control The second cause of the abnormality in the first video IMG-1 is congestion control on the transmitting side of the first video IMG-1 (first infrastructure camera 400-1, management device 300). When the communication speed decreases, the transmitting side may actively perform congestion control to avoid communication interruption. In congestion control, the transmitting side reduces the resolution or frame rate of the first video IMG-1 being transmitted, thereby degrading the quality of the first video IMG-1. As a result, the quality (resolution, frame rate) of the first video IMG-1 presented to the remote supporter X decreases. Abnormalities in the first video IMG-1 caused by such congestion control on the transmitting side will be referred to as "second abnormality" below.

[0031] The abnormality detection unit 10 receives the first video IMG-1 and determines whether or not congestion control is being performed on the transmitting side based on the first video IMG-1. When congestion control is performed, the received bitrate of the first video IMG-1 decreases significantly. Therefore, the abnormality detection unit 10 can determine whether or not congestion control is being performed based on the fluctuation in the received bitrate of the first video IMG-1. For example, the resolution when congestion control is not performed is the first resolution. On the other hand, the resolution when congestion control is performed is the second resolution, which is lower than the first resolution. The second resolution is predetermined. The first bitrate is the bitrate corresponding to the first resolution. The second bitrate is the bitrate corresponding to the second resolution. The difference between the first bitrate and the second bitrate is significantly larger than the fluctuation range of the bitrate unrelated to congestion control. Therefore, the abnormality detection unit 10 compares the received bitrate of the first video IMG-1 with the average value of the first bitrate and the second bitrate. If the received bitrate falls below the average value of the first bitrate and the second bitrate, the abnormality detection unit 10 determines that congestion control is being performed. If it determines that congestion control is being performed, the abnormality detection unit 10 determines that a second abnormality has occurred.

[0032] 2-1-3. Third abnormality caused by the shooting environment The third cause of the abnormality in the first image IMG-1 is the shooting environment when the first infrastructure camera 400-1 captures the first image IMG-1. For example, in backlit environments, so-called "overexposure" or "underexposure" may occur in the first image IMG-1. As another example, in dark environments, the first image IMG-1 will become dark. As a result, the quality (visibility) of the first image IMG-1 presented to the remote supporter X will decrease. Abnormalities in the first image IMG-1 caused by such shooting environment issues will be referred to as "third abnormality" below.

[0033] The anomaly detection unit 10 receives the first video IMG-1 and automatically determines the environment (scene) in which the first video IMG-1 was taken based on the first video IMG-1. An example of a technique for determining the shooting environment based on the video (image) is the technique described in Non-Patent Document 1 mentioned above. If it is determined that the first video IMG-1 was taken in a backlit or dark environment, the anomaly detection unit 10 determines that a third anomaly has occurred.

[0034] 2-1-4. Fourth abnormality caused by camera malfunction The fourth cause of the anomaly in the first video IMG-1 is an anomaly in the first infrastructure camera 400-1 itself. For example, if the first infrastructure camera 400-1 malfunctions, the first video IMG-1 will not be acquired at all and will not be presented to the remote supporter X. As another example, if the axis of the first infrastructure camera 400-1 becomes misaligned and the field of view changes, the target vehicle 100T may not be properly included in the field of view. Such anomalies in the first video IMG-1 caused by an anomaly in the first infrastructure camera 400-1 itself will be referred to as the "fourth anomaly" below.

[0035] For example, each infrastructure camera 400 has a self-diagnostic function. The first infrastructure camera 400-1 can detect its own abnormalities (failures, misalignment) through its self-diagnostic function. As another example, the first infrastructure camera 400-1 may recognize a predetermined marker placed at a predetermined position within its field of view and detect misalignment based on the recognition result. The first infrastructure camera 400-1 notifies the abnormality determination unit 10 of the abnormality detection. The abnormality determination unit 10 determines whether or not a fourth abnormality has occurred based on whether or not such notification has been received.

[0036] 2-2. Anomaly Handling Process Figure 3 is a conceptual diagram illustrating the anomaly response process in the remote support system 1. The remote support system 1 includes an anomaly response unit 20. When the anomaly detection unit 10 determines that an anomaly has occurred in the first video IMG-1, the anomaly response unit 20 executes an anomaly response process in response. Various examples of anomaly response processes are possible.

[0037] Figure 3 shows examples of anomaly response processing, including "alternative video presentation processing," "video recovery processing," and "camera recovery processing." Alternative video presentation processing presents an alternative video SIMG to the remote supporter X as a substitute for the first video IMG-1. Video recovery processing resolves the anomaly in the first video IMG-1 by restoring its quality. Camera recovery processing resolves the anomaly in the first video IMG-1 by repairing the anomaly in the first infrastructure camera 400-1. The anomaly response unit 20 may also perform anomaly response processing according to the type of anomaly in the first video IMG-1. Details of each anomaly response processing will be described below.

[0038] 2-2-1. Alternative video presentation process Figure 4 is a conceptual diagram illustrating the processes related to the alternative image presentation process. In the alternative image presentation process, an alternative camera 500 capable of capturing the target area is used instead of the first infrastructure camera 400-1.

[0039] For example, alternative camera 500 is another infrastructure camera 400, different from the first infrastructure camera 400-1. Another example is that alternative camera 500 is a camera mounted on a mobile device separate from the target vehicle 100T that is the subject of remote support. The mobile device may be a vehicle or a flying object (e.g., a drone). Typically, the other mobile device is located around the target vehicle 100T, and its camera photographs the target vehicle 100T and its surroundings. For example, the other mobile device may be a following vehicle traveling behind the target vehicle 100T. Another example is that the other mobile device may be a flying object positioned above the target vehicle 100T.

[0040] The anomaly response unit 20 acquires an alternative video SIMG captured by the alternative camera 500. The alternative video SIMG may show the target vehicle 100T, which is the subject of remote support. Alternatively, the target vehicle 100T may not currently be shown in the alternative video SIMG, but may be shown in the alternative video SIMG in the future. The anomaly response unit 20 then presents the alternative video SIMG to the remote supporter X in place of the first video IMG-1. The remote supporter X can then refer to the alternative video SIMG, which does not show an anomaly, in place of the first video IMG-1 where the anomaly occurred, and continue providing remote support. This ensures the accuracy of the remote support.

[0041] There may be multiple alternative camera candidates capable of capturing the target area. In the example shown in Figure 4, there are three alternative camera candidates 500-1, 500-2, and 500-3. The anomaly response unit 20 acquires multiple alternative video candidate SIMG-i captured by each of the multiple alternative camera candidates 500-i (i=1~3). The anomaly response unit 20 then selects (adopts) at least one of the multiple alternative video candidate SIMG-i as the alternative video SIMG.

[0042] For example, the anomaly response unit 20 considers the resolution or frame rate of each of the multiple alternative video candidate SIMG-i. Then, the anomaly response unit 20 selects at least one of the multiple alternative video candidate SIMG-i with the highest resolution or frame rate as the alternative video SIMG.

[0043] As another example, the anomaly response unit 20 may select as the alternative video SIMG a candidate SIMG-i that currently displays the target vehicle 100T, which is the subject of remote support. More specifically, the anomaly response unit 20 acquires camera information CAM for each of the multiple alternative camera candidates 500-i. The camera information CAM indicates the position, orientation, field of view, etc., of the alternative camera candidate 500-i. For example, if the alternative camera candidate 500-i is an infrastructure camera 400, the camera information CAM indicates the installation position, installation orientation, field of view, etc., of the infrastructure camera 400. Typically, the installation position of the infrastructure camera 400 is expressed in latitude and longitude in an absolute coordinate system. The camera information CAM for the infrastructure camera 400 is provided by the infrastructure camera 400 or the management device 300. As another example, if the alternative camera candidate 500 is a camera mounted on a mobile device, the camera information CAM indicates the installation position, installation orientation, field of view, etc., of the camera on the mobile device, in addition to the position and orientation of the mobile device. The position of the moving object is expressed in latitude and longitude in an absolute coordinate system. Camera information CAM regarding a camera mounted on the moving object is provided by the moving object. Furthermore, the anomaly response unit 20 acquires the position information of the target vehicle 100T from the target vehicle 100T. The position information of the target vehicle 100T is included in the vehicle information VCL transmitted from the target vehicle 100T. Based on the camera information CAM and the position information of the target vehicle 100T, the anomaly response unit 20 can select an alternative camera candidate 500-i that has the target vehicle 100T within its field of view. Then, the anomaly response unit 20 selects the alternative video candidate SIMG-i captured by the selected alternative camera candidate 500-i as the alternative video SIMG.

[0044] As described above, the anomaly response unit 20 may perform anomaly response processing according to the type of anomaly in the first video IMG-1. For example, in the case of a first anomaly caused by a deterioration in communication quality in the communication path of the first video IMG-1, the anomaly response unit 20 performs alternative video presentation processing. As another example, in the case of a second anomaly caused by congestion control on the transmitting side, the anomaly response unit 20 may perform alternative video presentation processing. As yet another example, if the first infrastructure camera 400-1 fails, the anomaly response unit 20 may perform alternative video presentation processing.

[0045] As described above, the alternative video presentation process presents the remote supporter X with an alternative video SIMG that replaces the first video IMG-1 where the anomaly occurred. The remote supporter X can then continue remote support by referring to the alternative video SIMG, which does not show any anomalies, instead of the first video IMG-1 where the anomaly occurred. This ensures the accuracy of the remote support.

[0046] 2-2-2. Video Recovery Process Figure 5 is a conceptual diagram illustrating the processes related to image recovery processing. As described above, the quality (visibility) of the first image IMG-1 presented to the remote supporter X is reduced due to a third anomaly caused by the shooting environment of the first image IMG-1. For example, in backlit environments, so-called "overexposure" or "underexposure" may occur in the first image IMG-1. As another example, in dark environments, the first image IMG-1 becomes dark. Image recovery processing is preferably applied in cases of such third anomalies, and resolves the anomaly of the first image IMG-1 by restoring its quality (visibility).

[0047] For example, the anomaly response unit 20 acquires the first video image IMG-1 and automatically determines the environment (scene) in which the first video image IMG-1 was captured. An example of a technique for determining the shooting environment based on the video (image) is the technique described in Non-Patent Document 1 mentioned above. If it is determined that the first video image IMG-1 was captured in a backlit or dark environment, the anomaly response unit 20 performs a software-based brightness correction process to improve the visibility of the first video image IMG-1. The brightness correction process corrects the brightness of the first video image IMG-1, which was captured in a scene such as at night or in backlight, to an appropriate level, thereby improving visibility. This brightness correction process can be implemented, for example, by the technique described in Non-Patent Document 2 mentioned above.

[0048] As another example, the anomaly response unit 20 may adjust the parameters (e.g., aperture) of the first infrastructure camera 400-1 so that the visibility of the first video IMG-1 is improved. In this case, the anomaly response unit 20 communicates with the first infrastructure camera 400-1 and instructs the first infrastructure camera 400-1 to adjust the parameters (e.g., aperture).

[0049] As described above, the video recovery process corrects the first video IMG-1 using software, or adjusts the parameters of the first infrastructure camera 400-1, so that the visibility of the first video IMG-1 is improved. The remote supporter X can continue remote support by referring to the first video IMG-1 with improved visibility. This ensures the accuracy of the remote support.

[0050] 2-2-3. Camera Recovery Process Figure 6 is a conceptual diagram illustrating the process related to camera recovery processing. As mentioned above, if the first infrastructure camera 400-1 experiences an axis misalignment and its field of view changes, the target vehicle 100T may not be properly included in the field of view. Camera recovery processing is preferably applied in such cases of axis misalignment, and the abnormality in the first image IMG-1 is resolved by correcting the axis misalignment of the first infrastructure camera 400-1.

[0051] More specifically, the anomaly response unit 20 instructs the first infrastructure camera 400-1 to perform calibration. For example, a predetermined marker M is placed at a predetermined position within the field of view of the first infrastructure camera 400-1. The first infrastructure camera 400-1 recognizes the predetermined marker M and performs calibration based on the recognition result so that the axial misalignment is eliminated.

[0052] As described above, the camera recovery process resolves the anomaly in the first video IMG-1 by correcting the axial misalignment of the first infrastructure camera 400-1. The remote supporter X can then continue remote support by referring to the first video IMG-1, which has had the anomaly resolved. This ensures the accuracy of the remote support.

[0053] 2-3. Anomaly detection unit and anomaly response unit The location of the abnormality detection unit 10 and the abnormality response unit 20 is not limited as long as necessary information such as the first video IMG-1 can be obtained. As described above, the vehicle 100, the remote support terminal 200, and the management device 300 can communicate with each other via a communication network. In other words, various types of information can be shared between the vehicle 100, the remote support terminal 200, and the management device 300. Therefore, the abnormality detection unit 10 and the abnormality response unit 20 may be included in any of the vehicle 100, the remote support terminal 200, and the management device 300. The abnormality detection unit 10 and the abnormality response unit 20 may be distributed among two or more of the vehicle 100, the remote support terminal 200, and the management device 300. For example, the abnormality detection unit 10 may be included in the management device 300, and the abnormality response unit 20 may be included in the remote support terminal 200.

[0054] In general terms, the abnormality detection unit 10 and the abnormality response unit 20 are implemented by one or more processors and one or more storage devices. The one or more processors perform various information processing. The one or more storage devices store various information necessary for processing by the one or more processors.

[0055] Figure 7 is a flowchart showing the processes related to anomaly detection processing and anomaly response processing in the remote support system 1. In step S10, one or more processors acquire the first video image IMG-1 captured by the first infrastructure camera 400-1. The first video image IMG-1 is presented to the remote supporter X. In step S20, one or more processors perform anomaly detection processing to determine whether or not an anomaly has occurred in the first video image IMG-1 (see Section 2-1 above). If no anomaly has occurred in the first video image IMG-1 (Step S20; No), the process proceeds to step S30. In step S30, one or more processors present the first video image IMG-1 to the remote supporter X. On the other hand, if an anomaly has occurred in the first video image IMG-1 (Step S20; Yes), the process proceeds to step S40. In step S40, one or more processors perform anomaly response processing (see Section 2-2 above). One or more processors may perform anomaly response processing according to the type of anomaly in the first video image IMG-1.

[0056] Furthermore, it is not necessarily required that the target vehicle 100T be visible in the first video IMG-1 at the time the anomaly detection process and anomaly response process are executed. This is because if there is a possibility that the target vehicle 100T may be visible in the first video IMG-1 or the alternative video SIMG in the future, it is meaningful to perform the anomaly response process in advance.

[0057] 2-4. Effects As described above, according to this embodiment, the first video image IMG-1 captured by the first infrastructure camera 400-1 is presented to the remote supporter X. If an abnormality occurs in the first video image IMG-1, an abnormality response process is executed. The abnormality response process either resolves the abnormality in the first video image IMG-1 or presents the remote supporter X with an alternative video image SIMG to replace the first video image IMG-1. The remote supporter X can continue remote support by referring to the first video image IMG-1 or the alternative video image SIMG after the abnormality has been resolved. This ensures the accuracy of remote support. It also ensures convenience for the remote supporter X.

[0058] 3. Examples of vehicles 3-1. Example Configuration Figure 8 is a block diagram showing an example configuration of vehicle 100. Vehicle 100 is equipped with a communication device 110, a sensor group 120, a running device 130, and a control device 150.

[0059] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with the remote support terminal 200 and the management device 300.

[0060] The sensor group 120 includes recognition sensors, vehicle status sensors, position sensors, etc. The recognition sensors recognize (detect) the surrounding conditions of the vehicle 100. Examples of recognition sensors include an on-board camera C, LIDAR (Laser Imaging Detection and Ranging), radar, etc. The vehicle status sensors detect the state of the vehicle 100. The vehicle status sensors include a speed sensor, acceleration sensor, yaw rate sensor, steering angle sensor, etc. The position sensors detect the position and orientation of the vehicle 100. For example, the position sensors include a GNSS (Global Navigation Satellite System).

[0061] The running gear 130 includes a steering gear, a drive gear, and a braking gear. The steering gear steers the wheels. For example, the steering gear includes an electric power steering (EPS) system. The drive gear is a power source that generates driving force. Examples of drive gears include an engine, an electric motor, an in-wheel motor, etc. The braking gear generates braking force.

[0062] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processor 160) and one or more storage devices 170 (hereinafter simply referred to as storage devices 170). The processor 160 performs various processes. For example, the processor 160 includes a CPU (Central Processing Unit). The storage devices 170 store various information necessary for processing by the processor 160. Examples of storage devices 170 include volatile memory, non-volatile memory, HDD (Hard Disk Drive), SSD (Solid State Drive), etc. The control device 150 may also include one or more ECUs (Electronic Control Units).

[0063] The vehicle control program PROG1 is a computer program executed by the processor 160. The processor 160 executes the vehicle control program PROG1, thereby realizing the functions of the control device 150. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded on a computer-readable recording medium.

[0064] 3-2. Driving Environment Information The control device 150 uses the sensor group 120 to acquire driving environment information ENV, which indicates the driving environment of the vehicle 100. The driving environment information ENV is stored in the storage device 170.

[0065] The driving environment information (ENV) includes surrounding situation information that shows the recognition results from the recognition sensors. For example, the surrounding situation information includes video footage captured by the on-board camera C. The surrounding situation information may also include object information regarding objects around the vehicle 100. Examples of objects around the vehicle 100 include pedestrians, other vehicles (preceding vehicles, parked vehicles, etc.), white lines, traffic lights, signs, roadside structures, etc. The object information indicates the relative position and relative speed of the object with respect to the vehicle 100.

[0066] Furthermore, the driving environment information (ENV) includes vehicle status information that indicates the vehicle status detected by the vehicle status sensor.

[0067] Furthermore, the driving environment information (ENV) includes position information indicating the position and orientation of the vehicle 100. The position information is obtained by a position sensor. High-precision position information may also be obtained by localization processing using map information and surrounding environment information (object information).

[0068] 3-3. Vehicle Driving Control The control device 150 performs vehicle driving control to control the movement of the vehicle 100. Vehicle driving control includes steering control, drive control, and braking control. The control device 150 performs vehicle driving control by controlling the driving device 130 (steering device, drive device, and braking device).

[0069] The control device 150 may perform automatic driving control based on the driving environment information ENV. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the driving environment information ENV. Furthermore, the control device 150 generates a target trajectory necessary for the vehicle 100 to drive according to the driving plan, based on the driving environment information ENV. The target trajectory includes a target position and a target speed. The control device 150 then performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0070] 3-4. Processing related to remote support The following describes the case where remote support is provided to vehicle 100. The control device 150 communicates with the remote support terminal 200 via the communication device 110.

[0071] The control device 150 transmits vehicle information VCL to the remote support terminal 200. The vehicle information VCL is information necessary for remote support by the remote supporter X and includes at least a part of the driving environment information ENV described above. For example, the vehicle information VCL includes surrounding situation information (especially video). The vehicle information VCL may also include vehicle status information and vehicle location information.

[0072] Furthermore, the control device 150 receives remote support information SUP from the remote support terminal 200. Remote support information SUP is information related to remote support provided by remote supporter X. For example, remote support information SUP includes the amount of operation performed by remote supporter X. The control device 150 performs vehicle driving control according to the received remote support information SUP.

[0073] 4. Examples of remote support terminals Figure 9 is a block diagram showing an example configuration of the remote support terminal 200. The remote support terminal 200 includes a communication device 210, an output device 220, an input device 230, and a control device 250.

[0074] The communication device 210 communicates with the vehicle 100 and the management device 300.

[0075] The output device 220 outputs various types of information. For example, the output device 220 includes a display device. The display device presents various types of information to the remote supporter X by displaying the information. As another example, the output device 220 may include a speaker.

[0076] The input device 230 receives input from the remote supporter X. Examples of the input device 230 include a touch panel, buttons, remote control components, etc. The remote control components are components operated by the remote supporter X (remote supporter) when remotely driving the vehicle 100. For example, the remote control components include a steering wheel, accelerator pedal, brake pedal, turn signals, etc. The remote control components may also be a touch panel.

[0077] The control device 250 controls the remote support terminal 200. The control device 250 includes one or more processors 260 (hereinafter simply referred to as processor 260) and one or more storage devices 270 (hereinafter simply referred to as storage devices 270). The processors 260 perform various processes. For example, the processor 260 includes a CPU. The storage devices 270 store various information necessary for processing by the processors 260. Examples of storage devices 270 include volatile memory, non-volatile memory, HDD, SSD, etc.

[0078] The remote support program PROG2 is a computer program executed by the processor 260. The processor 260 executes the remote support program PROG2, thereby realizing the functions of the control unit 250. The remote support program PROG2 is stored in the storage device 270. Alternatively, the remote support program PROG2 may be recorded on a computer-readable recording medium. The remote support program PROG2 may also be provided via a network.

[0079] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives vehicle information VCL transmitted from the vehicle 100. The control device 250 presents the vehicle information VCL, including video, to the remote supporter X by displaying it on a display device. The remote supporter X can recognize the status of the vehicle 100 and the surrounding conditions based on the vehicle information VCL displayed on the display device.

[0080] The remote supporter X operates the input device 230 as needed to provide remote support to the vehicle 100. For example, the remote supporter X issues various instructions (e.g., a start instruction) via the input device 230. When remotely driving the vehicle 100, the remote supporter X operates the remote driving components. The amount of operation of the remote driving components is detected by a sensor installed on the remote driving components. The control device 250 generates remote support information SUP, which includes the instructions or operation amounts input by the remote supporter X. The control device 250 then transmits the remote support information SUP to the vehicle 100 via the communication device 210.

[0081] Furthermore, the control device 250 acquires video IMGs captured by the infrastructure camera 400 via the management device 300. The acquired video IMGs are stored in the storage device 270. The control device 250 presents the video IMGs to the remote supporter X via the output device 220. More specifically, the control device 250 displays the video IMGs on a display device.

[0082] Furthermore, the control device 250 may also include the functions of the abnormality determination unit 10 and the abnormality response unit 20 described above. The control device 250 acquires the first video IMG-1 captured by the first infrastructure camera 400-1. The control device 250 also acquires the alternative video SIMG captured by the alternative camera 500. The alternative video SIMG is stored in the storage device 270. In addition, the control device 250 may acquire other information necessary for processing, such as camera information CAM. The control device 250 performs abnormality determination processing and abnormality response processing based on the acquired information (see Section 2 above). The control device 250 may display the alternative video SIMG on a display device.

[0083] 5. Examples of control devices Figure 10 is a block diagram showing an example configuration of the management device 300. The management device 300 includes a communication device 310 and a control device 350.

[0084] The communication device 310 communicates with the vehicle 100, the remote support terminal 200, and the infrastructure camera 400.

[0085] The control device 350 controls the management device 300. The control device 350 includes one or more processors 360 (hereinafter simply referred to as processor 360) and one or more storage devices 370 (hereinafter simply referred to as storage devices 370). The processors 360 perform various processes. For example, the processor 360 includes a CPU. The storage devices 370 store various information necessary for processing by the processors 360. Examples of storage devices 370 include volatile memory, non-volatile memory, HDD, SSD, etc.

[0086] The management program PROG3 is a computer program executed by the processor 360. The processor 360 executes the management program PROG3, thereby realizing the functions of the control unit 350. The management program PROG3 is stored in the storage device 370. Alternatively, the management program PROG3 may be recorded on a computer-readable recording medium. The management program PROG3 may also be provided via a network.

[0087] The control device 350 communicates with the vehicle 100 and the remote support terminal 200 via the communication device 310. The control device 350 receives vehicle information VCL transmitted from the vehicle 100. The control device 350 then transmits the received vehicle information VCL to the remote support terminal 200. The control device 350 also receives remote support information SUP transmitted from the remote support terminal 200. The control device 350 then transmits the received remote support information SUP to the vehicle 100.

[0088] Furthermore, the control device 350 communicates with the infrastructure camera 400 via the communication device 310 and acquires the video image (IMG) captured by the infrastructure camera 400. The control device 350 provides the video image (IMG) to the remote support terminal 200. The control device 350 may also acquire camera information (CAM) and provide the camera information (CAM) to the remote support terminal 200.

[0089] Furthermore, the control device 350 may also include the functions of the abnormality determination unit 10 and the abnormality response unit 20 described above. The control device 350 acquires the first video IMG-1 captured by the first infrastructure camera 400-1. The control device 350 also acquires the alternate video SIMG captured by the alternate camera 500. The alternate video SIMG is stored in the storage device 370. The control device 350 performs abnormality determination processing and abnormality response processing based on the acquired information (see Section 2 above). The control device 350 may also transmit the alternate video SIMG to the remote support terminal 200. [Explanation of symbols]

[0090] 1…Remote support system, 10…Anomaly detection unit, 20…Anomaly response unit, 100…Vehicle, 200…Remote support terminal, 300…Management device, 400…Infrastructure camera, IMG…Video, SIMG…Alternate video, VCL…Vehicle information

Claims

1. A remote support system for providing remote support to mobile objects, Equipped with one or more processors, The one or more processors described above are: A first video is captured by a first infrastructure camera installed in the target area where the moving object is moving, and the first video in which the moving object is currently or is expected to be captured in the future is acquired via communication. The first video is presented to the remote supporter who provides remote support for the mobile body. If an anomaly occurs in the first video, the system performs an alternative video presentation process to present the remote supporter with an alternative video in which the moving object is currently or is expected to be visible in the future, as a replacement for the first video. It is configured in such a way, The aforementioned alternative image presentation process is: To acquire the location information of the moving object and the camera location information, Select an alternative camera capable of photographing the target area based at least on the position information of the moving object and the camera position information, and acquire the alternative image captured by the alternative camera. To present the alternative video to the remote supporter instead of the first video. Includes, The camera position information includes the position information of at least one of the first infrastructure camera and the alternative camera. Remote support system.

2. A remote support system according to claim 1, If there are multiple alternative camera candidates capable of capturing the target area, the alternative image presentation process is performed as follows: To acquire the position information of the moving object and the camera position information, Selecting a plurality of alternative camera candidates based at least on the position information of the moving object and the camera position information, and obtaining a plurality of alternative video candidates captured by each of the plurality of alternative camera candidates, Selecting at least one of the aforementioned multiple alternative video candidates as the alternative video. Includes, The camera position information further includes the position information of the plurality of alternative camera candidates. Remote support system.

3. A remote support system according to claim 1, The alternative camera is an infrastructure camera different from the first infrastructure camera, or a camera mounted on a mobile body different from the mobile body. Remote support system.

4. A remote support method, performed by a computer, for providing remote support to a mobile object, The process involves acquiring, via communication, a first video image captured by a first infrastructure camera installed in the target area where the moving object is moving, which shows the moving object currently or is expected to show in the future. The first video is presented to the remote supporter who provides remote support for the mobile body, If an abnormality occurs in the first video, an alternative video presentation process is performed to present to the remote supporter an alternative video in which the moving object is currently or is expected to be in the future, which will replace the first video. Includes, The aforementioned alternative image presentation process is: To acquire the location information of the moving object and the camera location information, Select an alternative camera capable of photographing the target area based at least on the position information of the moving object and the camera position information, and acquire the alternative image captured by the alternative camera. To present the alternative video to the remote supporter instead of the first video. Includes, The camera position information includes the position information of at least one of the first infrastructure camera and the alternative camera. Remote support methods.

5. A remote support system for providing remote support to mobile objects, Equipped with one or more processors, The one or more processors described above are: The first video footage captured by the first infrastructure camera installed in the target area where the mobile object is moving is acquired via communication. The first video is presented to the remote supporter who provides remote support for the mobile body. If an anomaly occurs in the first video, an anomaly response process is executed to resolve the anomaly in the first video, or to present an alternative video to the remote supporter as a replacement for the first video. It is configured in such a way, The abnormality in the first video is, The first anomaly caused by a deterioration in communication quality in the communication path of the first video, The second abnormality caused by congestion control on the transmitting side of the first video, The third anomaly caused by the shooting environment of the first video, The fourth abnormality caused by the abnormality of the first infrastructure camera and It is one of the following: The one or more processors are configured to perform the abnormality response processing according to the type of abnormality in the first video, The anomaly response process in the case of the first or second anomaly is an alternative image presentation process. The aforementioned alternative image presentation process is: The process involves acquiring the alternative video footage captured by an alternative camera capable of capturing the aforementioned target area, To present the alternative video to the remote supporter instead of the first video. Includes, If there are multiple alternative camera candidates capable of capturing the target area, the alternative image presentation process is performed as follows: To acquire multiple alternative video candidates captured by each of the aforementioned multiple alternative camera candidates, Selecting at least one of the aforementioned multiple alternative video candidates as the alternative video. Includes, Of the multiple alternative video candidates, at least one with the highest resolution or frame rate is selected as the alternative video. Remote support system.

6. A remote support system for providing remote support to mobile objects, Equipped with one or more processors, The one or more processors described above are: The first video footage captured by the first infrastructure camera installed in the target area where the mobile object is moving is acquired via communication. The first video is presented to the remote supporter who provides remote support for the mobile body. If an anomaly occurs in the first video, an anomaly response process is executed to resolve the anomaly in the first video, or to present an alternative video to the remote supporter as a replacement for the first video. It is configured in such a way, The abnormality in the first video is, The first anomaly caused by a deterioration in communication quality in the communication path of the first video, The second abnormality caused by congestion control on the transmitting side of the first video, The third anomaly caused by the shooting environment of the first video, The fourth abnormality caused by the abnormality of the first infrastructure camera and It is one of the following: The one or more processors are configured to perform the abnormality response processing according to the type of abnormality in the first video, The fourth abnormality includes a failure of the first infrastructure camera, The error response process in the event of the failure of the first infrastructure camera is an alternative image presentation process. The aforementioned alternative image presentation process is: The process involves acquiring the alternative video footage captured by an alternative camera capable of capturing the aforementioned target area, To present the alternative video to the remote supporter instead of the first video. Includes, If there are multiple alternative camera candidates capable of capturing the target area, the alternative image presentation process is performed as follows: To acquire multiple alternative video candidates captured by each of the aforementioned multiple alternative camera candidates, Selecting at least one of the aforementioned multiple alternative video candidates as the alternative video. Includes, Of the multiple alternative video candidates, at least one with the highest resolution or frame rate is selected as the alternative video. Remote support system.

7. A remote support system for providing remote support to mobile objects, Equipped with one or more processors, The one or more processors described above are: The first video footage captured by the first infrastructure camera installed in the target area where the mobile object is moving is acquired via communication. The first video is presented to the remote supporter who provides remote support for the mobile body. If an anomaly occurs in the first video, an anomaly response process is executed to resolve the anomaly in the first video, or to present an alternative video to the remote supporter as a replacement for the first video. It is configured in such a way, The abnormality in the first video is, The first anomaly caused by a deterioration in communication quality in the communication path of the first video, The second abnormality caused by congestion control on the transmitting side of the first video, The third anomaly caused by the shooting environment of the first video, The fourth abnormality caused by the abnormality of the first infrastructure camera and It is one of the following: The one or more processors are configured to perform the abnormality response processing according to the type of abnormality in the first video, The fourth abnormality includes the axial misalignment of the first infrastructure camera, The abnormality response process in the case of the axial misalignment of the first infrastructure camera is a camera recovery process. The camera recovery process includes calibrating the first infrastructure camera using a marker placed within the field of view of the first infrastructure camera. Remote support system.

Images