Remote support system, remote support method, and remote support program

Abstract

[Project] Provides technology that can effectively utilize surveillance cameras for remote support of vehicles. [Solution] The remote support system of the present disclosure acquires images related to the operation of a vehicle (20) from a plurality of surveillance cameras (10) installed on or near the vehicle's driving path. Next, the remote support system selects an image to be displayed on a display device (53) for a remote operator from among the images acquired by the plurality of surveillance cameras (10), based on information regarding the current and future relative relationship between each of the plurality of surveillance cameras (10) and the vehicle (20). Then, the remote support system displays the selected image on the display device (53).

Description

Technology Field

[0001] The present disclosure relates to a remote support system, a remote support method, and a remote support program for providing remote support to a vehicle. Background Technology

[0002] The prior art disclosed in U.S. Patent Application Publication No. 2021 / 0089018 receives safety conditions necessary for remotely controlling a vehicle and generates a remote control signal when the safety conditions are satisfied. Prior art literature

[0003] U.S. Patent Application Publication No. 2021 / 0089018 The problem to be solved

[0004] The above-described prior art is a type of remote support by a remote operator. In remote support, it is considered that images acquired by a surveillance camera are used in conjunction with, or instead of, images acquired by, a vehicle-mounted camera. However, when multiple surveillance cameras exist, there is room for consideration regarding which surveillance camera's images are used. This is because the performance of the remote support changes depending on the images used. The above-described prior art does not provide any solution regarding the selection of surveillance cameras in remote support.

[0005] The present disclosure has been made in consideration of the above-mentioned problem. The present disclosure aims to provide a technology that can effectively utilize a plurality of surveillance cameras for remote support of a vehicle. means of solving the problem

[0006] To achieve the above objective, the present disclosure provides a remote support system for providing remote support to a vehicle. The remote support system of the present disclosure comprises a display device for displaying images to a remote operator, at least one processor, and at least one memory storing a plurality of instructions executable by the at least one processor. The plurality of instructions are configured to execute the following processing on the at least one processor. The first processing is to acquire images related to the operation of a vehicle from a plurality of surveillance cameras installed on or near the vehicle's driving path. The second processing is to select an image to be displayed on the display device from among the images acquired by the plurality of surveillance cameras, based on information regarding the current and future relative relationship between each of the plurality of surveillance cameras and the vehicle. The third processing is to display the selected image on the display device.

[0007] In addition, to achieve the above objective, the present disclosure provides a remote support method for providing remote support to a vehicle. The remote support method of the present disclosure comprises the following steps. The first step is to acquire images related to the operation of a vehicle from a plurality of surveillance cameras installed on or near the vehicle's driving path. The second step is to select an image to be displayed on a display device for a remote operator from among the images acquired by the plurality of surveillance cameras, based on information regarding the current and future relative relationship between each of the plurality of surveillance cameras and the vehicle. The third step is to display the selected image on the display device.

[0008] Additionally, to achieve the above objective, the present disclosure provides a remote support program for providing remote support to a vehicle. The remote support program of the present disclosure is configured to execute the above remote support method on a computer. The remote support program of the present disclosure may be recorded on a non-transient computer-readable recording medium. Effects of the invention

[0009] According to the technology of the present disclosure, regarding a plurality of surveillance cameras, by considering not only the current relative relationship with the vehicle but also the future relative relationship with the vehicle, it becomes possible to determine which surveillance camera provides a more useful image for performing remote support. Accordingly, among the images acquired by the plurality of surveillance cameras, an image useful for remote support can be selected and displayed to a remote operator. Brief explanation of the drawing

[0010] FIG. 1 is a block diagram showing the configuration of a remote driving system related to an embodiment of the present disclosure. Figure 2 is a drawing showing an example of a surveillance camera installation. Figure 3 is a diagram showing an example of method 1 for selecting surveillance camera images displayed on a display device. Figure 4 is a diagram showing an example of method 2 for selecting surveillance camera images displayed on a display device. Figure 5 is a diagram showing an example of method 3 for selecting surveillance camera images displayed on a display device. Figure 6 is a diagram showing an example of method 4 for selecting surveillance camera images displayed on a display device. Figure 7 is a diagram showing an example of a method for switching surveillance camera images displayed on a display device. Figure 8 is a diagram showing an example of switching a surveillance camera image displayed on a display device. FIG. 9 is a diagram showing an example of processing method 1 of a surveillance camera image displayed on a display device. FIG. 10 is a diagram showing an example of method 2 for processing surveillance camera images displayed on a display device. Figure 11 is a diagram showing a modified example of a method for displaying surveillance camera images. Specific details for implementing the invention

[0011] 1. Configuration of the remote operation system

[0012] The remote support system of the present disclosure is applicable to any of the following: a remote driving system for remotely driving a vehicle, a remote assistance system for remotely supporting a vehicle, and a remote monitoring system for remotely monitoring a vehicle. In the present embodiment, the remote support system of the present disclosure is applied to a remote driving system for remotely driving a vehicle. FIG. 1 is a block diagram showing the configuration of a remote driving system (100) related to the present embodiment. The remote driving system (100) related to the present embodiment comprises a vehicle (20) equipped with a vehicle-mounted camera (24), a monitoring camera (10), a management server (30), and a remote cockpit (50).

[0013] The vehicle (20) is a vehicle capable of remote driving. The vehicle (20) is equipped with at least one vehicle-mounted camera (24). The vehicle-mounted camera (24) includes at least a camera that captures the front of the vehicle (20). Preferably, the vehicle-mounted camera (24) includes a camera that captures the front obliquely to the right of the vehicle (20) and a camera that captures the front obliquely to the left of the vehicle (20). Additionally, the vehicle-mounted camera (24) may include a camera that captures the rear of the vehicle (20), a camera that captures the rear obliquely to the right of the vehicle (20), and a camera that captures the rear obliquely to the left of the vehicle (20).

[0014] The vehicle (20) is equipped with a vehicle-mounted computer (21). The vehicle-mounted computer (21) includes a remote driving kit (22) for remote driving. The remote driving kit (22) is connected to a communication network using wireless communication. The remote driving kit (22) transmits vehicle-mounted camera images (40) captured by a vehicle-mounted camera (24) to a management server (30) via the communication network. Additionally, if the vehicle (20) is a vehicle equipped with an automatic driving function, the vehicle-mounted computer (21) includes an automatic driving kit (23) for automatic driving. The automatic driving kit (23) uses images captured by the vehicle-mounted camera (24), particularly images captured by the front camera, for the recognition of surrounding objects for automatic driving.

[0015] The remote driving kit (22) obtains information regarding the status of the vehicle (20), such as vehicle speed, driving distance, and remaining fuel, from an internal sensor (26) mounted on the vehicle (20). The remote driving kit (22) transmits the information obtained from the internal sensor (26) and the information obtained by calculation by the vehicle-mounted computer (21) to the management server (30) as vehicle information (42). The information obtained by calculation includes information regarding the planned driving route or direction of travel of the vehicle (20).

[0016] The remote driving kit (22) receives a control amount command (48) for remote driving from the management server (30). The remote driving kit (22) controls the actuator (28) of the vehicle (20) according to the control amount command (48). The actuator (28) includes a driving actuator that drives the vehicle (20), a braking actuator that brakes the vehicle (20), and a steering actuator that steers the vehicle (20). Additionally, when automatic driving is performed, the control amount of the actuator (28) is calculated by the automatic driving kit (23) to drive the vehicle (20) along a target track.

[0017] The surveillance camera (10) is a surveillance camera as part of social infrastructure, particularly transportation infrastructure. The surveillance camera (10) is installed in large numbers along the road. Additionally, the surveillance camera (10) is a network camera connected to a communication network. The surveillance camera (10) transmits captured images to a management server (30) via the communication network.

[0018] Here, an example of the installation of a surveillance camera (10) is explained using FIG. 2. As shown in FIG. 2, the form of installation of the surveillance camera (10) is diverse. For example, the surveillance camera (10) includes a camera (10-1) installed on or near the lane to photograph the front of the lane, and a camera (10-2) installed on or near the lane to photograph the rear of the lane. In addition, the surveillance camera (10) includes a camera (10-3) installed on the side of the lane to photograph the lane from the side. However, the surveillance camera (10) related to the vehicle (20) is not limited to cameras installed on or near the lane (2) where the vehicle (20) is traveling. Cameras installed on or near the opposing lane (4) may also be included in the surveillance camera (10) related to the vehicle (20). Additionally, the surveillance camera (10) includes a camera (10-4) installed at the intersection (6) and taking diagonal shots of the inside of the intersection (6).

[0019] Returning to FIG. 1, the description of the configuration of the remote driving system (100) continues. The remote cockpit (50) is a cockpit for a remote operator to remotely operate the vehicle (20). The remote cockpit (50) reproduces a driving environment for a driver to manually drive the vehicle (20). The remote cockpit (50) is equipped with a display device (53). The display device (53) is a device for reproducing the scenery outside the vehicle as seen from the driver's seat of the vehicle (20). The display device (53) has multiple screens. The screens include a main screen (54C, 54L, 54R) that displays images from the vehicle-mounted camera (24) and sub-screens (56C, 56L, 56R) that display images from the surveillance camera (10). The main screen consists of a central screen (54C) that displays images of the front, a left screen (54L) that displays images of the front obliquely to the left, and a right screen (54R) that displays images of the front obliquely to the right. However, the rear view is displayed in a picture-in-picture manner on the central screen (54C). In the same way, the rear view is displayed diagonally to the left on the left screen (54L), and the rear view is displayed diagonally to the right on the right screen (54R). The sub-screens consist of three screens: the central screen (56C), the left screen (56L), and the right screen (56R). However, there may be two sub-screens or one.

[0020] The remote cockpit (50) is equipped with a controller (52). The display content of the display device (53) is controlled by the controller (52). The controller (52) is connected to the management server (30). The controller (52) displays vehicle-mounted camera images (41) transmitted from the management server (30) on the main screen (54C, 54L, 54R) and displays surveillance camera images (45) transmitted from the management server (30) on the sub-screen (56C, 56L, 56R). Additionally, the controller (52) has the function of receiving vehicle information (42) from the management server (30) and presenting the received vehicle information (42) to the remote operator. Furthermore, the controller (52) also has the function of transmitting control amount commands (48) input by the remote operator to the management server (30).

[0021] The management server (30) is equipped with a processor (32) and a program memory (34) coupled to the processor (32) so as to be communicable. The number of processors (32) and program memory (34) constituting the management server (30) may each be multiple. The program memory (34) is a computer-readable recording medium. The program memory (34) stores at least one program executable by the processor (32). The program consists of multiple instructions (36). The instructions (36) stored in the program memory (34) include instructions for the processor (32) to implement the remote support method of the present disclosure. The instructions correspond to the remote support program of the present disclosure.

[0022] The management server (30) is equipped with an image memory (38) that is communicably coupled to the processor (32). The vehicle-mounted camera image (40) received from the vehicle-mounted computer (21) and the surveillance camera image (44) received from the surveillance camera (10) are stored in the image memory (38). The processor (32) reads out the vehicle-mounted camera image (40) stored in the image memory (38) and generates a vehicle-mounted camera image (41) to be displayed on the main screen (54C, 54L, 54R). The vehicle-mounted camera image (41) may be the vehicle-mounted camera image (40) itself, or it may be a vehicle-mounted camera image (40) that has been processed as needed. Additionally, the processor (32) reads out an image selected according to a predetermined rule from among the multiple surveillance camera images (44) stored in the image memory (38) and generates a surveillance camera image (45) to be displayed on the sub-screen (56C, 56L, 56R). The surveillance camera image (45) may be the surveillance camera image (44) itself, or it may be a processed version of the surveillance camera image (44) as needed. Additionally, the function of selecting a surveillance camera image to be displayed from among a plurality of surveillance camera images (44) and processing the selected image as needed may be provided in the controller (52) instead of the management server (30).

[0023] 2. Selection of Surveillance Camera Footage

[0024] As shown in the example in FIG. 2, there are multiple surveillance cameras (10) associated with the vehicle (20), and multiple surveillance camera images (44) are input to the management server (30). Meanwhile, in the example shown in FIG. 1, the number of surveillance camera images that can be displayed on the display device (53) is at most three. Increasing the number of screens increases the number of surveillance camera images that can be displayed. However, there is a physical limit to the number of screens, and if the number of screens is increased excessively, it results in an information overload for the remote operator. Therefore, it is necessary to select a surveillance camera image that is useful to the remote operator from among the multiple surveillance camera images.

[0025] The selection of surveillance camera images is made based on information regarding the current and future relative relationship between each surveillance camera (10) related to the vehicle (20) and the vehicle (20). By considering not only the current relative relationship with the vehicle (20) but also the future relative relationship with the vehicle (20) with respect to the multiple surveillance cameras (10), it becomes clear which surveillance camera provides a more useful image for remote driving. As a result, it becomes possible to select an image useful for remote driving from among the multiple surveillance camera images (44). The method for selecting surveillance camera images will be explained below.

[0026] <Selection Method 1>

[0027] In selection method 1, the image displayed on the display device (53) is an image showing the area that the vehicle (20) is scheduled to pass through. In selection method 1, the scheduled time for the vehicle (20) to pass through the field of view of each surveillance camera (10) is referenced as information regarding the current and future relative relationship between each surveillance camera (10) and the vehicle (20). Then, the image of the surveillance camera in which the area that the vehicle (20) is scheduled to pass through within a predetermined time is included in the field of view is selected first.

[0028] FIG. 3 is a drawing illustrating a specific example of selection method 1. In FIG. 3, three surveillance cameras (10A, 10B, 10C) are installed along the road. Each surveillance camera (10A, 10B, 10C) has a field of view (11A, 11B, 11C). A vehicle (20) travels along this road. FIG. 3 shows the position of the vehicle (20) on the road at times (T11, T12, T13, T14). Additionally, at each time, the area that the vehicle (20) is scheduled to pass within a predetermined time is indicated by an arrow line in the drawing. The predetermined time is set, for example, based on the reaction time of a remote operator to a change in the surrounding environment. Therefore, the area that the vehicle (20) is scheduled to pass within a predetermined time can be rephrased as the area that the vehicle (20) is scheduled to pass in the near future.

[0029] At time (T11), the area where the vehicle (20) is scheduled to pass is only within the field of view (11A) of the surveillance camera (10A). Therefore, at time (T11), the image acquired by the surveillance camera (10A) is selected first.

[0030] At time (T12), a portion of the area where the vehicle (20) is scheduled to pass is within the field of view (11B) of the surveillance camera (10B), and the remainder is within the field of view (11A) of the surveillance camera (10A). In this case, the surveillance camera with a larger area of ​​the planned passage within its field of view is given priority. Additionally, if the size of the area of ​​the planned passage within its field of view is equal between the surveillance cameras, for example, the surveillance camera installed further forward, that is, the surveillance camera with a future area of ​​the planned passage further forward within its field of view, is given priority. Therefore, at time (T12), the image acquired by the surveillance camera (10B) is given priority.

[0031] At time (T13), the area where the vehicle (20) is scheduled to pass is only within the field of view (11B) of the surveillance camera (10B). Therefore, at time (T13), the image acquired by the surveillance camera (10B) is selected first.

[0032] At time (T14), most of the area where the vehicle (20) is scheduled to pass falls within the field of view (11C) of the surveillance camera (10C), and a portion of the remainder falls within the field of view (11B) of the surveillance camera (10B). Therefore, at time (T14), the image acquired by the surveillance camera (10C) is selected first.

[0033] As shown in the example above, by prioritizing the selection of the image from the surveillance camera that includes the area where the vehicle (20) is scheduled to pass in the near future, the remote operator can remotely drive the vehicle (20) while predicting the situation that may occur to the vehicle (20).

[0034] <Selection Method 2>

[0035] In selection method 2, the image displayed on the display device (53) is an image showing the area that the vehicle (20) is scheduled to pass through. In selection method 2, the distance from a predetermined area on the vehicle (20) to the vehicle (20) is referenced as information regarding the current and future relative relationship between each surveillance camera (10) and the vehicle (20). Then, when the vehicle (20) approaches a point at a predetermined distance from the predetermined area, the image of the surveillance camera in which the predetermined area is within the field of view is selected first. The predetermined area can be set arbitrarily, but it is preferable that it be an area requiring particular attention from the remote operator. Examples of such areas include intersections, crosswalks, curves with poor visibility, and accident-prone locations.

[0036] FIG. 4 is a drawing illustrating a specific example of selection method 2. In FIG. 4, a surveillance camera (10D) is installed on the road where the vehicle (20) is traveling, and a surveillance camera (10E) is also installed at the intersection (6) located ahead. The field of view of the surveillance camera (10D) includes the area up to the front of the intersection. The field of view of the surveillance camera (10E) includes the entire intersection. FIG. 4 shows the position of the vehicle (20) on the road at times (T21, T22, T23). Additionally, a point (P1) at a predetermined distance from the intersection (6) is shown in the drawing. The predetermined distance is set, for example, based on the time required for a remote operator to confirm safety when bringing the vehicle (20) into the intersection (6).

[0037] At time (T21), the vehicle (20) has not reached point (P1). Therefore, at time (T21), the image acquired by the surveillance camera (10D) is selected first. Also, at time (T22), the vehicle (20) has not reached point (P1). Therefore, at time (T22), the image acquired by the surveillance camera (10D) is selected first.

[0038] At time (T23), the vehicle (20) is reaching point (P1). Therefore, at time (T23), the image acquired by the surveillance camera (10E) is selected first. By doing so, the remote operator can remotely drive the vehicle (20) while observing the situation at the intersection (6) and safely pass through the intersection (6).

[0039] <Selection Method 3>

[0040] In selection method 3, the image displayed on the display device (53) is an image of the vehicle (20). In selection method 3, the time during which the vehicle (20) is in the field of view of each surveillance camera (10) is referenced as information regarding the current and future relative relationship between each surveillance camera (10) and the vehicle (20). Then, the image of the surveillance camera in which the time during which the vehicle (20) is in the field of view is longer is selected first.

[0041] FIG. 5 is a drawing illustrating a specific example of selection method 3. In FIG. 5, a surveillance camera (10F) is installed in the lane where the vehicle (20) is traveling, and a surveillance camera (10G) is installed in the opposite lane. The surveillance camera (10F) is a camera that photographs the rear of the lane, and the surveillance camera (10G) is a camera that photographs the front of the opposite lane. The surveillance camera (10F) is installed in the direction of travel ahead of the surveillance camera (10G) when viewed from the vehicle (20).

[0042] The period during which the vehicle (20) is within the field of view (11F) of the surveillance camera (10F) is from time (T31) to time (T34). Meanwhile, the period during which the vehicle (20) is within the field of view (11G) of the surveillance camera (10G) is from time (T31) to time (T33). Therefore, at least for the period from time (T31) to time (T34), the image acquired by the surveillance camera (10F) is selected preferentially.

[0043] As described above, the image of the surveillance camera in which the vehicle (20) is captured for a longer period of time is selected preferentially, thereby reducing the frequency of switching the image display on the display device (53). Additionally, as a variation of selection method 3, the higher the speed of the vehicle (20), the higher the priority of the image of the surveillance camera in which the vehicle (20) is in the field of view for a longer period of time. As the vehicle speed increases, the time the vehicle (20) is captured by a single surveillance camera becomes shorter, and thus, the frequency of switching the image during high-speed movement of the vehicle (20) can be reduced.

[0044] <Selection Method 4>

[0045] In selection method 4, the image displayed on the display device (53) is an image in which the vehicle (20) is captured. In selection method 4, the ratio of the field of view of each surveillance camera (10) to the vehicle (20) is referenced as information regarding the current and future relative relationship between each surveillance camera (10) and the vehicle (20). Then, the image of the surveillance camera in which the ratio of the field of view occupied by the vehicle (20) is greater than or equal to a first predetermined ratio and less than a second predetermined ratio is selected preferentially.

[0046] FIG. 6 is a drawing illustrating a specific example of selection method 4. In FIG. 6, three surveillance cameras (10H, 10J, 10K) are installed along a road. In this example, the ratio of the field of view (11H) occupied by the vehicle (20) for the surveillance camera (10H) is greater than or equal to a second predetermined ratio. The ratio of the field of view (11K) occupied by the vehicle (20) for the surveillance camera (10K) is less than a first predetermined ratio. And, only the ratio of the field of view (11J) occupied by the vehicle (20) for the surveillance camera (10J) is greater than or equal to a first predetermined ratio and less than a second predetermined ratio. Therefore, the image acquired by the surveillance camera (10J) is selected preferentially.

[0047] When the vehicle (20) moves further forward, sooner or later the proportion of the vehicle (20) in the field of view (11J) of the surveillance camera (10J) increases to a second predetermined ratio or higher, while the proportion of the vehicle (20) in the field of view (11K) of the surveillance camera (10K) increases to a first predetermined ratio or higher and less than a second predetermined ratio. In that case, the image acquired by the surveillance camera (10K) is selected preferentially instead of the image acquired by the surveillance camera (10J).

[0048] If the ratio of the field of view occupied by the vehicle (20) is too small, it is difficult to recognize the state of the vehicle (20) or the situation in which the vehicle (20) is positioned. Conversely, if the ratio of the field of view occupied by the vehicle (20) is too large, useful information for remote driving, in particular, the surrounding situation of the vehicle (20), cannot be obtained from the video of the surveillance camera. The above first predetermined ratio and second predetermined ratio are set to values ​​that make it possible to recognize the state of the vehicle (20) and the surrounding situation of the vehicle (20) from the video of the surveillance camera.

[0049] <Selection Method 5>

[0050] In selection method 5, the image displayed on the display device (53) is an image of the vehicle (20). In selection method 5, the shooting direction of each surveillance camera (10) for the vehicle (20) is referenced. Then, an image from a surveillance camera that captures the vehicle (20) from a specific direction, for example, from the rear, is selected first. An image of the vehicle (20) captured from the rear causes less discomfort when the remote operator drives while viewing the image.

[0051] <Selection Method 6>

[0052] In selection method 6, the image displayed on the display device (53) is an image of the surroundings of the vehicle (20). In selection method 6, the difference between the field of view of each surveillance camera (10) and the field of view of the vehicle-mounted camera (24) is referenced. And, for example, the image of the surveillance camera that captures a place that is difficult to see or a blind spot in the image of the vehicle-mounted camera (24) is selected first.

[0053] <Selection Method Based on Other Information>

[0054] You may prioritize selecting an image from a surveillance camera that captures an object approaching the path of the vehicle (20), or an object that appears likely to move within or near the path.

[0055] 3. Display of surveillance camera footage on a display device

[0056] In this embodiment, the image of a surveillance camera is selected by the selection method of any of the above. Each surveillance camera is associated with one of three sub-screens (56C, 56L, 56R) according to a predetermined distribution rule. That is, the display location of the selected surveillance camera image, that is, the screen on which the selected surveillance camera image is displayed, is predetermined for each surveillance camera. If the remote operator knows in advance which surveillance camera image is displayed on which screen, the remote operator can understand which image is displayed on each screen without confusion.

[0057] Here, we examine the example shown in FIG. 7. In this example, multiple surveillance cameras (10L, 10M, 10N, 10P, 10Q) are installed in the section leading to the intersection (6) and around the intersection (6). The surveillance camera (10L) is installed on the left side of the intersection (6) and films the right direction. The surveillance camera (10N) is installed on the right side of the intersection (6) and films the left direction. The surveillance camera (10M) is installed at the right corner of the intersection (6) and films the right direction. The surveillance camera (10P) is installed in front of the intersection (6) and films the front. The surveillance camera (10Q) is installed on the opposite side of the intersection (6) and films the rear.

[0058] In this embodiment, the image of the surveillance camera displayed on the sub-screens (56C, 56L, 56R) of the display device (53) is changed according to the distance to the intersection (6) of the vehicle (20) and the driving route passing through the intersection (6). By doing so, an image useful for remote driving can be displayed to the remote operator in an optimal viewing manner using multiple screens.

[0059] FIG. 7 shows an example of the transition of surveillance camera images displayed on three sub-screens (56C, 56L, 56R) when a vehicle (20) traveling straight turns right at an intersection (6). FIG. 7 shows two examples.

[0060] First, the example on the left side of FIG. 7 will be explained. At time (T41) when the vehicle (20) is located far from the intersection (6), the image (45P) of the surveillance camera (10P) is displayed on the central screen (56C). No image is displayed on the left screen (56L) and the right screen (56R).

[0061] At the time (T42) when the vehicle (20) approaches the entrance of the intersection (6), the image on the central screen (56C) is turned off. Meanwhile, the image (45N) of the surveillance camera (10N) is displayed on the left screen (56L), and the image (45L) of the surveillance camera (10L) is displayed on the right screen (56R). These images (45N, 45L) are images that best capture the appearance of the place where the vehicle (20) enters when the vehicle (20) turns right at the intersection (6), that is, images that best observe the place that requires caution when the vehicle (20) turns right at the intersection (6).

[0062] At the time (T43) when the vehicle (20) reaches the vicinity of the center of the intersection (6), the image on the central screen (56C) is turned off. Additionally, the image (45N) of the surveillance camera (10N) continues to be displayed on the left screen (56L), and the image (45L) of the surveillance camera (10L) continues to be displayed on the right screen (56R).

[0063] Then, at the time (T44) when the vehicle (20) passes the intersection (6), the image (45M) of the surveillance camera (10M) is displayed on the central screen (56C). Meanwhile, the images on the left screen (56L) and the right screen (56R) are turned off.

[0064] Next, the example on the right side of FIG. 7 is described. At time (T41) when the vehicle (20) is far from the intersection (6), the image (45P) of the surveillance camera (10P) is displayed on the central screen (56C). No image is displayed on the left screen (56L) and the right screen (56R).

[0065] At the time (T42) when the vehicle (20) approaches the entrance of the intersection (6), the image (45N) of the surveillance camera (10N) is displayed on the left screen (56L), and the image (45L) of the surveillance camera (10L) is displayed on the right screen (56R). Additionally, the image on the central screen (56C) switches from the image (45P) of the surveillance camera (10P) to the image (45M) of the surveillance camera (10M).

[0066] At the time (T43) when the vehicle (20) reaches the vicinity of the center of the intersection (6), the image (45M) of the surveillance camera (10M) is continuously displayed on the central screen (56C). Additionally, the image (45N) of the surveillance camera (10N) is continuously displayed on the left screen (56L), and the image (45L) of the surveillance camera (10L) is continuously displayed on the right screen (56R).

[0067] Then, at the time (T44) when the vehicle (20) passes the intersection (6), the images on the left screen (56L) and the right screen (56R) are turned off. Meanwhile, the image (45M) of the surveillance camera (10M) continues to be displayed on the central screen (56C). That is, after the vehicle (20) enters the intersection (6), the image (45M) of the surveillance camera (10M) continues to be displayed on the central screen (56C). This allows for a reduction in the frequency of switching the image display.

[0068] However, up to this point, the explanation has been given on the premise that the display device (53) is provided with three sub-screens (56C, 56L, 56R), but there may be cases where there is only one screen for displaying surveillance camera footage. In that case, for example, surveillance camera footage is selected in the following order of priority.

[0069] Whether the vehicle (20) is moving straight, turning right, or turning left at the intersection (6), if the vehicle speed is high or low, an example of the priority of surveillance camera images is surveillance camera (10P), surveillance camera (10Q), surveillance camera (10L), and surveillance camera (10N) in order of highest priority. This means that if surveillance camera (10P) is present, the image of surveillance camera (10P) is displayed, and if surveillance camera (10P) is not present, the image of surveillance camera (10Q) is displayed. Surveillance camera (10L) and surveillance camera (10N) are of equal rank.

[0070] When a vehicle (20) turns right at an intersection (6) at medium speed, in comparison between the surveillance camera (10N) and the surveillance camera (10L), the surveillance camera (10N) has a higher priority than the surveillance camera (10L). When the vehicle (20) turns right at an intersection (6), the area requiring attention is obscured behind the vehicle (20) in the surveillance camera (10L), but according to the surveillance camera (10N), it is possible to obtain an image in which the area requiring attention during the right turn is not obscured behind the vehicle (20).

[0071] In the comparison between the surveillance camera (10M) and the surveillance camera (10N), the surveillance camera (10M) has a higher priority than the surveillance camera (10N). Since the image of the surveillance camera (10M) best captures the appearance of the place where the vehicle (20) turns right and enters the intersection (6), this can be selected first when turning right. In addition, in the comparison between the surveillance camera (10L), the surveillance camera (10P), and the surveillance camera (10Q), the priority is higher in the order of surveillance camera (10L), surveillance camera (10P), and surveillance camera (10Q).

[0072] When the vehicle (20) travels straight through the intersection (6) at medium speed, an example of the priority of the surveillance camera images is surveillance camera (10N), surveillance camera (10L), surveillance camera (10P), and surveillance camera (10Q) in order of highest priority. Surveillance camera (10L) and surveillance camera (10N) are of equal rank.

[0073] 4. Processing of surveillance camera footage

[0074] The surveillance camera image displayed on the display device (53) is processed as needed. For example, an example of processing the surveillance camera image when the surveillance camera image is switched as shown in FIG. 8 is described. In the example shown in FIG. 8, two surveillance cameras (10R, 10S) are installed at the intersection (T-shaped) (8). The surveillance camera (10R) is installed at the dead end of the intersection (8) and photographs the left side of the intersection (8). The surveillance camera (10S) is installed at the dead end of the intersection (8) and photographs the right side of the intersection (8).

[0075] FIG. 8 illustrates an example of the transition of surveillance camera images displayed on two sub-screens (56L, 56R) when a vehicle (20) traveling straight turns right at an intersection (8). At time (T51), when the vehicle (20) is far from the intersection (8), no images are displayed on the left screen (56L) and the right screen (56R). At time (T52), when the vehicle (20) approaches the entrance of the intersection (8), the image (45R) of the surveillance camera (10R) is displayed on the left screen (56L), and the image (45S) of the surveillance camera (10S) is displayed on the right screen (56R). At the time (T53) when the vehicle (20) reaches the vicinity of the center of the intersection (8), the image (45R) of the surveillance camera (10R) is continuously displayed on the left screen (56L), and the image (45S) of the surveillance camera (10S) is continuously displayed on the right screen (56R). Then, at the time (T54) when the vehicle (20) passes the intersection (8), the images on the left screen (56L) and the right screen (56R) are turned off.

[0076] <Processing Method 1>

[0077] FIG. 9 is a drawing showing an example of processing method 1 of a surveillance camera image displayed on a display device (53). FIG. 9 corresponds to the surveillance camera image displayed on the left screen (56L) and the surveillance camera image displayed on the right screen (56R) in the example shown in FIG. 8. However, in the example shown in FIG. 9, a left-right flipping process is performed on each of the surveillance camera image on the left screen (56L) and the surveillance camera image on the right screen (56R).

[0078] Judging from the view of the surveillance cameras (10R, 10S), at each time point (T52) and time point (T53), the vehicle (20) that has entered the intersection (8) will be captured on both the left screen (56L) and the right screen (56R). However, according to processing method 1, the vehicle (20) is covered with a mask (60L, 60R) for each of the surveillance camera images of the left screen (56L) and the right screen (56R).

[0079] At the intersection (8) where the direction of travel of the vehicle (20) changes, remote driving while viewing footage of the vehicle (20) taken from the outside is likely to cause confusion for the remote operator. By covering the vehicle (20) with a mask (60L, 60R), information useful for remote driving included in the surveillance camera footage can be preserved while preventing confusion for the remote operator. Additionally, instead of covering the vehicle (20) by mask processing, the vehicle (20) may be removed from the surveillance camera footage using an image processing application.

[0080] <Processing Method 2>

[0081] FIG. 10 is a drawing showing an example of method 2 for processing surveillance camera images displayed on a display device (53). FIG. 10 corresponds to the surveillance camera image displayed on the left screen (56L) and the surveillance camera image displayed on the right screen (56R) in the example shown in FIG. 8. However, in the example shown in FIG. 10, a left-right flipping process is performed on each of the surveillance camera image on the left screen (56L) and the surveillance camera image on the right screen (56R).

[0082] In processing method 2, the vehicle (20) is not obscured so that the relationship between the vehicle (20) and its surroundings can be understood. However, in order to know the direction of travel of the vehicle (20) at the intersection (8), processing is performed to superimpose AR arrows (62L, 62R) indicating the direction of travel of the vehicle (20) onto the respective surveillance camera images of the left screen (56L) and the right screen (56R). Since the remote operator can intuitively understand the direction of the vehicle (20) from the AR arrows (62L, 62R), confusion caused by the vehicle (20) being captured on the screen (56L, 56R) can be reduced.

[0083] If the vehicle (20) is at a low speed, the confusion time allowed to the remote operator is extended. Therefore, if the speed of the vehicle (20) is below a predetermined speed, processing by processing method 2 may be performed, and if the speed of the vehicle (20) is above a predetermined speed, processing by processing method 1 may be performed.

[0084] 5. Variations of the display method for surveillance camera images

[0085] FIG. 11 is a diagram showing a modified example of a method for displaying surveillance camera images. In modified example A, surveillance camera images (45C, 45L, 45R) are superimposed on vehicle-mounted camera images (41C, 41L, 41R) on three screens (54C, 54L, 54R). In modified example B, surveillance camera images (45L, 45R) are superimposed on vehicle-mounted camera images (41L, 41R) on left and right screens (54L, 54R). In modified example C, two surveillance camera images (45L, 45R) are superimposed on vehicle-mounted camera images (41C) on the central screen (54C). The position where surveillance camera images are displayed on each screen is preferably a position that is blind spot in the vehicle-mounted camera images or a position of the reflector of the vehicle (20).

[0086] By overlaying the surveillance camera image onto the vehicle-mounted camera image, the remote operator can view the surveillance camera image while simultaneously viewing the vehicle-mounted camera image. Additionally, the surveillance camera image may be overlaid onto the vehicle-mounted camera image only for a limited period from when the vehicle (20) approaches the intersection until it passes the intersection. By limiting the overlaying of the surveillance camera image to cases where remote driving is particularly required, the computational load of the processor (32) can be reduced while preventing the image overlaid onto the vehicle-mounted camera image from interfering with the remote operator. Explanation of the symbols

[0087] 6 intersections 10 surveillance cameras 20 vehicles 21 Vehicle-mounted computer 24 Vehicle-mounted Cameras 30 management servers 32 processors 34 Program Memory 36 Instructions 38 video memory 40 Vehicle-mounted Camera Video 41 Vehicle-mounted camera image after processing 44 surveillance camera footage 45 Surveillance camera footage after processing 50 Remote Cockpit 53 display device 54C, 54L, 54R Main Screen 56C, 56L, 56R sub-screens

Claims

Claim 1 A remote support system for providing remote support to a vehicle, comprising: a display device for displaying images to a remote operator; at least one processor; and at least one memory storing a plurality of instructions executable by the at least one processor. The plurality of instructions are configured to cause the at least one processor to execute the following: acquiring images related to the operation of the vehicle from a plurality of surveillance cameras installed on or near the vehicle's driving path; selecting an image to be displayed on the display device from among the images acquired by the plurality of surveillance cameras based on information regarding the current and future relative relationship between each of the plurality of surveillance cameras and the vehicle; and, when displaying the selected image on the display device, performing a process to remove a vehicle that has entered an intersection or a process to obscure a vehicle that has entered an intersection. Claim 2 A remote support system according to claim 1, wherein the video related to the operation of the vehicle includes a video of an area to be passed by the vehicle, and selecting the video to be displayed on the display device includes prioritizing the selection of a video from a surveillance camera in which the area to be passed by the vehicle within a predetermined time is within the field of view. Claim 3 A remote support system according to claim 1, wherein the video related to the operation of the vehicle includes a video of an area to be passed by the vehicle, and selecting the video to be displayed on the display device includes, when the vehicle approaches a point within a predetermined distance from a predetermined area on the driving path, prioritizing the selection of a video from a surveillance camera in which the predetermined area is within the field of view. Claim 4 A remote support system according to claim 1, wherein the video related to the operation of the vehicle includes a video in which the vehicle is captured, and selecting the video to be displayed on the display device includes prioritizing the selection of a video from a surveillance camera in which the vehicle is in the field of view for a longer period of time. Claim 5 A remote support system according to claim 4, wherein the plurality of instructions are configured to have at least one processor increase the priority of the image of a surveillance camera in which the vehicle is in the field of view for a longer period as the speed of the vehicle increases. Claim 6 A remote support system according to claim 1, wherein the video related to the operation of the vehicle includes a video in which the vehicle is captured, and selecting the video to be displayed on the display device includes preferentially selecting a video from a surveillance camera in which the ratio of the field of view occupied by the vehicle is greater than or equal to a first predetermined ratio and less than a second predetermined ratio. Claim 7 A remote support system according to claim 1, wherein the display device includes a plurality of screens, and the plurality of instructions are configured to cause at least one processor to associate each of the plurality of surveillance cameras with any of the plurality of screens according to a predetermined distribution rule, and to select an image to be displayed from among images acquired by the associated surveillance camera for each of the plurality of screens. Claim 8 A remote support system according to claim 1, wherein the plurality of instructions are configured to cause the at least one processor to execute a different display mode for displaying the selected image on the display device according to the current and future relative relationship between the surveillance camera that captured the selected image and the vehicle. Claim 9 A remote support system according to claim 8, wherein the display device comprises a plurality of screens, the selected image comprises a plurality of selected images, and the plurality of instructions are configured to cause the at least one processor to execute a different arrangement of the plurality of selected images to the plurality of screens according to the current and future relative relationship between the surveillance camera corresponding to each of the plurality of selected images and the vehicle. Claim 10 A remote support system according to claim 1, characterized in that the image displayed on the display device is an image in which the vehicle is not captured. Claim 11 delete Claim 12 A remote support system according to claim 10, wherein the plurality of instructions are configured to cause at least one processor to display an image of the vehicle that is not captured on the display device during the period from when the vehicle approaches an intersection until it passes through the intersection. Claim 13 A remote support system according to claim 12, wherein the plurality of instructions are configured to execute, in at least one processor, displaying an image of the vehicle not captured on the display device when the speed of the vehicle is greater than or equal to a predetermined speed, and displaying an image of the vehicle captured on the display device when the speed of the vehicle is less than a predetermined speed. Claim 14 A remote support system according to claim 13, wherein the plurality of instructions are configured to cause at least one processor to display an arrow indicating the direction of travel of the vehicle at the intersection superimposed on the display device when displaying an image of the vehicle on the display device. Claim 15 A remote support system according to claim 1, wherein selecting an image to be displayed on the display device includes, in the case where the vehicle turns right or left at an intersection, prioritizing the selection of an image that best captures the appearance of the place where the vehicle enters by turning right or left at the intersection. Claim 16 A remote support system according to claim 15, wherein the priority selection of the image in which the vehicle enters the intersection by turning right or left includes selecting an image from a surveillance camera installed on the right side of the intersection and capturing the left direction, or selecting an image from a surveillance camera installed on the left side of the intersection and capturing the right direction. Claim 17 A remote support system according to claim 1, wherein the plurality of instructions are configured to execute, on at least one processor, displaying a vehicle-mounted camera image captured by a vehicle-mounted camera mounted on the vehicle on the display device and superimposing the selected image onto the vehicle-mounted camera image. Claim 18 A remote support system according to claim 17, wherein the plurality of instructions are configured to execute the superposition of the selected image onto the vehicle-mounted camera image during the period from when the vehicle approaches an intersection until it passes through the intersection. Claim 19 A remote support method for providing remote support to a vehicle, characterized by comprising: acquiring images related to the operation of the vehicle from a plurality of surveillance cameras installed on or near the vehicle's driving path; selecting an image to be displayed on a display device for a remote operator from among the images acquired by the plurality of surveillance cameras based on information regarding the current and future relative relationship between each of the plurality of surveillance cameras and the vehicle; and, when displaying the selected image on the display device, performing a process to remove a vehicle that has entered an intersection or a process to obscure a vehicle that has entered an intersection. Claim 20 A non-transient computer-readable recording medium storing a remote support program for providing remote support to a vehicle, wherein the remote support program is configured to execute on a computer the following: acquiring images related to the operation of the vehicle from a plurality of surveillance cameras installed on or near the vehicle's driving path; selecting an image to be displayed on a display device for a remote operator from among the images acquired by the plurality of surveillance cameras based on information regarding the current and future relative relationship between each of the plurality of surveillance cameras and the vehicle; and performing a process to remove a vehicle entering an intersection or a process to obscure a vehicle entering an intersection when the selected image is displayed on the display device.

Images