Remote operation system, program and method

Abstract

To guide a line of sight of an operator from a display device in a front direction to a display device in the same direction as an operation direction of a turn indicator when the operator who performs remote driving operates the turn indicator.SOLUTION: A remote driving system comprises one or more storage devices, one or more processors and at least three display devices. The storage device records camera image information (information of front center, left and right camera images) of a mobile and information about an indicated direction of a turn indicator of the mobile. The one or more processors select an image on which emphasized display is performed from the left and right camera images based on the information of the indicated direction. A symbol superposed image is then generated by superposing and displaying a symbol on the selected image. Further, the symbol superposed image is displayed on the display device in the same direction of the indicated direction.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a technology for remotely controlling the operation of a mobile object. [Background technology]

[0002] Patent Document 1 discloses a technology related to an electronic mirror system that displays images of the left and right rear of a vehicle on two display devices arranged on the left and right of the instrument panel. With this technology, when a turn signal is operated, the brightness level of the camera image in the same direction as the turn signal is increased. This makes it possible to improve the visibility of the camera image in the same direction as the turn signal when turning left or right. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-036924 A Summary of the Invention [Problem to be solved by the invention]

[0004] In remote driving of a moving object, at least three display devices are used so that an operator who performs remote driving can check the situation around the moving object. The at least three display devices display, for example, images in the front direction, left direction, and right direction of the moving object. The display device in the front direction displays a front camera image captured to see the front. The display devices in the left direction and right direction display front camera images captured to see the left front and right front, respectively.

[0005] When an operator who is driving remotely operates the turn signal, it is expected that the operator will move his / her gaze from the display device facing forward to a display device facing the same direction as the turn signal operation. However, it has become clear that in reality, the gaze tends to stop to the side of the display device facing forward. Therefore, an improvement is needed to guide the operator's gaze to the display device facing the same direction as the turn signal indication.

[0006] One object of the present invention is to provide a technology that, when an operator driving remotely operates a turn signal, can guide the operator's gaze from a display device facing forward to a display device in the same direction as the turn signal is operated. [Means for solving the problem]

[0007] The first aspect relates to a remote driving system for remotely controlling the operation of a moving object. The remote operation system is Camera image information including a camera image of the center of the front of the moving body, a camera image of the left side including the front left of the moving body, and a camera image of the right side including the front right of the moving body, and information on the indication direction of the turn signal of the moving body; Velocity information of the moving object; one or more storage devices having the above-mentioned recorded therein; one or more processors; At least three display devices that display the front camera image in the center, the left camera image, and the right camera image. Equipped with. The left camera image includes a front left front camera image and a left side camera image. The right camera image includes a front camera image on the right front side and a side camera image on the right side. The one or more processors determine whether or not a turn signal is being operated based on the information on the indicated direction. When the one or more processors determine that a turn signal is being operated, In the case where the speed information indicates a speed equal to or greater than a predetermined speed, the camera image in the same direction as the designated direction from among the left and right side camera images is selected as the image to be highlighted. When the one or more processors determine that the turn signal is being operated and the speed information indicates a speed less than a predetermined speed, they select the camera image in the same direction as the indicated direction from among the front left and front right front camera images as the image to be highlighted. The one or more processors generate a symbol superimposed image by superimposing a symbol on the selected image. Then, the one or more processors display the symbol superimposed image on one of the at least three display devices that displays a camera image in the same direction as the indicated direction.

[0009] No. 2 The viewpoint is 1 of In addition to the above, the present invention has the following features: 。 1 Alternatively, the multiple processors further generate a left superimposed image in which a left side camera image is superimposed on a front left front camera image, and a right superimposed image in which a right side camera image is superimposed on a front right front camera image. The one or more processors select from the left superimposed image and the right superimposed image. ,strength Select the image to be displayed.

[0010] No. 3 The above aspect relates to a remote driving program for remotely controlling the operation of a mobile object. The remote operation program is Camera image information including a camera image of the center of the front of the moving body, a camera image of the left side including the front left of the moving body, and a camera image of the right side including the front right of the moving body, and information on the indication direction of the turn signal of the moving body and the speed information of the moving object. A process of obtaining A process of determining whether or not a turn signal is being operated based on information about the indicated direction; It is determined that the turn signal is being operated. and if the speed information indicates a speed equal to or greater than a predetermined speed, a process of selecting, as an image to be highlighted, a camera image in the same direction as the indicated direction from among a left side camera image included in the left camera image and a right side camera image included in the right camera image; a process of determining that a turn signal is being operated and, when the speed information indicates a speed less than a predetermined speed, selecting, as an image to be highlighted, a camera image in the same direction as the indicated direction from among a front left front camera image included in the left camera image and a front right front camera image included in the right camera image; A process of generating a symbol superimposed image by superimposing a symbol on the selected image; A process of displaying the symbol superimposed image on a display device that displays a camera image in the same direction as the indicated direction among at least three display devices that display camera images in the center, left, and right directions, respectively; is executed by a computer.

[0012] No. 4 The viewpoint is 3 In addition to the above, the following features are also present:。 far The intermittent operation program further includes: The computer is caused to execute a process of generating a left superimposed image in which a left side camera image is superimposed on a front left front camera image, and a right superimposed image in which a right side camera image is superimposed on a front right front camera image. The process of selecting an image to be highlighted is performed based on the left superimposed image and the right superimposed image.

[0013] A fifth aspect relates to a remote driving method for remotely controlling the operation of a moving object. The remote operation method is Camera image information including a camera image of the center of the front of the moving body, a camera image of the left side including the front left of the moving body, and a camera image of the right side including the front right of the moving body, and information on the indication direction of the turn signal of the moving body and the speed information of the moving object. A process of obtaining A process of determining whether or not a turn signal is being operated based on information about the indicated direction; It is determined that the turn signal is being operated. and if the speed information indicates a speed equal to or greater than a predetermined speed, a process of selecting, as an image to be highlighted, a camera image in the same direction as the indicated direction from among a left side camera image included in the left camera image and a right side camera image included in the right camera image; a process of determining that a turn signal is being operated and, when the speed information indicates a speed less than a predetermined speed, selecting, as an image to be highlighted, a camera image in the same direction as the indicated direction from among a front left front camera image included in the left camera image and a front right front camera image included in the right camera image; A process of generating a symbol superimposed image by superimposing a symbol on the selected image; A process of displaying the symbol superimposed image on a display device that displays a camera image in the same direction as the indicated direction among at least three display devices that display camera images in the center, left, and right directions, respectively; Includes.

[0015] No. 6 The viewpoint is 5 In addition to the above, the following features are also present: 。 far The alternate operation method is The method further includes processing for generating a left superimposed image in which a left side camera image is superimposed on a front left front camera image, and a right superimposed image in which a right side camera image is superimposed on a front right front camera image. The process of selecting an image to be highlighted is performed based on the left superimposed image and the right superimposed image. Effect of the Invention

[0016] According to the first aspect, an image to be highlighted is selected from among the left and right camera images based on information on the direction indicated by the blinker of the mobile object. Then, a symbol superimposed image is generated by superimposing a symbol on the selected image. Furthermore, the symbol superimposed image is displayed on one of at least three display devices that displays a camera image in the same direction as the indicated direction. This makes it possible to guide the operator's line of sight from the display device that displays the front camera image of the center forward position to the display device that displays the camera image of the left or right direction that is the same as the direction indicated by the blinker. This leads to improved operability of the mobile object when remotely operated and ensuring the driving safety of the mobile object.

[0017] Furthermore, No. 1From the viewpoint of, the left camera image includes a front camera image of the front left and a side camera image of the left, and the right camera image includes a front camera image of the front right and a side camera image of the right. In selecting an image to be highlighted, if the speed of the speed information is equal to or higher than a predetermined speed, a camera image of the left and right side camera images in the same direction as the indicated direction is selected. If the speed of the speed information is less than a predetermined speed, a camera image of the front camera image of the front left and right is selected in the same direction as the indicated direction. The operation of the turn signal is performed regardless of the speed of the moving body, but the operation of the turn signal when turning left or right is generally performed while moving at a low speed. On the other hand, the operation of the turn signal while moving at a high speed is considered to be for the purpose of changing lanes. Here, when making a left or right turn, the operator needs to check the safety of the front in order to pay particular attention to objects such as pedestrians that exist at the left or right turn destination. On the other hand, when changing lanes, the operator needs to check the safety of the side and rear in order to pay particular attention to the following moving body that is approaching the moving body in the lane to which the lane is changed. In this regard, according to the second viewpoint, the camera image is selected based on the result of comparing the speed of the speed information with a predetermined speed, so that, for example, the operator can be prompted to pay attention to the front camera image in the same direction as the indicated direction during low-speed movement, and the operator can be prompted to pay attention to the side camera image in the same direction as the indicated direction during high-speed movement. This leads to improved operability of the mobile body by remote driving and ensuring the traveling safety of the mobile body.

[0018] No. 2 According to the viewpoint, in the superimposed image in the same direction as the indication direction of the turn signal, one of the front camera image and the side camera image is highlighted, 1 The same effect as that from the viewpoint of the point of view can be obtained. In addition, the front camera image and the side camera image in the direction corresponding to the indicated direction are displayed as one superimposed image on the display device corresponding to the indicated direction, so that the operator can shift his / her gaze to the highlighted image without feeling uncomfortable. This leads to improved operability of the mobile body by remote driving and ensuring the driving safety of the mobile body.

[0019] No. 3 According to this aspect, the same effect as that of the first aspect can be obtained.

[0021] No. 4 From this point of view, 2 The same effect can be obtained from the viewpoint of

[0022] No. 5 According to this aspect, the same effect as that of the first aspect can be obtained.

[0024] No. 6 From this point of view, 2 The same effect can be obtained from the viewpoint of [Brief description of the drawings]

[0025] [Figure 1] 1 is a block diagram showing a configuration example of a remote operation system according to a first embodiment of the present invention. [Diagram 2] 1 is a block diagram showing an example of functions of a remote operation system according to a first embodiment of the present invention. [Diagram 3] 5 is a flowchart showing a process performed by a highlighted image selection unit of the remote driving system according to the first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing an example of a processing result (example 1 of a case where a highlighting is displayed on a front camera image) by a symbol superimposing unit of the remote driving system according to the first embodiment of the present invention. [Diagram 5] FIG. 13 is a diagram showing an example of a processing result (example 2 of a case where a highlighting is displayed on a front camera image) by the symbol superimposition unit of the remote driving system according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a diagram showing an example (an example in which there is one left camera image and one right camera image) displayed on the display device of the remote driving system according to the first embodiment of the present invention. [Figure 7] FIG. 2 is a diagram showing an example (an example in which there are a plurality of left camera images and a plurality of right camera images) displayed on the display device of the remote driving system according to the first embodiment of the present invention. [Figure 8]FIG. 11 is a block diagram showing a configuration example of a remote operation system according to a second embodiment of the present invention. [Figure 9] FIG. 11 is a block diagram showing an example of functions of a remote operation system according to a second embodiment of the present invention. [Figure 10] 10 is a flowchart showing a process performed by a highlighted image selection unit of the remote driving system according to the second embodiment of the present invention. [Figure 11] FIG. 11 is a diagram showing an example of what is displayed on display devices of the remote operation system according to the second embodiment of the present invention (an example in which camera images are displayed on three display devices). [Figure 12] FIG. 11 is a diagram showing an example of what is displayed on display devices of the remote operation system according to the second embodiment of the present invention (an example in which camera images are displayed on three or more display devices). [Figure 13] FIG. 11 is a block diagram showing an example of functions of a remote operation system according to a third embodiment of the present invention. [Figure 14] FIG. 11 is a block diagram showing an example of a processing result by an image superimposition unit of the remote driving system according to the third embodiment of the present invention. [Figure 15] FIG. 11 is a diagram showing an example of what is displayed on a display device of a remote operation system according to the third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A remote operation system, a remote operation program, and a remote operation method according to embodiments of the present invention will be described with reference to the accompanying drawings.

[0027] 1. First embodiment 1-1. Configuration example The remote driving system according to this embodiment is for remotely controlling the operation of a moving object. Fig. 1 is a block diagram showing a configuration example of a remote driving system 10 according to this embodiment. The remote driving system 10 includes an information processing device 20 and a display device 30.

[0028] The information processing device 20 performs various information processing. The information processing device 20 includes one or more processors 40 (hereinafter simply referred to as processor 40) and one or more storage devices 50 (hereinafter simply referred to as storage devices 50). The processor 40 executes various processes. For example, the processor 40 includes a CPU. The storage device 50 stores various information such as camera image information 60 and turn signal indication direction information 130. Examples of the storage device 50 include volatile memory, non-volatile memory, HDD, SSD, etc. The processor 40 executes a remote driving program, which is a computer program, to realize the functions of the information processing device 20. The remote driving program is stored in the storage device 50. The remote driving program may be recorded in a computer-readable recording medium. The remote driving program may be provided via a network.

[0029] The camera image information 60 includes a front camera image 70 at the front center, a left camera image 75, and a right camera image 85. Here, the front camera image 70 at the front center is an image for viewing the situation at the front center of the moving object. The left camera image 75 is an image for viewing the situation at the front left, left side, and left rear of the moving object. The right camera image 85 is an image for viewing the situation at the front right, right side, and right rear of the moving object. These camera images are acquired from multiple cameras mounted on the moving object. The turn signal indication direction information 130 is information indicating the indication direction to the left or right.

[0030] The display devices 30 display various information. There are at least three display devices 30 for displaying a front camera image 70 at the center of the front, a left camera image 75, and a right camera image 85. Examples of the display devices 30 include a liquid crystal display, an organic EL display, a head mounted display, a touch panel, and the like.

[0031] 1-2. Details of information processing In order to highlight the camera image displayed on the display device 30 in the same direction as the turn signal indication direction and to guide the operator's gaze to the highlighted camera image, the information processing of this embodiment includes characteristic processing as described below.

[0032] 2 is a block diagram showing an example of functions related to information processing according to this embodiment. The information processing device 20 of the remote driving system 10 includes, as functional blocks, an information input unit 400, a highlighted image selection unit 420, a symbol superimposition unit 430, and a display processing unit 440. These functional blocks are realized by the processor 40 executing a remote driving program.

[0033] The information input unit 400 performs a process of inputting various information recorded in the storage device 50. Of the various information, a front camera image 70 of the center forward view included in the camera image information 60 is input to a display processing unit 440, a left camera image 75 and a right camera image 85 included in the camera image information 60 are input to a symbol superimposition unit 430, and information 130 on the indication direction of the turn signal is input to a highlighted display image selection unit 420.

[0034] The highlight image selection unit 420 selects an image to be highlighted from among the camera images including the left camera image 75 and the right camera image 85, based on the input turn signal direction information 130. Then, highlight selection information 520 indicating information on the selected image is generated. The processing of the highlight image selection unit 420 will be described in detail later.

[0035] The symbol superimposition unit 430 generates a symbol superimposition image 530 in which a highlight item is superimposed on an image selected for highlighting from among camera images including the left camera image 75 and the right camera image 85, based on the highlight selection information 520 generated by the highlight image selection unit 420. Note that in the highlight selection information 520, the image of the left camera image 75 or the right camera image 85, which is in a direction different from the indication direction of the winker, is generated as an image without symbol superimposition 540, since no symbol superimposition is performed on the image.

[0036] The display processing unit 440 performs processing for outputting the input front camera image 70 of the center forward area, the symbol superimposed image 530, and the symbol non-superimposed image 540 to the display device 30.

[0037] FIG. 3 is a flowchart showing the process of the highlighted display image selection unit 420 based on the information 130 on the turn signal direction.

[0038] In step S100, the highlighted display image selection unit 420 determines whether the input blinker direction information 130 includes information on the left or right direction of the blinker indication.

[0039] If it is determined that the information on the left or right direction of the blinker is included (step S100; Yes), the process proceeds to step S110. Otherwise (step S100; No), the process proceeds to step S120.

[0040] In step S110, the highlight image selection unit 420 selects the left camera image 75 or the right camera image 85 in the same direction as the turn signal indication as the image to be highlighted, and generates highlight selection information 520 indicating information about the selected image.

[0041] In step S120, the highlight display image selection unit 420 selects no image as an image to be highlighted, and generates highlight display selection information 520 indicating information about the selected image.

[0042] In step S 130 , the emphasis image selection unit 420 outputs the emphasis selection information 520 to the display processing unit 440 .

[0043] 1-4. Example of symbol overlay image display 4 shows display examples of a symbol superimposed image 530 and a symbol non-superimposed image 540 generated by the symbol superimposition unit 430. Specifically, when the image to be highlighted in the input highlighting selection information 520 is the left camera image 75, the symbol superimposed image 530 is an image in which a symbol 550 of a highlighted item is superimposed on the left camera image 75 of the image to be highlighted. On the other hand, the symbol non-superimposed image 540 is a right camera image 85 in a direction different from the indication direction of the winker.

[0044] The shape of the symbol 550 superimposed on the left camera image 75 may be a frame shape along the periphery of the left camera image 75 as shown in Fig. 4, or an arrow shape indicating the direction of the turn signal as shown in Fig. 5. The brightness and color of the symbol 550 superimposed on the left camera image 75 are arbitrary, and it is desirable to adjust them appropriately to enhance the operator's recognition of the symbol superimposed image 530. At least one of the brightness and color of the symbol 550 may be adjusted in time series, in which case the symbol 550 is displayed so that at least one of the brightness and color changes in time series (including blinking of the symbol 550).

[0045] 1-5. Example of an image displayed on a display device 6 is a display example of an image displayed on the display device 30. Here, an example is given in which a front camera image 70 of the center forward is displayed on the display device 30 facing forward, a symbol superimposed image 530 is displayed on the display device 30 facing left, and a symbol non-superimposed image 540 is displayed on the display device 30 facing right.

[0046] As an example of displaying the symbol superimposed image 530 and the symbol non-superimposed image 540 on the display device 30, when there are multiple camera images including a left camera image 75 and a right camera image 85, the multiple camera images may be displayed on each display device 30 as shown in Fig. 7. Alternatively, multiple symbol superimposed images 530 in the same direction as the indication direction of the turn signal may be displayed on at least one display device 30 that displays a camera image in the same direction as the indication direction, and multiple symbol non-superimposed images 540 in a direction different from the indication direction of the turn signal may be displayed on at least one display device 30 that displays a camera image in a direction different from the indication direction.

[0047] Effects When an operator operates a turn signal during remote driving, the operator's line of sight can be guided from the display device in the front direction to the display device in the same direction as the turn signal. Therefore, even when the operator operates the turn signal, the operability of the mobile body by remote driving can be improved and the running safety of the mobile body can be ensured.

[0048] 2. Second embodiment The second embodiment differs from the first embodiment in the configuration example and details of information processing, and descriptions that overlap with the first embodiment will be omitted as appropriate.

[0049] In the second embodiment, attention is paid to a scene in which a turn signal is operated while a moving object is traveling. The turn signal is operated regardless of the speed of the moving object, but the turn signal is generally operated when turning left or right while the moving object is moving at a low speed. On the other hand, the turn signal is operated while moving at a high speed for the purpose of changing lanes. Here, when turning left or right, the operator needs to check the safety ahead in order to pay special attention to objects such as pedestrians that exist at the left or right turn destination. On the other hand, when changing lanes, the operator needs to check the safety on the sides and rear in order to pay special attention to a following moving object approaching the moving object in the lane to which the moving object is to be changed. Therefore, in the second embodiment, a camera image to be highlighted is selected based on the speed information of the moving object so that the operator can be prompted to pay attention to a camera image in front in the same direction as the indicated direction while moving at a low speed, and to pay attention to a camera image in the side and rear in the same direction as the indicated direction while moving at a high speed.

[0050] FIG. 8 is a block diagram showing a configuration example of a remote driving system according to the second embodiment. The difference from the first embodiment is that in the left camera image 75 and the right camera image 85 included in the camera image information 60 recorded in the storage device, the left camera image 75 further includes a front left front camera image 80 and a left side camera image 100, the right camera image 85 further includes a front right front camera image 90 and a right side camera image 110, and speed information 140 is further recorded in the storage device. The front left front camera image 80 and the front right front camera image 90 are images acquired from multiple cameras mounted on the moving body to view the situation to the left and right in front of the moving body. The left side camera image 100 and the right side camera image 110 are images acquired from multiple cameras mounted on the moving body to view the situation to the side and rear of the moving body. The speed information 140 is vehicle speed information of the moving body.

[0051] 9 is a block diagram showing an example of the functions of a remote driving system according to the second embodiment. The difference from the first embodiment is that a front left front camera image 80, a front right front camera image 90, a left side camera image 100, and a right side camera image 110 included in the camera image information 60 are input to a symbol superimposition unit 430, and speed information 140 is input to an emphasis display image selection unit 420.

[0052] The highlight image selection unit 420 selects an image to be highlighted from among camera images including the front left front camera image 80, the front right front camera image 90, the left side camera image 100, and the right side camera image 110, based on the input turn signal indication direction information 130 and speed information 140. Then, highlight selection information 520 indicating information on the selected image is generated. The processing of the highlight image selection unit 420 will be described in detail later.

[0053] The symbol superimposition unit 430 generates a symbol superimposition image 530 in which a highlighting item is superimposed on an image selected for highlighting from among camera images including a front left front camera image 80, a front right front camera image 90, a left side camera image 100, and a right side camera image 110, based on the highlighting selection information 520 generated by the highlighting image selection unit 420. Note that in the highlighting selection information 520, the image of the left camera image 75 or the right camera image 85 in a direction different from the indication direction of the winker is not subjected to symbol superimposition, and is therefore generated as an image without symbol superimposition 540.

[0054] FIG. 10 is a flowchart showing the process performed by the highlighted image selection unit 420 of the remote driving system according to the second embodiment.

[0055] In step S200, the highlighted display image selection unit 420 determines whether the input turn signal indication direction information 130 includes information on the left or right direction of the turn signal indication direction.

[0056] If it is determined that the information on the left or right direction of the blinker is included (step S200; Yes), the process proceeds to step S210. Otherwise (step S200; No), the process proceeds to step S240.

[0057] In step S210, the highlighted image selection unit 420 determines whether the speed of the input speed information 140 is equal to or greater than a predetermined speed.

[0058] If it is determined that the speed is equal to or greater than the predetermined speed (step S210; Yes), the process proceeds to step S220. If the speed is less than the predetermined speed (step S210; No), the process proceeds to step S230.

[0059] In step S220, the highlight image selection unit 420 selects the left side camera image 100 or the right side camera image 110 in the same direction as the turn signal indication direction as the image to be highlighted, and generates highlight selection information 520 indicating information about the selected image.

[0060] In step S230, the highlight image selection unit 420 selects the front left front camera image 80 or the front right front camera image 90 in the same direction as the turn signal indication direction as the image to be highlighted, and generates highlight selection information 520 indicating information about the selected image.

[0061] In step S240, the highlight image selection unit 420 selects no image as an image to be highlighted, and generates highlight selection information 520 indicating information about the selected image.

[0062] In step S 250 , the emphasis image selection unit 420 outputs the emphasis selection information 520 to the display processing unit 440 .

[0063] FIG. 11 is a display example of an image displayed on the display device 30 according to the second embodiment. A front camera image 70 of the center of the front is displayed on the display device 30 in the front direction. Also, an example is given in which a symbol superimposed image 530 in the same direction as the indication direction of the turn signal is displayed on the display device 30 displaying a camera image in the same direction as the indication direction, and a symbol superimposed non-executed image 540 of a camera image in a direction different from the indication direction of the turn signal is displayed on the display device 30 displaying a camera image in a direction different from the indication direction. Specifically, when the camera image selected to be highlighted in the highlight image selection unit 420 is the left side camera image 100, a symbol superimposed image 530 in which a symbol 550 is superimposed on the left side camera image 100 is displayed on the display device 30 in the left direction. Then, a symbol superimposed non-executed image 540 of either the front right front camera image 90 or the right side camera image 110 is displayed on the display device 30 in the right direction. When the camera image selected to be highlighted is the front left front camera image 80, a symbol superimposed image 530 in which a symbol 550 is superimposed on the front left front camera image 80 is displayed on the left display device 30. Then, a symbol non-superimposed image 540 of either the front right front camera image 90 or the right side camera image 110 is displayed on the right display device 30.

[0064] As an example of displaying the symbol superimposed image 530 and the symbol non-superimposed image 540 on the display device 30, each image may be displayed using the display device 30 as shown in Fig. 12. Also, the symbol superimposed image 530 and the symbol non-superimposed image 540 in the same direction as the indication direction of the turn signal may be displayed on a display device 30 that displays at least one camera image in the same direction as the indication direction, and the symbol non-superimposed image 540 in a direction different from the indication direction of the turn signal may be displayed on a display device 30 that displays at least one camera image in a direction different from the indication direction.

[0065] As described above, according to the second embodiment, the camera image is selected based on the comparison result between the speed of the speed information and the predetermined speed, so that, for example, the operator can be prompted to pay attention to the front camera image in the same direction as the indicated direction during low-speed movement, and the operator can be prompted to pay attention to the side camera image in the same direction as the indicated direction during high-speed movement. This leads to improved operability of the moving body by remote driving and ensuring the traveling safety of the moving body.

[0066] 3. Third embodiment The third embodiment is based on the same configuration example as the second embodiment, and is a modified example of an image displayed on the display device 30. Descriptions that overlap with the previous embodiments will be omitted as appropriate.

[0067] 13 is a block diagram showing an example of the functions of a remote driving system according to the third embodiment. The difference from the second embodiment is the addition of an image superimposition unit 410. The image superimposition unit 410 generates a left superimposed image 500 in which the left side camera image 100 is superimposed on the front left front camera image 80, and a right superimposed image 510 in which the right side camera image 110 is superimposed on the front right front camera image 90, based on the input front left front camera image 80, front right front camera image 90, left side camera image 100, and right side camera image 110.

[0068] 14 is a display example of a left superimposed image 500 and a right superimposed image 510 generated by the image superimposing unit 410. Specifically, the left superimposed image 500 is obtained by superimposing a left side camera image 100 at a predetermined position on a front left front camera image 80, and the right superimposed image 510 is obtained by superimposing a right side camera image 110 at a predetermined position on a front right front camera image 90.

[0069] FIG. 15 is a display example of an image displayed on the display device 30 according to the third embodiment. The front camera image 70 of the center of the front is displayed on the display device 30 in the front direction. Also, an example is given in which the symbol superimposed image 530 in the same direction as the indication direction of the turn signal is displayed on the display device 30 displaying one of the left superimposed image 500 or the right superimposed image 510 in the same direction as the indication direction, and the symbol superimposed non-executed image 540 in a direction different from the indication direction of the turn signal is displayed on the display device 30 displaying the superimposed image in a direction different from the indication direction. Specifically, when the camera image selected to be highlighted in the highlighting image selection unit 420 is the left side camera image 100, the symbol superimposed image 530 in which the symbol 550 is superimposed on the side camera image 100 to the left of the left superimposed image 500 is displayed on the display device 30 in the left direction. Then, the right superimposed image 510 is displayed as the symbol superimposed non-executed image 540 on the display device 30 in the right direction. When the camera image selected for highlighting in the highlight image selection unit 420 is the front left front camera image 80, a symbol superimposed image 530 in which a symbol 550 is superimposed on the front left front camera image 80 of the left superimposed image 500 is displayed on the left display device 30. Then, the right superimposed image 510 is displayed on the right display device 30 as an image 540 without symbol superimposition.

[0070] According to the third embodiment, the same effect as the second embodiment can be obtained by highlighting one of the front camera image or the side camera image in the superimposed image in the same direction as the blinker indication direction. Also, the front camera image and the side camera image in the direction corresponding to the indication direction are displayed as one superimposed image on the display device corresponding to the indication direction, so that the operator can shift his / her gaze to the highlighted image without feeling uncomfortable. This leads to improved operability of the mobile body by remote driving and ensuring the driving safety of the mobile body.

[0071] According to the third embodiment, the side camera image corresponding to the designated direction is superimposed on the front camera image corresponding to the designated direction at a position close to the display device in the front direction, so that the operator can minimize the movement of the line of sight with respect to the highlighted side camera image. Therefore, the operator can instantly grasp the surrounding situation of the moving object shown in the front camera image or the side camera image in the same direction as the designated direction. This further improves the operability of the moving object by remote driving and leads to increased driving safety of the moving object. [Explanation of symbols]

[0072] 10 Remote Operation System 20 Information processing device 30 Display device 40 processors 50 Storage device 60 Camera Image Information 70 Front camera image in the center 75 Left camera image 80 Front left front camera image 85 Right camera image 90 Front right front camera image 100 Left side camera image 110 Right side camera image 130 Turn signal direction information 140 Speed ​​information 400 Information input section 410 Image Overlay Unit 420 Highlighted Image Selection Section 430 Symbol Overlay Section 440 Display Processing Unit 500 Left Overlay Image 510 Right Overlay Image 520 Highlight Selection Information 530 Symbol Overlay Image 540 Images without symbol overlay 550 Symbols

Claims

1. A remote driving system for remotely controlling the operation of a moving object, comprising: one or more storage devices in which camera image information including a camera image of the center of the front of the moving body, a camera image of the left side including the left front of the moving body, and a camera image of the right side including the right front of the moving body, information on the indication direction of a turn signal of the moving body, and speed information of the moving body are recorded; one or more processors; At least three display devices for displaying the front center, left, and right camera images, respectively; Equipped with The left camera image includes a front left front camera image and a left side camera image, The right camera image includes a front right front camera image and a right side camera image, The one or more processors: determining whether the turn signal is being operated based on the information on the indicated direction; When it is determined that the turn signal is being operated, When the speed information indicates a speed equal to or greater than a predetermined speed, a camera image in the same direction as the designated direction is selected from among the left and right side camera images as an image to be highlighted; When the speed information indicates a speed less than a predetermined speed, a camera image in the same direction as the designated direction is selected from the front left and front right front camera images as an image to be highlighted; generating a symbol superimposed image by superimposing a symbol on the selected image; The symbol superimposed image is displayed on one of the at least three display devices that displays a camera image in the same direction as the indicated direction. Remote driving system.

2. 2. The remote driving system according to claim 1, The one or more processors further include: generating a left superimposed image by superimposing the left side camera image on the front left front camera image, and a right superimposed image by superimposing the right side camera image on the front right front camera image; An image to be highlighted is selected from the left superimposed image and the right superimposed image. Remote driving system.

3. A remote operation program for remotely controlling the operation of a moving object by a computer, A process of acquiring camera image information including a camera image of a center of a front of the moving body, a camera image of a left side including a front left of the moving body, and a camera image of a right side including a front right of the moving body, information on an indication direction of a turn signal of the moving body, and speed information of the moving body; A process of determining whether or not the turn signal is being operated based on the information on the indicated direction; a process of selecting, when it is determined that the turn signal is being operated and the speed information is equal to or greater than a predetermined speed, a camera image in the same direction as the indicated direction from among a left side camera image included in the left camera image and a right side camera image included in the right camera image, as an image to be highlighted; a process of determining that the turn signal is being operated and, when the speed information indicates a speed less than a predetermined speed, selecting, as an image to be highlighted, a camera image in the same direction as the indicated direction from among a front left front camera image included in the left camera image and a front right front camera image included in the right camera image; A process of generating a symbol superimposed image by superimposing a symbol on the selected image; a process of displaying the symbol superimposed image on a display device that displays a camera image in the same direction as the indicated direction among at least three display devices that respectively display the front center, left, and right camera images; by the computer. Remote driving program.

4. The remote operation program according to claim 3, and causing the computer to further execute a process of generating a left superimposed image in which the left side camera image is superimposed on the front left front camera image, and a right superimposed image in which the right side camera image is superimposed on the front right front camera image, The process of selecting the image to be highlighted is performed based on the left superimposed image and the right superimposed image. Remote driving program.

5. A remote operation method for remotely controlling an operation of a moving object, comprising: A process of acquiring camera image information including a camera image of a center of a front of the moving body, a camera image of a left side including a front left of the moving body, and a camera image of a right side including a front right of the moving body, information on an indication direction of a turn signal of the moving body, and speed information of the moving body; A process of determining whether or not the turn signal is being operated based on the information on the indicated direction; a process of selecting, when it is determined that the turn signal is being operated and the speed information is equal to or greater than a predetermined speed, a camera image in the same direction as the indicated direction from among a left side camera image included in the left camera image and a right side camera image included in the right camera image, as an image to be highlighted; a process of determining that the turn signal is being operated and, when the speed information indicates a speed less than a predetermined speed, selecting, as an image to be highlighted, a camera image in the same direction as the indicated direction from among a front left front camera image included in the left camera image and a front right front camera image included in the right camera image; A process of generating a symbol superimposed image by superimposing a symbol on the selected image; a process of displaying the symbol superimposed image on a display device that displays a camera image in the same direction as the indicated direction among at least three display devices that respectively display the front center, left, and right camera images; Includes Remote operation method.

6. The remote driving method according to claim 5, The method further includes a process of generating a left superimposed image in which the left side camera image is superimposed on the front left front camera image, and a right superimposed image in which the right side camera image is superimposed on the front right front camera image, The process of selecting the image to be highlighted is performed based on the left superimposed image and the right superimposed image. Remote operation method.

Images