Remote operation system, program and method

The system adjusts sub-camera image transmittance based on driving conditions and audio inputs to address blind spots in remote operation systems, ensuring safer remote vehicle or robot operation.

JP7578082B2Active Publication Date: 2024-11-06TOYOTA JIDOSHA KK
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2021146252
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-09-08
Publication Date
2024-11-06
Estimated Expiration
2041-09-08

AI Technical Summary

Technical Problem

Existing remote operation systems for vehicles or robots face issues with sub-camera images superimposed on main camera images creating blind spots, particularly during turns or changes in vehicle conditions, leading to reduced situational awareness and safety.

Method used

A system that adjusts the transmittance of sub-camera images based on driving conditions, speed, turn signal direction, and audio inputs to optimize the superimposed display on the main camera image, ensuring critical views are visible when needed.

Benefits of technology

Enhances the operator's ability to accurately perceive the surroundings, thereby improving safety during remote operation by minimizing blind spots and enhancing situational awareness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007578082000001
    Figure 0007578082000001
  • Figure 0007578082000002
    Figure 0007578082000002
  • Figure 0007578082000003
    Figure 0007578082000003
Patent Text Reader

Abstract

To change a display mode of a sub camera image which is displayed while being superposed on a part of an image of a main camera in accordance with a travel situation of a mobile when performing remote driving of the mobile.SOLUTION: A remote driving system comprises a storage device, one or more processors and a display device. The storage device stores therein camera image information of a mobile, travel situation information of the mobile and voice information of an occupant. The camera image information includes a main camera image and a sub camera image. The one or more processors generate a transmissivity corrected image, which is corrected by controlling transmissivity of the sub camera image, based on at least one of the travel situation information and the voice information. The one or more processors also generate a superpose image which displays the transmissivity corrected image while superposing it on the main camera image. The one or more processors further display the superpose image on the display device.SELECTED DRAWING: Figure 8
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a technology for remotely controlling the operation of a moving object, and in particular to a technology for remotely controlling the operation of a moving object by controlling the transmittance of an image superimposed on a part of an image captured by a plurality of cameras mounted on the moving object. [Background technology]

[0002] Patent Document 1 discloses a technology related to display control for displaying a superimposed item on an image. In this technology, the transparency of the superimposed item is increased as the magnification ratio of the image increases, thereby making it easier to see the underlying image that is obscured by the superimposed item. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-109662 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, in order to remotely support the operation of a moving body such as a vehicle or a robot, an image captured by a camera mounted on the moving body is useful. The image captured by the camera is recorded in a recording device in an information processing device of the remote driving system. A remote operator looks at the image displayed on the display device of the remote driving system, grasps the situation around the moving body, and remotely drives the operation of the moving body. Here, the image displayed on the display device refers to an image in which a sub-camera image captured to see the side, rear, etc. is superimposed on a main camera image captured to see the front.

[0005] If the sub camera image is always displayed superimposed on a part of the main camera image, the part of the superimposed main camera image becomes a blind spot, which causes the following problem. For example, if the sub camera image taken to see the side is always superimposed on the lower part of the main camera image, it is not possible to check the vicinity of the vehicle when turning left at an intersection, etc. Therefore, an improvement is desired to change the manner in which the sub camera image is superimposed depending on the traveling situation of the moving object.

[0006] One object of the present invention is to provide a technology that, when remotely driving a moving body, can change the display mode of a sub-camera image that is superimposed on a portion of the image of the main camera depending on the driving conditions of the moving body. [Means for solving the problem]

[0007] The first aspect relates to a remote driving system for remotely controlling the operation of a moving object. The remote operation system is One or more storage devices in which camera image information captured by a plurality of cameras mounted on the moving body, and at least one of driving situation information indicating the driving situation of the moving body and voice information of the occupants of the moving body are recorded; one or more processors; Display device and Equipped with. The camera image information includes a main camera image and a sub camera image. The one or more processors generate a transmittance corrected image that is corrected by controlling the transmittance of the sub camera image based on at least one of the driving situation information and the audio information. The one or more processors generate a superimposed image that displays the transmittance corrected image superimposed on the main camera image. The one or more processors then display the overlaid image on a display device.

[0008] The second aspect has the same features as the first aspect, as well as the following features. The sub camera image includes a side camera image and a rear camera image, and the driving condition information includes speed information. In controlling the transmittance of the sub-camera image, the one or more processors set the transmittance of the side camera image or the rear camera image to be increased linearly, nonlinearly, or stepwise as the speed of the speed information decreases from the current value. Then, the one or more processors, in controlling the transmittance of the sub-camera image, set the transmittance of the side camera image or the rear camera image to be lowered linearly, nonlinearly, or stepwise as the speed of the speed information increases from the current value.

[0009] The third aspect has the following features in addition to those of the first aspect. The sub camera image includes a side camera image, and the driving condition information includes information on the turn signal indication direction. The one or more processors further set the transmittance of the side camera image in the same direction as the information on the turn signal indication direction to be high.

[0010] A fourth aspect has the same features as any one of the first to third aspects, and further includes the following features. The sub-camera images include indoor camera images. The one or more processors determine whether or not the voice information includes a voice calling out from an occupant when controlling the transmittance of the sub camera image. Then, when the one or more processors determine that the voice information includes a voice calling out from an occupant, they set the transmittance of the interior camera image to be low.

[0011] The fifth aspect relates to a remote driving program for remotely controlling the operation of a mobile object. The remote operation program is A process of acquiring camera image information including a main camera image and a sub camera image captured by a plurality of cameras mounted on the moving body, and at least one of driving situation information indicating a driving situation of the moving body and voice information of a passenger of the moving body; A process of generating a transmittance corrected image corrected by controlling the transmittance of the sub camera image based on at least one of the driving situation information and the audio information; A process of generating a superimposed image in which the transmittance corrected image is superimposed on the main camera image; A process of displaying the superimposed image on a display device. is executed by a computer.

[0012] A sixth aspect relates to a remote driving method for remotely controlling the operation of a moving object. The remote operation method is A process of acquiring camera image information including a main camera image and a sub camera image captured by a plurality of cameras mounted on the moving body, and at least one of driving situation information indicating a driving situation of the moving body and voice information of a passenger of the moving body; A process of generating a transmittance corrected image corrected by controlling the transmittance of the sub camera image based on at least one of the driving situation information and the audio information; A process of generating a superimposed image in which the transmittance corrected image is superimposed on the main camera image; A process of displaying the superimposed image on a display device. Includes. Effect of the Invention

[0013] According to the first aspect, the transmittance of the sub camera image is controlled based on at least one of driving condition information and audio information. Furthermore, a superimposed image is generated by superimposing a transmittance corrected image corrected by controlling the transmittance of the sub camera image on a part of the main camera image. The superimposed image is then displayed on the display device. This makes it easier for the remote operator to accurately grasp the situation around the moving object based on at least one of the driving condition and the audio of the occupant. This leads to ensuring the safety of the moving object when remotely driven.

[0014] According to the second aspect, the transmittance of the side camera image or the rear camera image is controlled according to the speed of the speed information. This allows the remote operator to more accurately grasp the situation around the moving object. This leads to ensuring the safety of the moving object when remotely operated.

[0015] According to the third aspect, the transmittance of the side camera image is set based on the information on the turn signal direction. This allows the remote operator to more accurately grasp the situation around the moving object. This leads to ensuring the safety of the moving object when remotely driven.

[0016] According to the fourth aspect, the transmittance of the interior camera image is set to be low when the voice information includes a voice call from the occupant. This makes it easier for the remote operator to accurately grasp whether any abnormality has occurred inside the vehicle. This leads to ensuring the safety of the vehicle being remotely operated.

[0017] According to the fifth aspect, the same effects as those of the first aspect can be obtained.

[0018] According to the sixth aspect, the same effects as those of the first aspect can be obtained. [Brief description of the drawings]

[0019] [Figure 1] 1 is a block diagram showing a configuration example of a remote operation system according to a first embodiment of the present invention. [Diagram 2] 2 is a block diagram showing various information recorded in a storage device of the remote operation system according to the first embodiment of the present invention. FIG. [Diagram 3] 1 is a block diagram showing an example of functions of a remote operation system according to a first embodiment of the present invention; [Figure 4] 5 is a graph showing processing by a transmittance control unit based on speed information of the remote operation system according to the first embodiment of the present invention. [Diagram 5]5 is a flowchart showing a process performed by a transmittance control unit based on turn signal information of the remote driving system according to the first embodiment of the present invention. [Figure 6] 5 is a flowchart showing a process performed by a transmittance control unit based on voice information of the remote operation system according to the first embodiment of the present invention. [Figure 7] 3A to 3C are diagrams illustrating an example of a processing result by an image superimposition unit of the remote driving system according to the first embodiment of the present invention. [Figure 8] 4A to 4C are diagrams showing examples of display on a display device when processing of a transmittance control unit is carried out based on speed information of the remote operation system according to the first embodiment of the present invention. [Figure 9] 4A to 4C are diagrams showing examples of display on a display device when processing by a transmittance control unit is carried out based on turn signal information of the remote driving system according to the first embodiment of the present invention. [Figure 10] 4A to 4C are diagrams showing examples of display on a display device when processing by a transmittance control unit is carried out based on voice information of the remote operation system according to the first embodiment of the present invention. [Figure 11] FIG. 11 is a block diagram showing an example of the configuration of a remote operation system according to a second embodiment of the present invention. [Figure 12] FIG. 11 is a block diagram showing an example of functions of a remote operation system according to a second embodiment of the present invention. [Figure 13] 10 is a flowchart showing a process performed by a transmittance control unit based on operator operation information of a remote operation system according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A remote operation system, a remote operation program, and a remote operation method according to embodiments of the present invention will be described with reference to the accompanying drawings.

[0021] 1. First embodiment 1-1. Configuration example The remote driving system according to this embodiment is for remotely controlling the operation of a moving object. Fig. 1 is a block diagram showing a configuration example of a remote driving system 10 according to this embodiment. The remote driving system 10 includes an information processing device 20 and a display device 30.

[0022] The information processing device 20 performs various information processing. The information processing device 20 includes one or more processors 40 (hereinafter simply referred to as processor 40) and one or more storage devices 50 (hereinafter simply referred to as storage devices 50). The processor 40 executes various processes. For example, the processor 40 includes a CPU. The storage device 50 stores various information such as camera image information 60, driving situation information 70, and sound information 80. Examples of the storage device 50 include a volatile memory, a non-volatile memory, a HDD, and an SSD. The processor 40 executes a remote driving program, which is a computer program, to realize the functions of the information processing device 20. The remote driving program is stored in the storage device 50. The remote driving program may be recorded in a computer-readable recording medium. The remote driving program may be provided via a network.

[0023] The display device 30 displays various information. Examples of the display device 30 include a liquid crystal display, an organic EL display, a head mounted display, and a touch panel.

[0024] FIG. 2 is a block diagram showing details of various types of information, including camera image information 60, driving situation information 70, and sound information 80, recorded in a storage device.

[0025] The camera image information 60 includes a main camera image 100 and a sub camera image 200. Furthermore, the main camera image 100 includes a left main camera image 110, a front main camera image 120, and a right main camera image 130, and the sub camera image 200 includes a side camera image 210 (the side camera image 210 includes a left side camera image 211 and a right side camera image 212), a rear camera image 220, and an interior camera image 230. Here, the main camera image 100 refers to an image acquired from multiple cameras mounted on a moving body to view the situation in front of the moving body. The sub camera image 200 refers to an image acquired from multiple cameras mounted on a moving body to view the situation to the side, rear, and interior of the moving body.

[0026] The driving condition information 70 includes speed information 300 and turn signal direction information 310. The speed information 300 is vehicle speed information of the moving object, and the turn signal direction information 310 is information on the direction (left or right) in which the moving object should move, which is input to the turn signal when an operator (remote operator) who remotely drives the moving object operates the turn signal.

[0027] The sound information 80 is information about sounds inside and outside the vehicle. The sound information 80 includes voice information 320 of the vehicle's occupants (i.e., the driver of the vehicle and passengers other than the driver). The voice information 320 includes voices of conversations between occupants, voices made by the occupants regarding indoor and outdoor conditions, voices made by the occupants to call out to an operator, and the like.

[0028] 1-2. Details of information processing The image displayed on the display device 30 is an image in which the sub camera image 200 is superimposed on a part of the main camera image 100. When the sub camera image 200 is superimposed on a part of the main camera image 100, the part of the main camera image 100 becomes a blind spot. Turning left at an intersection in a blind spot state reduces the safety of a mobile object being remotely driven. Therefore, the sub camera image 200 superimposed on a part of the main camera image 100 is corrected according to the traveling conditions of the mobile object so that the part of the main camera image 100 can be seen. The information processing according to this embodiment includes characteristic processing as described below.

[0029] 3 is a block diagram showing an example of functions related to information processing according to this embodiment. The information processing device 20 of the remote driving system 10 includes, as functional blocks, an information input unit 400, an image transmittance control unit 410, an image superimposition unit 420, and a display processing unit 430. These functional blocks are realized by the processor 40 executing a remote driving program.

[0030] The information input unit 400 performs a process of inputting various pieces of information recorded in the storage device 50. Of the various pieces of information, the sub camera image 200, the speed information 300, the turn signal direction information 310, and the audio information 320 are input to the image transmittance control unit 410, and the main camera image 100 is input to the image superimposition unit 420.

[0031] The image transmittance control unit 410 generates a transmittance corrected image 500 of the sub camera image 200 by correcting the input sub camera image 200 by controlling the transmittance based on at least one of the input speed information 300, the information 310 of the turn signal direction, and the audio information 320. The process of the image transmittance control unit 410 based on the various information of the speed information 300, the information 310 of the turn signal direction, and the audio information 320 will be described in detail later.

[0032] Image superimposition section 420 generates superimposed image 510 by superimposing transmittance corrected image 500 generated by image transmittance control section 410 on a part of main camera image 100 .

[0033] The display processing unit 430 performs processing for outputting the superimposed image 510 generated by the image superimposing unit 420 to the display device 30.

[0034] 4 is a graph showing an example of processing by the image transmittance control unit 410 for the side camera image 210 and the rear camera image 220 of the sub camera image 200 based on the speed information 300. As the speed of the speed information 300 decreases from the current value, the transmittance of the side camera image 210 and the rear camera image 220 is set to be higher linearly, nonlinearly, or stepwise, and as the speed of the speed information 300 increases from the current value, the transmittance of the side camera image 210 and the rear camera image 220 is set to be lower linearly, nonlinearly, or stepwise.

[0035] A transmittance-corrected image 500 of the side camera image 210 and the rear camera image 220 corrected according to the transmittance setting is generated, and the transmittance-corrected image 500 is output to the image superimposition unit 420.

[0036] FIG. 5 is a flowchart showing the process of the image transmittance control unit 410 for the side camera image 210 of the sub camera image 200 based on the information 310 on the turn signal indication direction.

[0037] In step S100, image transmittance control section 410 determines whether or not input turn signal direction information 310 includes information on the left or right turn signal direction.

[0038] If it is determined that the information on the left or right turn signal direction is included (step S100; Yes), the process proceeds to step S110. Otherwise (step S100; No), the process proceeds to step S120.

[0039] In step S110, image transmittance control unit 410 sets the transmittance of side camera image 210 in the same direction as turn signal indication direction information 310 to be high for input side camera image 210, and generates transmittance-corrected image 500 corrected by the transmittance setting. Sets the transmittance of side camera image 210 in a direction different from turn signal indication direction information 310 to be low, and generates transmittance-corrected image 500 corrected by the transmittance setting.

[0040] In step S120, image transmittance control section 410 sets the transmittance of input side camera image 210 to be low, and generates transmittance corrected image 500 corrected according to the transmittance setting.

[0041] In step S 130 , the image transmittance control unit 410 outputs the transmittance corrected image 500 to the display processing unit 430 .

[0042] FIG. 6 is a flowchart showing the process of image transmittance control section 410 for indoor camera image 230 based on audio information 320. As shown in FIG.

[0043] In S200, the image transmittance control unit 410 determines, based on the input audio information 320, whether or not the audio information 320 is included.

[0044] If it is determined that the audio information 320 is included (step S200; Yes), the process proceeds to step S210. Otherwise (step S200; No), the process proceeds to step S230.

[0045] In step S210, the image transmittance control unit 410 determines whether or not the voice information 320 includes a voice calling out from the occupant.

[0046] If it is determined that the voice information 320 includes a voice calling from the occupant (step S210; Yes), the process proceeds to step S220. Otherwise (step S210; No), the process proceeds to step S230.

[0047] In step S220, image transmittance control section 410 sets the transmittance of input indoor camera image 230 to be low, and generates transmittance corrected image 500 corrected according to the transmittance setting.

[0048] In step S230, image transmittance control section 410 sets a high transmittance for input indoor camera image 230. A transmittance corrected image 500 that is corrected according to the transmittance setting is generated.

[0049] When the moving object is stopped, in order to be able to grasp the situation of the indoor camera image 230, the process of step S230 is performed regardless of the voice information 320. When the moving object is traveling, except when it is determined that the voice information 320 includes a voice calling out from the occupant (step S210; Yes), it is not necessary to grasp the situation of the indoor camera image 230, so as the speed of the speed information 300 increases, the transmittance of the indoor camera image 230 is set to be higher, so that the indoor camera image 230 becomes invisible.

[0050] In step S 240 , the image transmittance control unit 410 outputs the transmittance corrected image 500 to the display processing unit 430 .

[0051] 1-3. Example of superimposed image display 7 is a display example of superimposed image 510 generated by image superimposition unit 420. Superimposed image 510 is obtained by superimposing transmittance-corrected image 500 generated by image transmittance control unit 410 at a predetermined position on main camera image 100. Specifically, transmittance-corrected image 500 of left side camera image 211 is superimposed on left main camera image 110, transmittance-corrected images 500 of rear camera image 220 and interior camera image 230 are superimposed on front main camera image 120, and transmittance-corrected image 500 of right side camera image 212 is superimposed on right main camera image 130.

[0052] FIG. 8 is a display example when the transmittance of the side camera image 210 and the rear camera image 220 of the sub camera image 200 is controlled based on the speed information 300. Specifically, the display when the transmittance of the left side camera image 211 and the rear camera image 220 of the side camera image 210 is controlled is shown as an example. As the speed of the speed information 300 becomes smaller, the transmittance of the left side camera image 211 and the rear camera image 220 is set to be higher, so that the part superimposed and displayed in the left main camera image 110 becomes visible. On the other hand, as the speed of the speed information 300 becomes larger, the transmittance of the left side camera image 211 and the rear camera image 220 is set to be lower, so that the left side camera image 211 and the rear camera image 220 become visible.

[0053] 9 is a display example when the transmittance of the side camera image 210 of the sub camera image 200 is controlled based on the information 310 of the turn signal direction. When the information 310 of the turn signal direction is present, the transmittance of the side camera image 210 (left side camera image 211 or right side camera image 212) in the same direction as the turn signal direction is set to be high, so that the main camera image 100 (left main camera image 110 or right main camera image 130) in the same direction as the turn signal direction becomes visible. On the other hand, when the information 310 of the turn signal direction is not present, the transmittance of the side camera image 210 is set to be low, so that the side camera image 210 (left side camera image 211 or right side camera image 212) becomes visible.

[0054] 10 is a display example when the transmittance of the indoor camera image 230 is controlled based on the audio information 320. When the audio information 320 is present and includes a voice calling out from the occupant, the transmittance of the indoor camera image 230 is set to be low, so that the indoor camera image 230 becomes visible. On the other hand, when the audio information 320 is present and does not include a voice calling out from the occupant, or when the audio information 320 does not exist, the transmittance of the indoor camera image 230 is set to be high, so that the main camera image 120 in front of the main camera image 100 becomes visible.

[0055] When the moving object is stopped, the transmittance is set low regardless of the audio information 320 so that the condition of the indoor camera image 230 can be grasped, and the indoor camera image 230 is made visible. However, when the moving object is moving, it is not necessary to grasp the condition of the indoor camera image 230 except when the information includes a voice call from the occupant. Therefore, as the speed of the speed information 300 increases, the transmittance of the indoor camera image 230 is set to increase so that the indoor camera image 230 becomes invisible.

[0056] 1-4.Effects When image transmittance control is performed during remote driving, the operator can easily grasp the situation around the moving object accurately. This leads to ensuring the safety of the moving object when remotely driven. Thus, according to the present embodiment, even when image transmittance control is performed during remote driving, it leads to ensuring the safety of the moving object when remotely driven.

[0057] 2. Second embodiment The second embodiment differs from the first embodiment in the details of information processing. The basic configuration of the remote driving system 10 is the same as that of the first embodiment. Descriptions that overlap with the first embodiment will be omitted as appropriate.

[0058] In the second embodiment, attention is focused on the image transmittance control unit 410 of the side camera image 210 based on the information 310 of the turn signal direction. When the information 310 of the turn signal direction to be input includes information on the left or right direction of the turn signal direction, the transmittance of the side camera image 210 in the same direction as the turn signal direction becomes high, so that the side camera image 210 becomes difficult to see. In other words, it becomes difficult to accurately grasp the situation on the side of the moving object. As a result, the safety of the moving object by remote driving is reduced, such as an entrapment accident when turning left at an intersection. Therefore, in the second embodiment, in the details of the information processing, the transmittance of the side camera image 210 can be corrected by the operation of the operator.

[0059] 11 is a block diagram showing a configuration example of a remote operation system 10 according to the second embodiment. The remote operation system 10 includes an information processing device 20, a display device 30, and an input device 90. The input device 90 is an interface for receiving input from a remote operator. Examples of the input device 90 include a touch panel, a keyboard, and a mouse. The information processing device 20 and the display device 30 are the same as those in the first embodiment.

[0060] 2-1. Configuration example 12 is a block diagram showing an example of functions related to information processing according to the second embodiment. The information processing device 20 of the remote driving system 10 includes, as functional blocks, an information input unit 400, an image transmittance control unit 410, an image superimposition unit 420, and a display processing unit 430. These functional blocks are realized by the processor 40 executing a remote driving program. The image superimposition unit 420 and the display processing unit 430 are the same as those in the first embodiment.

[0061] The information input unit 400 performs a process of inputting various pieces of information recorded in the storage device 50. Of the various pieces of information, the sub camera image 200, the speed information 300, the turn signal direction information 310, the voice information 320, and the operator operation information 330 are input to the image transmittance control unit 410, and the main camera image 100 is input to the image superimposition unit 420.

[0062] The image transmittance control unit 410 generates a transmittance corrected image 500 of the sub camera image 200 by correcting the input sub camera image 200 by controlling the transmittance based on at least one of the input speed information 300, the information 310 of the turn signal direction, the voice information 320, and the operator operation information 330. The process of the image transmittance control unit 410 based on the various information of the speed information 300, the information 310 of the turn signal direction, the voice information 320, and the operator operation information 330 will be described in detail later.

[0063] 2-2.Examples of information processing FIG. 13 is a flowchart showing a process of the image transmittance control unit 410 for the side camera image 210 of the sub camera image 200 based on the turn signal direction information 310 and the operator operation information 330 according to the second embodiment.

[0064] In step S300, image transmittance control section 410 determines whether or not input turn signal direction information 310 includes information on the left or right turn signal direction.

[0065] If it is determined that the information on the left or right turn signal direction is included (step S300; Yes), the process proceeds to step S310. Otherwise (step S300; No), the process proceeds to step S340.

[0066] In step S310, the image transmittance control unit 410 determines whether the input operator operation information 330 includes information on the transmittance specified by the operator.

[0067] If it is determined that the transmittance information specified by the operator is included (step S310; Yes), the process proceeds to step S320. Otherwise (step S310; No), the process proceeds to step S330.

[0068] In step S320, the image transmittance control unit 410 sets the transmittance of the side camera image 210 in the same direction as the turn signal direction information 310 to a value based on the transmittance specified by the operator, for the input side camera image 210, and generates a transmittance corrected image 500 that is corrected by the transmittance setting.

[0069] In step S330, image transmittance control unit 410 sets the transmittance of side camera image 210 in the same direction as turn signal indication direction information 310 to be high for input side camera image 210, and generates transmittance-corrected image 500 corrected by the transmittance setting. Sets the transmittance of side camera image 210 in a direction different from turn signal indication direction information 310 to be low, and generates transmittance-corrected image 500 corrected by the transmittance setting.

[0070] In step S340, image transmittance control section 410 sets the transmittance of input side camera image 210 to be low, and generates transmittance corrected image 500 corrected according to the transmittance setting.

[0071] In step S350, the image transmittance control unit 410 outputs the transmittance corrected image 500 to the display processing unit 430.

[0072] As described above, according to the second embodiment, even if there is information 310 about the turn signal direction, the transmittance of the side camera image 210 can be corrected by the operator. This makes it easier for the operator to accurately grasp the situation on the sides of the moving object. This leads to ensuring the safety of the moving object when remotely driven. [Explanation of symbols]

[0073] 10 Remote Operation System 20 Information processing device 30 Display device 40 processors 50 Storage device 60 Camera Image Information 70 Driving status information 80 Sound Information 90 Input Devices 100 Main Camera Images 110 Left main camera image 120 Front main camera images 130 Right main camera image 200 sub camera images 210 Side camera images 211 Left side camera image 212 Right side camera image 220 Rear camera image 230 Indoor Camera Images 300 Speed ​​information 310 Turn signal direction information 320 Audio Information 330 Operator Operation Information 400 Information input section 410 Image transmittance control section 420 Image Overlay Unit 430 Display Processing Unit 500 Transparency Corrected Images 510 Overlay Images

Claims

1. A remote driving system for remotely controlling the operation of a moving object, comprising: one or more storage devices in which camera image information captured by a plurality of cameras mounted on the moving body, and at least one of driving status information indicating a driving status of the moving body and voice information of a passenger of the moving body are recorded; one or more processors; Display device and Equipped with The camera image information includes a main camera image and a sub camera image, The one or more processors: A transmittance corrected image is generated by controlling the transmittance of the sub-camera image based on at least one of the driving situation information and the audio information; generating a superimposed image for superimposing the transmittance corrected image on the main camera image; The superimposed image is displayed on the display device. Remote driving system.

2. The remote driving system according to claim 1, The sub-camera image includes a side camera image and a rear camera image, The driving condition information includes speed information, The one or more processors, in controlling the transmittance of the sub-camera image, As the speed of the speed information decreases from a current value, the transmittance of the side camera image or the rear camera image is set to be increased linearly, nonlinearly, or stepwise; The transmittance of the side camera image or the rear camera image is set to be lowered linearly, nonlinearly, or stepwise as the speed of the speed information increases from a current value. Remote driving system.

3. The remote driving system according to claim 1, The sub-camera image includes a side camera image, The driving condition information includes information on a turn signal direction, The one or more processors further include: The transmittance of the side camera image in the same direction as the information on the turn signal direction is set to be high. Remote driving system.

4. A remote driving system according to any one of claims 1 to 3, The sub-camera image includes an indoor camera image, The one or more processors, in controlling the transmittance of the sub-camera image, determining whether the voice information includes a voice calling out from the occupant; When it is determined that the voice information includes a voice calling from the occupant, The transmittance of the indoor camera image is set to be low. Remote driving system.

5. A remote operation program for remotely controlling the operation of a moving object by a computer, A process of acquiring camera image information including a main camera image and a sub camera image captured by a plurality of cameras mounted on the moving body, and at least one of driving situation information indicating a driving situation of the moving body and voice information of a passenger of the moving body; A process of generating a transmittance corrected image corrected by controlling the transmittance of the sub camera image based on at least one of the driving situation information and the audio information; A process of generating a superimposed image in which the transmittance corrected image is superimposed on the main camera image; displaying the superimposed image on a display device; by the computer. Remote driving program.

6. A remote operation method for remotely controlling an operation of a moving object, comprising: A process of acquiring camera image information including a main camera image and a sub camera image captured by a plurality of cameras mounted on the moving body, and at least one of driving situation information indicating a driving situation of the moving body and voice information of a passenger of the moving body; A process of generating a transmittance corrected image corrected by controlling the transmittance of the sub camera image based on at least one of the driving situation information and the audio information; A process of generating a superimposed image in which the transmittance corrected image is superimposed on the main camera image; displaying the superimposed image on a display device; Includes Remote operation method.

Citation Information

Patent Citations

  • Camera image display device for mobile object

    JP1998145777A

  • Image display method for remote controlled working machine

    JP2011009823A

  • Display control unit and control method thereof

    JP2014109662A

  • Remote handling equipment

    JP2019029922A