Remote operation assistance method, remote operation assistance device, and program

Abstract

A remote operation assistance method according to the present disclosure is executed by a remote operation assistance device for assisting remote operation in which an operator remotely operates a movable body on the basis of an image for remote operation captured by a camera mounted in the movable body. The remote operation assistance method comprises: executing assistance control for assisting the remote operation if the distance from the movable body to be remotely operated to an object at the periphery of the movable body is less than a threshold; if the image for remote operation is switched, comparing distance sense information indicating the operator's sense of distance with respect to the distance to the object in the image for remote operation before and after the switching; and if the comparison result of the distance sense information satisfies a predetermined assistance criterion, changing the threshold for remote operation between before and after the switching.

Description

Remote operation support method, remote operation support device, and program

[0001] The present disclosure relates to a remote operation assistance method, a remote operation assistance device, and a program.

[0002] In recent years, various types of autonomous vehicle services have been put into practical use, and remote control systems that can remotely monitor or operate these vehicles are being developed. In these remote control systems, when an autonomous vehicle requests assistance via remote operation, an operator in a remote control room can provide assistance such as moving the autonomous vehicle by remotely operating the vehicle while viewing images captured by a camera mounted on the autonomous vehicle.

[0003] For example, there is known a technique for training a driver's driving sense by comparing actual measured values ​​of various physical quantities related to vehicle running with the driver's sense.

[0004] Japanese Patent Application Laid-Open No. 2005-070224

[0005] However, in the prior art, when remotely operating different types of vehicles, if the angle of view or mounting position of the camera mounted on the vehicle differs, the position and size of the object, such as an obstacle, captured by the camera also differs, and the sense of distance from the vehicle to the object based on the image, i.e., the sense of distance, also changes, which could affect the operator's remote operation.

[0006] One of the objects of the present disclosure is to appropriately support the operator in remote operation.

[0007] In order to achieve the above-mentioned object, the remote operation assistance method of the present disclosure is a remote operation assistance method executed by a remote operation assistance device that assists an operator in remotely operating a mobile body based on a remote operation image captured by a camera mounted on the mobile body, and when the distance from the mobile body to be remotely operated to an object around the mobile body is shorter than a threshold, an assistance control is executed to assist the remote operation, and when the image for remote operation is switched, distance perception information indicating the operator's sense of distance to the object in the image for remote operation before and after the switch is compared, and when the comparison result of the distance perception information satisfies a predetermined assistance standard, the threshold for the remote operation is changed before and after the switch.

[0008] FIG. 1 is a diagram illustrating an example of a schematic configuration of a remote operation system according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of each device included in the remote operation system according to an embodiment. FIG. 3 is a diagram illustrating an example of the hardware configuration of a control device for a vehicle according to an embodiment. FIG. 4 is a diagram illustrating an example of an image captured by a camera mounted on a vehicle according to an embodiment. FIG. 5 is a diagram illustrating an example of a correspondence between the speed of a vehicle and the magnitude of a motion vector of a surrounding area according to an embodiment. FIG. 6 is a diagram illustrating an example of tilt information associated with each vehicle ID, which is distance perception information according to an embodiment. FIG. 7 is a diagram illustrating an example of a sense of distance related to an object in an image captured by a camera in wide-angle mode according to an embodiment. FIG. 8 is a diagram illustrating an example of a sense of distance related to an object in an image captured by a camera in telephoto mode according to an embodiment. FIG. 9 is a diagram illustrating an example of a warning screen according to an embodiment. FIG. 10 is a sequence diagram illustrating an example of an operation procedure when a target vehicle to be remotely operated is switched in the remote operation system according to an embodiment. FIG. 11 is a flowchart illustrating an example of the flow of a calculation process of distance perception information executed in the control device of a vehicle according to an embodiment. FIG. 12 is a flowchart illustrating an example of the flow of an assistance process executed in the control device of a remote operation assistance device according to an embodiment when an assistance request is received. Fig. 13 is a diagram showing an example of focal length information associated with each vehicle ID, which is distance sense information according to a modified example. Fig. 14 is a flowchart showing an example of the flow of distance sense information calculation processing executed in a vehicle control device according to a modified example. Fig. 15 is a diagram for explaining an example of assistance processing according to a modified example. Fig. 16 is a diagram for explaining an example of assistance processing according to a modified example. Fig. 17 is a diagram for explaining an example of assistance processing according to a modified example.

[0009] Hereinafter, embodiments of a remote operation assistance method, a remote operation assistance device, and a program according to the present disclosure will be described in detail with reference to the accompanying drawings.

[0010] In the description of the present disclosure, components having the same or substantially the same functions as those described above with respect to the previously-mentioned drawings may be given the same reference numerals, and descriptions thereof may be omitted as appropriate. Furthermore, even when the same or substantially the same parts are shown, the dimensions and proportions may be different depending on the drawing. Furthermore, for example, in order to ensure the visibility of the drawings, reference numerals may be given to only the main components in the description of each drawing, and reference numerals may not be given to components having the same or substantially the same functions as those described above with respect to the previously-mentioned drawings.

[0011] FIG. 1 is a diagram illustrating an example of a schematic configuration of a remote control system 1 according to an embodiment. As shown in FIG. 1, the remote control system 1 includes multiple vehicles 10A and 10B (two vehicles in the example of FIG. 1), a remote control assistance device 20, and a terminal device 30. While FIG. 1 illustrates two vehicles 10A and 10B as an example, the number of vehicles 10 included in the remote control system 1 can be changed as desired depending on design conditions, etc. In the following description, when the vehicles 10A and 10B are not to be distinguished from each other, they will be simply referred to as "vehicles 10." The same reference numerals will be used to designate the same elements constituting the vehicle 10.

[0012] In the example of FIG. 1, a vehicle 10, a remote operation assistance device 20, and a terminal device 30 can be connected to each other via a network 40 such as the Internet.

[0013] The vehicle 10 is an example of an autonomously driven mobile body and is used to provide various services. The remote operation support device 20 is a device that supports the remote operation of the vehicle 10. The terminal device 30 is a device that is operated by an operator in a remote control room.

[0014] As an example, in the remote operation system 1 according to the present embodiment, when the vehicle 10 falls into a state where it is unable to travel autonomously, for example, when an obstacle is detected on the path, the vehicle 10 transmits an assistance request requesting assistance through remote operation to the remote operation assistance device 20. Upon receiving the assistance request from the vehicle 10, the remote operation assistance device 20 transmits a remote operation request requesting remote operation by an operator to the terminal device 30.

[0015] As an example, in the remote operation system 1 according to the present embodiment, when, for example, the operator remotely operating the vehicle 10 needs to be changed, the terminal device 30 transmits an assistance request to the remote operation assistance device 20 in response to an instruction from the operator, requesting assistance by remote operation of the vehicle 10. Here, the case where the operator remotely operating the vehicle 10 needs to be changed may occur, for example, when complex or advanced remote operation is required, or when the remote operation of the vehicle 10 needs to be entrusted to another operator in order to perform remote operation in response to an assistance request from another vehicle 10. In this case, the remote operation assistance device 20, upon receiving the assistance request from the terminal device 30, transmits a remote operation request to the terminal device 30 of the other operator, requesting remote operation by the other operator.

[0016] As an example, in the remote operation system 1 according to the present embodiment, when an operator wants to switch the vehicle 10 to be remotely operated, for example, when remotely operating a vehicle 10 according to the work content for each task or when one operator is responsible for multiple vehicles 10 at multiple work sites, the terminal device 30 transmits an assistance request to the remote operation assistance device 20 in response to an instruction from the operator, requesting assistance through remote operation of the vehicle 10. This assistance request can also be expressed as a switching request requesting switching of the vehicle 10 to be remotely operated. In this case, upon receiving the switching request from the terminal device 30, the remote operation assistance device 20 transmits a remote operation request to the terminal device 30 requesting (permitting) the operator to remotely operate the vehicle 10 to be switched to.

[0017] That is, in the remote operation system 1 according to this embodiment, the assistance request may be transmitted from the vehicle 10 to the remote operation assistance device 20, or may be transmitted from the terminal device 30 to the remote operation assistance device 20 in response to an instruction operation from an operator. Furthermore, the vehicle 10 that will be the target of remote operation after switching may be autonomously traveling, may be remotely operated by another operator, or may be waiting to be remotely operated by an operator before the switching.

[0018] The operator of the terminal device 30 that receives the remote control request can operate the terminal device 30 while viewing images captured by a camera mounted on the vehicle 10, thereby remotely controlling the vehicle 10 and providing assistance such as moving the vehicle 10.

[0019] As described above, the remote operation assistance device 20 according to this embodiment is an apparatus configured to be able to execute a remote operation assistance method in which an operator remotely operates the vehicle 10 and assists in the remote operation performed based on a remote operation image captured by a camera mounted on the vehicle 10. In the following, this embodiment will be described taking as an example a case in which an assistance request is sent from the vehicle 10 to the remote operation assistance device 20, for example, when the vehicle 10 falls into a state in which autonomous driving is not possible.

[0020] 2 is a diagram showing an example of the configuration of each device (vehicle 10, remote operation support device 20, and terminal device 30) included in the remote operation system 1 according to the embodiment. Hereinafter, the configurations of the vehicle 10, remote operation support device 20, and terminal device 30 will be described with reference to FIG.

[0021] (Vehicle 10) First, the configuration of the vehicle 10 will be described. The vehicle 10 according to this embodiment is an example of a moving body. The configuration of one vehicle 10 will be described below as an example, but the configurations of the other vehicles 10 included in the remote control system 1 are similar. As shown in FIG. 2 , the vehicle 10 includes, as hardware elements, a communication device 110, a camera 120, a drive device 130, and a control device 140. Note that the hardware elements of the vehicle 10 are not limited to the configuration exemplified in FIG. 2 , and the vehicle 10 may include other hardware elements.

[0022] The communication device 110 is a device that communicates with devices external to the vehicle 10, such as the remote operation assistance device 20 and the terminal device 30, via the network 40. The camera 120 is mounted on the vehicle 10 and is arranged so as to be able to capture an image of at least the area in front of the vehicle 10. Note that it is sufficient that at least one camera is arranged as the camera 120 so as to be able to capture an image of at least the area in the direction in which the vehicle 10 is moving, among the areas in front, to the side, behind, above, and below the vehicle 10.

[0023] The drive device 130 is a device that drives the vehicle 10. The drive device 130 includes, for example, a wheel drive device that applies a rotational drive force to the wheels of the vehicle 10, and a steering drive device that steers the wheels.

[0024] The control device 140 is a device that comprehensively controls the operation of the vehicle 10. The control device 140 may be realized by a computer such as an ECU (Electronic Control Unit) provided inside the vehicle 10, a DCU (Domain Control Unit) such as a CDC (Cockpit Domain Controller) that integrates multiple ECUs, or an OBU (On Board Unit). FIG. 3 is a diagram showing an example of the hardware configuration of the control device 140 of the vehicle 10 according to the embodiment. Note that a control device 230 included in a remote operation assistance device 20 described below and a control device 340 included in a terminal device 30 each have the same hardware configuration as that shown in FIG. 3.

[0025] As shown in FIG. 3, the control device 140 includes a processor 150 , a read-only memory (ROM) 160 , a random access memory (RAM) 170 , and a device I / F (interface) unit 180 .

[0026] The processor 150 is, for example, a CPU (Central Processing Unit). The processor 150 executes, for example, a program to comprehensively control the operation of the control device 140 and realize various functions of the control device 140. The various functions of the control device 140 will be described later.

[0027] The ROM 160 is a non-volatile memory that stores various types of information including programs executed by the processor 150. The memory of the control device 140 is not limited to the ROM 160, and various types of recording media and recording devices such as a hard disk drive (HDD), a solid state drive (SSD), and flash memory can be used as appropriate. The RAM 170 is a volatile memory that has a working area for the processor 150. The device I / F unit 180 is an interface for connecting the control device 140 to other devices installed in the vehicle 10, such as the communication device 110, the camera 120, and the drive device 130.

[0028] Returning to FIG. 2 , the functions of the control device 140 will be described. As shown in FIG. 2 , the control device 140 includes a distance perception information calculation unit 141, a position information acquisition unit 142, an image acquisition unit 143, a driving control unit 144, an assistance request transmission unit 145, and an operation information reception unit 146. Note that the example in FIG. 2 illustrates only the functions necessary for explaining the main parts of this embodiment, but the functions of the control device 140 are not limited to these. In this embodiment, the processor 150 executes a program stored in the ROM 160 to realize each function of the control device 140, including the functions of each of the above-mentioned units. However, without being limited to this, some or all of these functions may be realized by dedicated hardware circuits (such as semiconductor integrated circuits).

[0029] The distance perception information calculation unit 141 calculates information indicating a sense of distance (distance perception information) based on an image captured by the camera 120. This sense of distance is the sense of distance to an arbitrary position on the image that is felt by an operator remotely operating the vehicle 10 while looking at the image (video) captured by the camera 120. This arbitrary position on the image includes the position of an arbitrary object included in the image, such as another vehicle, an obstacle such as a wall, or a road marking.

[0030] Here, the distance perception information is an index corresponding to the field of view (FOV) of the remote operation image viewed by the operator remotely operating the vehicle 10 during remote operation, and is information indicating the distance based on the FOV. Note that the FOV of the remote operation image is the range of the field of view captured in the image. Therefore, the FOV of the remote operation image is the range of the image capturing range of the camera 120 that is used as the remote operation image.

[0031] As an example, the FOV for the image for remote operation is based on information about the camera 120 of the vehicle 10. This information about the camera 120 includes information indicating the angle of view (Angle of View) of the camera 120, such as the focal length, the mounting position (e.g., height), the mounting angle (e.g., elevation angle / depression angle), and the imaging orientation (e.g., elevation angle / depression angle). The information about the camera 120 also includes information indicating a region of interest (ROI) within the imaging range of the camera 120 that is used (e.g., cropped) as the image for remote operation.

[0032] Note that camera 120 may be configured so that its focal length is changeable, in which case the information about camera 120 includes information indicating the set focal length or shooting mode (e.g., wide-angle mode / telephoto mode). Camera 120 may be configured so that its mounting height is changeable, in which case the information about camera 120 includes information indicating the settable mounting height. Vehicle 10 may be provided with multiple cameras 120, in which case the information about camera 120 includes information about the camera 120 among the multiple cameras 120 that is used to acquire images for remote control.

[0033] Note that the change in focal length (or imaging mode) of camera 120 may be achieved by changing the position of at least one optical element in the optical system of camera 120, by switching between using and not using at least one optical element in the optical system, or by switching between two or more optical systems with different focal lengths. Alternatively, the change in focal length (or imaging mode) of camera 120 may be achieved by electronic zoom using a part of the subject image on the imaging surface.

[0034] The distance perception information calculation unit 141 according to this embodiment calculates, as distance perception information, gradient information indicating a gradient that is a ratio between a change in the speed of the vehicle 10 and a change in the motion vector of the area surrounding the vehicle 10. This gradient information is information that indicates a gradient that is a ratio between a change in the motion vector and a change in the speed of the vehicle 10. The gradient information is information that indicates a gradient that is a ratio between a change in the speed of the vehicle 10 and a change in the motion vector of the area surrounding the vehicle 10 captured by the camera 120 mounted on the vehicle 10.

[0035] In this embodiment, before the service is started, the vehicle 10 is driven in a predetermined location, and the distance sense information calculation unit 141 calculates the slope of a straight line that indicates the correspondence between the speed of the vehicle 10 and the motion vector of the area surrounding the vehicle 10 based on the speed of the vehicle 10 and the image captured by the camera 120, and transmits slope information indicating the calculated slope to the remote operation assistance device 20. Note that the timing for calculating the slope information is not limited to before the service is started, and can be set arbitrarily. For example, the distance sense information calculation unit 141 may calculate and update the slope information during the service.

[0036] An example of a method for determining the peripheral area of ​​the vehicle 10 and the motion vectors of the peripheral area will be described below. FIG. 4 is a diagram showing an example of an image captured by the camera 120 mounted on the vehicle 10 according to the embodiment. While the vehicle 10 is traveling, the camera 120 capturing an image ahead of the vehicle 10 captures an image (moving image) of the road surface or the like flowing in the opposite direction to the traveling direction of the vehicle 10, as shown in FIG. 4. In this image, even if the speed is the same, the area closer to the edge (periphery) appears to move more rapidly, and therefore the object appears to move faster. In other words, as the vehicle 10 travels, the object moves faster in areas closer to the edge of the image, and the object moves more slowly as it moves from the edge to the inside of the image.

[0037] Therefore, for example, the distance information calculation unit 141 can search for similar blocks between frames for each block, which is a unit for dividing an image captured by the camera 120, to determine a motion vector (block matching method), and set blocks whose motion vector magnitude is equal to or greater than a reference value as the peripheral area. This allows the motion vectors of each block included in the peripheral area to be calculated simultaneously with the setting of the peripheral area. Note that the method for calculating the motion vector is not limited to the block matching method described above. For example, a gradient method can be used to calculate a motion vector for each pixel of the image, and pixels whose motion vector is equal to or greater than a reference value can be set as the peripheral area. Furthermore, the motion vector of the peripheral area can also be calculated using, for example, a motion vector output from a codec used in video transmission. Furthermore, for example, a predetermined peripheral area within the image captured by the camera 120 may be set in advance as the peripheral area for each camera 120.

[0038] Next, an example of a method for calculating tilt information will be described. FIG. 5 is a diagram showing an example of the correspondence between the speed of the vehicle 10 and the magnitude of the motion vectors of the surrounding area according to this embodiment. In this embodiment, as shown in FIG. 5, a coordinate system is set in which the horizontal axis represents the speed of the vehicle 10 and the vertical axis represents the average value of the magnitude of the motion vectors of each block included in the surrounding area. The distance perception information calculation unit 141 calculates coordinates indicating a point corresponding to the average value of the magnitude of the motion vectors of the surrounding area calculated as described above and the speed of the vehicle 10 at that time (represented as a "predetermined speed" in FIG. 5). The distance perception information calculation unit 141 then calculates an equation of a straight line connecting the calculated coordinates and the origin of the coordinate system (an example of a function indicating the correspondence between the speed of the vehicle 10 and the motion vectors of the surrounding area), and can calculate the slope of the equation of that line. Information indicating the slope calculated as described above serves as tilt information.

[0039] 5, the vertical axis represents the average magnitude of the motion vectors of the blocks included in the surrounding area, but this is not limiting, and the vertical axis may represent, for example, the cumulative magnitude of the motion vectors of the blocks included in the surrounding area. Even in this case, distance information calculation unit 141 can calculate tilt information in the same manner as above.

[0040] The distance sense information calculation unit 141 transmits the tilt information calculated as described above and a vehicle ID indicating information for identifying the vehicle 10 to the remote operation assistance device 20, and the remote operation assistance device 20 stores the tilt information received from the vehicle 10 in association with the vehicle ID. A specific configuration of the remote operation assistance device 20 will be described later. Note that, in this embodiment, the distance sense information calculation unit 141 has both the function of calculating tilt information and the function of transmitting tilt information, but this is not limited thereto, and for example, the function of calculating tilt information and the function of transmitting tilt information may be provided separately.

[0041] 2 , we will continue to explain the functions of the control device 140 of the vehicle 10. The location information acquisition unit 142 acquires location information indicating the location of the vehicle 10. Various known techniques can be used to acquire the location information, but for example, the location information acquisition unit 142 can receive GPS signals indicating radio waves transmitted from each of a plurality of GPS satellites, calculate the location of the vehicle 10 by three-dimensional positioning based on the received GPS signals, and acquire location information indicating the location.

[0042] The image acquisition unit 143 acquires images captured by the camera 120. The images acquired by the image acquisition unit 143 are also used to calculate the above-mentioned tilt information. Furthermore, while the vehicle 10 is traveling after the service operation has started, the images acquired by the image acquisition unit 143 are transmitted to the remote operation assistance device 20.

[0043] In the autonomous driving mode, which indicates a state in which the vehicle 10 is driving autonomously, the driving control unit 144 controls the driving of the vehicle 10 (controls the driving of the drive unit 130) so that the vehicle 10 approaches the target position, based on the target position and the position information acquired by the position information acquisition unit 142. On the other hand, in the remote control mode, which indicates a state in which the vehicle 10 is remotely operated by an operator, the driving control unit 144 controls the driving of the vehicle 10 in accordance with operation information indicating information input to the terminal device 30 in response to operation by the operator. In this embodiment, the driving modes of the vehicle 10 include the autonomous driving mode described above and the remote control mode described above, and the vehicle 10 basically drives in the autonomous driving mode and drives in the remote control mode when autonomous driving is not possible.

[0044] The assistance request sending unit 145 sends an assistance request requesting assistance by remote operation to the remote operation assistance device 20, for example, when the vehicle 10 is in a state in which it cannot travel autonomously in the autonomous travel mode. An example of a state in which the vehicle 10 is in a state in which it cannot travel autonomously is, for example, a state in which an object of a predetermined size or larger (a size at which it is determined that the vehicle 10 cannot travel straight) is present in the traveling direction of the vehicle 10. In this embodiment, when the assistance request sending unit 145 detects, based on the image acquired by the image acquisition unit 143, that an object of a predetermined size or larger is present in the traveling direction of the vehicle 10, the assistance request sending unit 145 sends an assistance request to the remote operation assistance device 20. The assistance request in this embodiment is information that includes at least information requesting assistance by remote operation and a vehicle ID that identifies the vehicle 10.

[0045] The operation information receiving unit 146 receives operation information transmitted from the terminal device 30 via the remote operation assistance device 20. After the assistance request transmitting unit 145 transmits an assistance request, the driving control unit 144 controls the driving of the vehicle 10 in accordance with the operation information received by the operation information receiving unit 146, and does not perform driving control based on the target position and position information. In other words, the driving mode of the vehicle 10 switches from the autonomous driving mode to the remote operation mode.

[0046] (Remote operation support device 20) Next, the configuration of the remote operation support device 20 will be described. As shown in Fig. 2, the remote operation support device 20 includes, as hardware elements, a communication device 210, a storage unit 220, and a control device 230. Note that the hardware elements of the remote operation support device 20 are not limited to the configuration exemplified in Fig. 2, and may include other hardware elements.

[0047] The communication device 210 is a device that communicates with external devices (for example, the vehicle 10, the terminal device 30, etc.) via the network 40.

[0048] The storage unit 220 stores tilt information in association with each vehicle 10. FIG. 6 is a diagram showing an example of distance sense information according to the embodiment, in which tilt information is associated with each vehicle ID. In this embodiment, as shown in FIG. 6, the storage unit 220 stores tilt information in association with each vehicle ID indicating information identifying a vehicle. Note that the storage format of the tilt information is not limited to the format shown in FIG. 6.

[0049] The control device 230 is a device that comprehensively controls the operation of the remote operation support device 20. In this embodiment, the control device 230 is configured as a computer device, and has the same hardware configuration as that shown in FIG.

[0050] Next, the functions of the control device 230 of the remote operation assistance device 20 will be described. As shown in FIG. 2 , the control device 230 includes a distance perception information receiving unit 231, an assistance request receiving unit 232, an acquisition unit 233, a determination unit 234, an assistance control unit 235, and a remote information transmitting / receiving unit 236. Note that the example in FIG. 2 illustrates only the functions necessary for explaining the main parts of this embodiment, but the functions of the control device 230 are not limited to these. In this embodiment, the processor 150 executes a program stored in the ROM 160 to realize each function of the control device 140, including the functions of each of the above-mentioned units. However, this is not limiting, and some or all of these functions may be realized by dedicated hardware circuits.

[0051] As described above, the distance perception information receiving unit 231 receives the tilt information (distance perception information) and the vehicle ID transmitted from the vehicle 10. The distance perception information receiving unit 231 then associates the tilt information received from the vehicle 10 with the vehicle ID and stores them in the storage unit 220 (see FIG. 6 ). Note that in this embodiment, the distance perception information receiving unit 231 has a function to receive tilt information and a function to store the tilt information in the storage unit 220, but this is not limiting, and for example, the function to receive tilt information and the function to store tilt information in the storage unit 220 may each be provided separately.

[0052] The assistance request receiving unit 232 receives the assistance request transmitted from the vehicle 10 .

[0053] The acquisition unit 233 acquires the tilt information described above. More specifically, when the assistance request receiving unit 232 receives an assistance request, the acquisition unit 233 in this embodiment identifies the vehicle ID included in the received assistance request. Then, the acquisition unit 233 acquires the tilt information associated with the identified vehicle ID from the storage unit 220. In this embodiment, the acquisition unit 233 acquires the tilt information each time the assistance request receiving unit 232 receives an assistance request. As described above, an assistance request is transmitted for each vehicle 10. Therefore, receiving an assistance request from one vehicle 10 and then receiving an assistance request from another vehicle 10 can be considered to mean that the vehicle 10 to be remotely controlled is switched.

[0054] When the image for remote control is switched, the determination unit 234 compares the distance sense information indicating the sense of distance before and after the switch, and determines whether the comparison result satisfies a predetermined support criterion. In other words, the determination unit 234 determines whether the change in the distance sense information before and after the switch satisfies the support criterion. Here, satisfying the support criterion according to this embodiment means that the tilt indicated by the tilt information has changed by more than a predetermined tilt.

[0055] In this embodiment, distance perception information indicating the operator's sense of distance is acquired using optical flow, which represents the movement of an object in an image using a motion vector. The motion vector is large in an image captured at a wide-angle angle of view (focal length) at which the operator perceives the distance as "far." On the other hand, the motion vector is small in an image captured at a telephoto angle of view (focal length) at which the operator perceives the distance as "close." Therefore, with a configuration that uses tilt information based on the motion vector as distance perception information, a large motion vector is closer to the wide-angle side than a small motion vector, and it can be determined that the object is perceived as far away. Similarly, a small motion vector is closer to the telephoto side than a large motion vector, and it can be determined that the object is perceived as close.

[0056] Specifically, the determination unit 234 determines whether the tilt indicated by the tilt information acquired by the acquisition unit 233 has decreased by a predetermined amount or more. If the tilt has decreased, the determination unit 234 determines that the sense of speed has decreased, i.e., the subject is on the telephoto side, and therefore determines that the sense of distance is "close" (see FIG. 8 ). For example, the determination unit 234 determines whether the tilt indicated by the tilt information acquired by the acquisition unit 233 has increased by a predetermined amount or more. If the tilt has increased, the determination unit 234 determines that the sense of speed has increased, i.e., the subject is on the wide-angle side, and therefore determines that the sense of distance is "far" (see FIG. 7 ).

[0057] In this embodiment, each time the image for remote control is switched, the determination unit 234 compares the tilt indicated by the latest tilt information with the tilt indicated by the tilt information acquired immediately before that, and determines whether the tilt indicated by the tilt information has changed, i.e., whether the sense of distance has changed.

[0058] Here, we will explain how the sense of distance changes in response to changes in the surrounding area of ​​the vehicle 10. For example, assume that the vehicle 10 to be remotely controlled (the vehicle 10 that sent the assistance request) switches between a vehicle 10 equipped with a camera 120 in a wide-angle mode with a large angle of view and a vehicle 10 equipped with a camera 120 in a telephoto mode with a small angle of view.

[0059] FIG. 7 is a diagram illustrating an example of the sense of distance to an object in an image captured by the camera 120 in wide-angle mode according to the embodiment. FIG. 8 is a diagram illustrating an example of the sense of distance to an object in an image captured by the camera 120 in telephoto mode according to the embodiment. As can be seen from FIGS. 7 and 8 , the same subject is relatively enlarged in the image captured by the camera 120 in telephoto mode, causing the operator to feel that the distance to an object, such as an obstacle, is "closer." Therefore, when switching from the state of FIG. 7 to the state of FIG. 8 , even if the actual distance from the vehicle 10 to the object remains unchanged, the operator's sense of distance changes from "far" to "close." This may cause the operator to become anxious due to concerns about a collision between the vehicle 10 and the object, or to reduce the speed of the vehicle 10 more than necessary.

[0060] On the other hand, when switching from the state in Fig. 8 to the state in Fig. 7, even if the actual distance from the vehicle 10 to the object remains the same, the operator's sense of distance changes from "close" to "far." This may cause the operator to close the distance between the vehicle 10 and the object or to increase the speed of the vehicle 10 more than necessary. Furthermore, if the distance between the vehicle 10 and the object is too close or if the speed exceeds the required speed, the vehicle 10 may come into contact with (e.g., collide with) the object in the direction of travel.

[0061] Note that when switching from the state in Fig. 7 to the state in Fig. 8, the surrounding area (see Fig. 4) is relatively reduced due to the expansion of the central portion, and the average magnitude of the motion vectors in the surrounding area decreases, and the slope indicated by the slope information also decreases. On the other hand, when switching from the state in Fig. 8 to the state in Fig. 7, the surrounding area is relatively expanded, and the average magnitude of the motion vectors in the surrounding area increases, and the slope indicated by the slope information also increases.

[0062] In other words, when the tilt indicated by the tilt information changes, the operator's sense of distance changes. Here, when the tilt indicated by the tilt information decreases, the operator's sense of distance indicated by the tilt information becomes closer. On the other hand, when the tilt indicated by the tilt information increases, the operator's sense of distance indicated by the tilt information becomes farther.

[0063] In this situation, the remote operation assistance device 20 of this embodiment is configured to execute control (e.g., an alert) to assist remote operation when the distance from the vehicle 10 to be remotely operated to an object around the vehicle 10 is shorter than the threshold for assistance control.

[0064] Therefore, when the image for remote operation is switched in response to a request for remote assistance, the remote operation assistance device 20 of this embodiment determines whether the tilt indicated by the tilt information has changed, and if it is determined that the tilt indicated by the tilt information has changed beyond a predetermined standard, i.e., the assistance standard is met, the remote operation assistance device 20 changes the threshold value for assistance control that assists the operator in remote operation before and after the switch. In other words, if the change in the tilt information before and after the image for remote operation is switched satisfies the assistance standard, i.e., if the sense of distance has changed, the assistance control unit 235 of this embodiment changes the threshold value for control that assists the remote operation of the vehicle 10 before and after the switch.

[0065] As an example, the assistance control unit 235 increases the assistance control threshold when the perceived distance indicated by the tilt information changes from "far" to "close" before and after the change of the remote operation image. This allows the timing of issuing an alert in the assistance control to be advanced if the perceived distance becomes "close" before and after the change of the remote operation image. When remote operation is performed based on a remote operation image that gives the perceived distance a "close" feeling, if an alert is issued at the timing before the change, the perceived distance to an object such as an obstacle becomes "close," which may cause the operator to feel anxious about a collision. On the other hand, by changing the assistance control threshold according to this embodiment, the timing of issuing an alert can be advanced in situations where the perceived distance to an object such as an obstacle is "close." This reduces the operator's anxiety caused by the perceived closeness, allowing them to continue remote operation with peace of mind.

[0066] As an example, the assistance control unit 235 reduces the assistance control threshold when the perceived distance indicated by the tilt information changes from "close" to "far" before and after the remote operation image is switched. This allows the timing of issuing an alert in assistance control to be delayed if the perceived distance becomes "far" before and after the remote operation image is switched. When remote operation is performed based on a remote operation image that gives the perceived distance a "far" feeling, if an alert is issued at the timing before the change, the operator may feel the distance to an object such as an obstacle is far, which can lead to an increased risk of collision, such as excessive speed or getting too close to the object. On the other hand, by changing the assistance control threshold according to this embodiment, the timing of issuing an alert can be delayed in situations where the perceived distance to an object such as an obstacle is far, allowing the operator to perform appropriate remote operation and safely continue remote operation.

[0067] Note that as the threshold value of the assist control is reduced, the actual distance between the vehicle 10 and an object such as an obstacle at the time when the alert is issued becomes smaller. Therefore, the smaller the threshold value of the assist control is, the more likely it is that an alert will not be able to respond, increasing the risk of a collision. Therefore, the assist control unit 235 according to the embodiment reduces the threshold value of the assist control within a range exceeding a lower limit value predetermined based on safety. This lower limit value is a threshold value that is predetermined with safety in mind, for example, based on the operator's reaction speed and the risk of a collision.

[0068] In addition, when the sense of distance indicated by the tilt information changes from "close" to "far" before and after the image for remote operation is switched, the assistance control unit 235 according to the embodiment may, instead of or in addition to reducing the threshold value of the assistance control, issue an alert to warn the operator not to increase the speed too much.

[0069] Here, assistance control for assisting remote operation according to this embodiment will be described. The assistance control for assisting remote operation includes a first assistance control when the sense of distance changes and a second assistance control that uses a threshold based on sense of distance information.

[0070] The assistance control unit 235 performs a first assistance control for assisting remote operation, which is a control for notifying the user that the sense of distance changes as the remote operation image is switched. As an example, the control for notifying the user that the sense of distance changes may be a control for warning the user of the risk of collision with an object such as an obstacle, such as "Be careful not to get too close," when the sense of distance becomes "far." FIG. 9 is a diagram illustrating an example of a warning screen according to the embodiment. For example, the assistance control unit 235 may perform a control for displaying a warning screen (assistance information) shown in FIG. 9 on the terminal device 30 when the sense of distance becomes "far."

[0071] Note that the control for notifying the user of the change in sense of distance when the sense of distance becomes "far" may be control for warning the user against speeding, such as "Be careful not to exceed the speed limit." For example, when the sense of distance becomes "far," the assistance control unit 235 may perform control for displaying a warning screen (assistance information) on the terminal device 30 that warns the user against speeding.

[0072] The control for notifying the operator that the sense of distance is changing may be control for notifying the operator that the sense of distance is getting closer in order to reduce anxiety when the sense of distance is getting closer. For example, when the sense of distance is getting closer, the assistance control unit 235 may perform control for displaying a notification screen (assistance information) on the terminal device 30 to notify the operator that the sense of distance is getting closer.

[0073] Furthermore, as a second assistance control for assisting remote operation, the assistance control unit 235 performs control to notify the approach of a surrounding object, such as "vehicle approaching" or "obstacle approaching," when the distance from the remotely operated vehicle 10 to an object around the vehicle 10 is shorter than the assistance control threshold changed according to the tilt information (distance sense information) as described above. For example, the assistance control unit 235 may perform control to display a warning screen (assistance information) on the terminal device 30 to warn of approach to a surrounding object when the distance to the surrounding object is shorter than the assistance control threshold based on the tilt information.

[0074] The distance to the surrounding objects may be acquired by any method. For example, the assistance control unit 235 may acquire the distance from the vehicle 10 to the surrounding objects based on the output of a sensor (not shown) of the vehicle 10, such as a sonar or LiDAR. For example, the assistance control unit 235 may acquire the distance from the vehicle 10 to the surrounding objects based on the distance in an image captured by the vehicle 10 and camera information such as the focal length of the camera 120. For example, the assistance control unit 235 may acquire the distance from the vehicle 10 to the surrounding objects based on the position information of the vehicle 10 and data of a three-dimensional structure information map related to the traveling route of the vehicle 10 stored in advance in the storage unit 220 or the like. The position information of the vehicle 10 may be acquired by a GNSS sensor (not shown) such as a GPS sensor of the vehicle 10, may be acquired based on the result of self-position estimation using dead reckoning technology, or may be acquired based on the result of comparing the image captured by the camera 120 with an image of the surrounding environment acquired in advance and stored in the storage unit 220 or the like.

[0075] Returning to FIG. 2 , the description of the functions of the control device 230 of the remote operation assistance device 20 will be continued. The remote information transmitting / receiving unit 236 transmits and receives remote information indicating information used for remotely operating the vehicle 10. The remote information includes, for example, operation information transmitted from the terminal device 30, images captured by the camera 120 of the vehicle 10, and other information. For example, the remote information transmitting / receiving unit 236 can receive operation information transmitted from the terminal device 30 and transmit the received operation information to the vehicle 10. Furthermore, as described above, while the vehicle 10 is traveling after the service operation has started, images captured by the camera 120 of the vehicle 10 are sent to the remote operation assistance device 20, so the remote information transmitting / receiving unit 236 can also transmit the images received from the vehicle 10 to the terminal device 30.

[0076] (Terminal Device 30) Next, the configuration of the terminal device 30 will be described. As shown in Fig. 2, the terminal device 30 includes, as hardware elements, a communication device 310, a display device 320, an operation device 330, and a control device 340. Note that the hardware elements of the terminal device 30 are not limited to the configuration exemplified in Fig. 2, and may include other hardware elements.

[0077] The communication device 310 is a device that communicates with an external device (e.g., the remote operation support device 20) via the network 40. The display device 320 is a device that displays various types of information and is configured, for example, with a liquid crystal display. The operation device 330 is a device with which an operator performs various operations.

[0078] The control device 340 is a device that comprehensively controls the operation of the terminal device 30. In this embodiment, the control device 340 is configured as a computer device, and has the same hardware configuration as that shown in FIG.

[0079] Next, the functions of the control device 340 will be described. As shown in FIG. 2, the control device 340 has an information receiving unit 341, a display control unit 342, and an operation information transmitting unit 343. Note that the example in FIG. 2 illustrates only the functions necessary for explaining the main parts of this embodiment, but the functions of the control device 340 are not limited to these. In this embodiment, the processor 150 executes a program stored in the ROM 160 to realize the functions of each of the above-mentioned units. However, this is not limiting, and some or all of these functions may be realized by dedicated hardware circuits.

[0080] The information receiving unit 341 receives various types of information transmitted from the remote operation assistance device 20. For example, the information receiving unit 341 can receive information such as the above-mentioned remote operation request, the above-mentioned assistance information, and images captured by the camera 120 of the vehicle 10 from the remote operation assistance device 20.

[0081] The display control unit 342 controls the display of various information on the display device 320. For example, the display control unit 342 can control the display of the above-mentioned remote operation request (e.g., a message requesting remote operation) on the display device 320, can control the display of the above-mentioned support information (e.g., a warning screen) on the display device 320, and can also control the display of an image captured by the camera 120 of the vehicle 10 on the display device 320. For example, in the above-mentioned remote operation mode, an image captured by the camera 120 of the vehicle 10 is transmitted to the terminal device 30 via the remote operation support device 20, and the display control unit 342 displays the image received from the remote operation support device 20 on the display device 320, allowing the operator to remotely control the vehicle 10 while checking the driving status of the vehicle 10.

[0082] The operation information transmitting unit 343 transmits operation information input in response to the operator's operation of the operation device 330 to the remote operation assistance device 20. For example, after the operator confirms the above-mentioned remote operation request displayed on the display device 320, the operator operates the operation device 330 to start remote operation of the vehicle 10, and the operation information transmitting unit 343 can transmit the operation information input in response to the operator's operation of the operation device 330 to the remote operation assistance device 20.

[0083] Next, an example of the operation of the remote control system 1 according to the embodiment will be described with reference to the drawings. Note that the operation procedures and processing flows described below are merely examples, and the order of steps can be changed, some steps can be deleted, and other steps can be added as desired.

[0084] 10 is a sequence diagram showing an example of an operation procedure when the remotely controlled vehicle is switched in the remote control system 1 according to the embodiment. Here, an example of an operation procedure of the remote control system 1 when the remotely controlled vehicle is switched from vehicle 10A to vehicle 10B will be described.

[0085] First, before the service is started, the vehicle 10A calculates the tilt information described above as distance information and transmits the calculated tilt information and the vehicle ID to the remote operation support device 20 (step S1). The remote operation support device 20 associates the tilt information (distance information) received from the vehicle 10A with the vehicle ID and stores them in the storage unit 220 (step S2). Similarly, the vehicle 10B transmits the tilt information and the vehicle ID as distance information to the remote operation support device 20 (step S3), and the remote operation support device 20 associates the tilt information (distance information) received from the vehicle 10B with the vehicle ID and stores them in the storage unit 220 (step S4).

[0086] The following describes the operation procedure after the service operation starts. In the example of FIG. 10 , the vehicle 10A first becomes unable to autonomously travel and transmits the above-mentioned assistance request to the remote operation assistance device 20 (step S5). Upon receiving the assistance request from the vehicle 10A, the remote operation assistance device 20 acquires tilt information corresponding to the vehicle ID of the vehicle 10A included in the assistance request from the storage unit 220 and determines whether the sense of distance indicated by the acquired tilt information has changed beyond the assistance criterion from the sense of distance indicated by the previously acquired tilt information (step S6). In this example, since it is assumed that no assistance request prior to the assistance request from the vehicle 10A has been received, the determination result of step S6 is negative. Since the determination result of step S6 is negative, the remote operation assistance device 20 transmits the above-mentioned remote operation request to the terminal device 30 without changing the threshold value for the second assistance control or transmitting the above-mentioned assistance information for the first assistance control (step S7).

[0087] The terminal device 30 displays the remote operation request received from the remote operation assistance device 20 (step S8). After confirming the remote operation request, the operator starts remote operation of the vehicle 10A, and the terminal device 30 transmits operation information corresponding to the operator's operation to the remote operation assistance device 20 (step S9). At this time, the remote operation assistance device 20 starts the second assistance control using the threshold value set at that time. The remote operation assistance device 20 transmits the operation information received from the terminal device 30 to the vehicle 10A (step S10), and the vehicle 10A travels in accordance with the operation information received from the remote operation assistance device 20. In other words, the vehicle 10A travels in accordance with the operator's remote operation.

[0088] Next, in the example of FIG. 10 , vehicle 10B becomes unable to autonomously drive and transmits the above-described assistance request to the remote operation assistance device 20 (step S11). Upon receiving the assistance request from vehicle 10B, the remote operation assistance device 20 acquires tilt information corresponding to the vehicle ID of vehicle 10B included in the assistance request from the storage unit 220 and determines whether the sense of distance indicated by the acquired tilt information has changed from the sense of distance indicated by the previously acquired tilt information by exceeding the assistance criterion (step S12). Here, it is assumed that, as the remote control target switches from vehicle 10A to vehicle 10B, the change in the sense of distance information indicating the sense of distance based on the remote operation image satisfies the predetermined assistance criterion. In this case, the determination result of step S12 is positive. Since the determination result of step S12 is positive, the remote operation assistance device 20 changes the threshold for the second assistance control (step S13) and transmits the above-described remote operation request and assistance information for the first assistance control to the terminal device 30 (step S14).

[0089] The terminal device 30 displays the remote operation request and the support information for the first support control received from the remote operation assistance device 20 (step S15). After confirming the remote operation request and the support information for the first support control, the operator begins remote operation of the vehicle 10B. For example, if the perceived distance becomes "far," a warning screen (support information) such as that shown in FIG. 9 is displayed on the display device 320 of the terminal device 30. This prevents the operator from being influenced by the perceived distance becoming "far" in the remote operation image and from closing the distance between the vehicle 10B and an object such as an obstacle too much or increasing the speed of the vehicle 10B too much. The terminal device 30 transmits operation information corresponding to the operator's operation to the remote operation assistance device 20 (step S16). At this time, the remote operation assistance device 20 initiates second support control using the threshold value changed based on the change in the perceived distance information. The remote operation assistance device 20 transmits the operation information received from the terminal device 30 to the vehicle 10B (step S17), and the vehicle 10B travels in accordance with the operation information received from the remote operation assistance device 20. In other words, the vehicle 10B travels according to the remote control by the operator. In addition, in the second assistance control, when the distance from the remotely operated vehicle 10 to an object around the vehicle 10 is shorter than a threshold, the remote operation assistance device 20 transmits assistance information for the second assistance control, such as "vehicle approaching" or "obstacle approaching," to the terminal device 30 to notify the approach of the object around the vehicle 10 (step S18).

[0090] FIG. 11 is a flowchart illustrating an example of the flow of a distance perception information calculation process executed by the control device 140 of the vehicle 10 according to the embodiment. FIG. 11 illustrates a case in which the above-described tilt information is calculated as the distance perception information. As shown in FIG. 11 , the distance perception information calculation unit 141 first acquires speed information indicating the speed of the vehicle 10 (step S101). Next, the distance perception information calculation unit 141 calculates a motion vector of the area surrounding the vehicle 10 based on the image captured by the camera 120 (step S102). Next, the distance perception information calculation unit 141 calculates a linear equation indicating the correspondence between the speed of the vehicle 10 and the motion vector of the area surrounding the vehicle 10 based on the speed information acquired in step S101 and the motion vector calculated in step S102, and calculates tilt information indicating the tilt of the linear equation (step S103). Next, the distance perception information calculation unit 141 transmits the tilt information calculated in step S103 and the vehicle ID to the remote operation assistance device 20 (step S104). The remote operation assistance device 20 (distance sense information receiving unit 231) stores the tilt information received from the vehicle 10 (distance sense information calculating unit 141) in association with the vehicle ID in the storage unit 220.

[0091] 12 is a flowchart showing an example of the flow of assistance processing executed by the control device 230 of the remote operation assistance device 20 according to the embodiment when an assistance request is received. As shown in FIG. 12 , the assistance request receiving unit 232 first receives the assistance request from the vehicle 10 (step S201). Next, the acquiring unit 233 acquires, from the storage unit 220, tilt information as sense of distance information corresponding to the vehicle ID included in the assistance request received in step S201 (step S202). Next, the determining unit 234 determines whether the change in the sense of distance information satisfies the assistance criterion based on the tilt information acquired in step S202 (step S203).

[0092] If the result of step S203 is positive (step S203: Yes), the assistance control unit 235 increases the threshold value of the second assistance control (step S205) if the sense of distance has decreased and become "closer" (step S204: Yes), and decreases the threshold value of the second assistance control (step S206) if the sense of distance has increased and become "farther" (step S204: No). On the other hand, if the result of step S203 is negative (step S203: No), the assistance control unit 235 does not change the threshold value of the second assistance control and proceeds to step S207.

[0093] The assistance control unit 235 then transmits a remote operation request indicating information requesting remote operation of the vehicle 10 in response to the assistance request to the terminal device 30 (step S207). At this time, the assistance control unit 235 may transmit assistance information for a first assistance control that assists the remote operation to the terminal device 30, as described above. Furthermore, as described above, the assistance control unit 235 determines whether the distance between the vehicle 10 to be remotely operated and a surrounding object becomes closer than the threshold for a second assistance control during the remote operation (S208).

[0094] If the result of step S208 is positive (step S208: Yes), the support control unit 235 transmits support information (alert) of the second support control that supports the remote operation to the terminal device 30 as described above (S209). On the other hand, if the result of step S208 is negative (step S208: No), the support control unit 235 does not issue an alert.

[0095] As described above, the remote operation assistance device 20 of this embodiment changes the threshold value of the second assistance control for issuing an alert when the operator's sense of distance to an object from the remote operation image changes, for example, when the vehicle 10 to be operated is switched. With this configuration, even when the remote operation target is switched to a vehicle 10 with a different sense of distance from the remote operation image, it is possible to reduce discomfort with the timing of the alert due to differences in sense of distance. Furthermore, with the above configuration, even when the remote operation target is switched to a vehicle 10 with a different sense of distance, it is possible to reduce the operator's anxiety due to differences in sense of distance and improve safety by reducing the risk of speeding or collision due to differences in sense of distance. Therefore, this embodiment can appropriately assist the operator in remote operation.

[0096] Note that some of the functions of each device in the remote operation system 1 according to the above-described embodiment may be realized by other devices in the remote operation system 1. For example, some of the functions related to the assistance control of the remote operation assistance device 20 according to the above-described embodiment may be realized in at least one of the vehicle 10 and the terminal device 30. For example, some of the functions related to the calculation of the sense of distance information of the vehicle 10 according to the above-described embodiment may be realized in at least one of the remote operation assistance device 20 and the terminal device 30. Alternatively, in the remote operation system 1 according to the above-described embodiment, the remote operation assistance device 20 and the terminal device 30 may be configured as an integrated unit.

[0097] In the above-described embodiment, the determination of "whether it is A or not" may be realized by determining only that it is A, by determining only that it is not A, or by determining both of these.

[0098] In the above embodiment, "any of A" means "at least one of A."

[0099] The programs executed by each device (e.g., control devices 140, 230, 340) of the remote control system 1 according to the above-described embodiment may be provided by being recorded in an installable or executable file format on a computer-readable recording medium (Computer Program Product) such as a CD-ROM, FD, CD-R, or DVD.

[0100] The programs executed by the devices in the remote control system 1 according to the above-described embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The programs executed by the devices in the remote control system 1 according to the above-described embodiment may be provided or distributed via a network such as the Internet.

[0101] Furthermore, the programs executed by the devices of the remote control system 1 according to the above-described embodiment may be provided by being pre-installed in a ROM or the like.

[0102] Although the embodiments of the present disclosure have been described above, the above-described embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and modifications thereof are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0103] Furthermore, the effects of the embodiments described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0104] Modifications will be described below.

[0105] (1) Modification 1 In the above-described embodiment, tilt information is used as the sense of distance information, but this is not limiting. The sense of distance information is not limited to tilt information, and may be camera information including at least one of focus information of the camera 120, the mounting height, the size of the imaging range (e.g., the angle of view), and the size of the range within the imaging range used as the image for remote operation (e.g., the field of view, the crop range), or depth information based on the image for remote operation.

[0106] The storage unit 220 according to this modification stores a focal length associated with each vehicle 10. FIG. 13 is a diagram showing an example of distance perception information according to this modification, where focal length information is associated with each vehicle ID. FIG. 13 illustrates a case where the focal length of the camera 120 is used as camera information. In this modification, as shown in FIG. 13, the storage unit 220 stores a focal length associated with each vehicle ID indicating information identifying the vehicle. Note that the storage format of the focal lengths is not limited to the format shown in FIG. 13.

[0107] FIG. 14 is a flowchart showing an example of the flow of a distance perception information calculation process executed by the control device 140 of the vehicle 10 according to a modified example. FIG. 14 illustrates a case where the focal length of the camera 120 is used as the camera information. As shown in FIG. 14 , the distance perception information calculation unit 141 first acquires the focal length as camera information from the camera 120 that acquires images for remote operation (step S301). The distance perception information calculation unit 141 then transmits the focal length and vehicle ID acquired in step S301 to the remote operation assistance device 20 (step S302). The remote operation assistance device 20 (distance perception information receiving unit 231) associates the focal length and vehicle ID received from the vehicle 10 (distance perception information calculation unit 141) and stores them in the storage unit 220.

[0108] In a configuration that uses focal length as distance perception information, the longer the focal length, the more telephoto the image, and the closer the perceived distance from the image. On the other hand, the shorter the focal length, the wider the angle, and the farther the perceived distance from the image. Therefore, when the image for remote operation is switched, the change in the perceived distance can be determined based on the change in focal length, as in the above-described embodiment, and assistance control can be realized according to the perceived distance.

[0109] Similarly, in a configuration in which the mounting height of the camera 120 on the vehicle 10 is used as distance perception information, the higher the mounting height, the more telephoto the image, and the "closer" the sense of distance perceived from the image. On the other hand, the lower the mounting height, the wider the angle, and the "farther" the sense of distance perceived from the image. Therefore, when the image for remote operation is switched, the change in the sense of distance can be determined based on the change in mounting height, as in the above-described embodiment, and assistance control according to the sense of distance can be realized.

[0110] Similarly, in a configuration in which the angle of view of the camera 120 is used as distance perception information, the larger the angle of view, the wider the angle, and the "farther" the sense of distance perceived from the image. On the other hand, the smaller the angle of view, the more telephoto the image, and the "closer" the sense of distance perceived from the image. Therefore, when the image for remote operation is switched, the change in the sense of distance can be determined based on the change in the angle of view, as in the above-described embodiment, and assistance control according to the sense of distance can be realized.

[0111] Similarly, in a configuration in which the field of view or crop range based on the image acquired by camera 120 is used as distance perception information, the larger the field of view or crop range, the wider the angle, and the "farther" the sense of distance perceived from the image. On the other hand, the smaller the field of view or crop range, the more telephoto the image, and the "closer" the sense of distance perceived from the image. Therefore, when the image for remote operation is switched, a change in the sense of distance can be determined based on the change in the field of view or crop range, as in the above-described embodiment, and assistance control according to the sense of distance can be realized.

[0112] Similarly, in a configuration using depth information based on an image for remote operation as distance perception information, the greater the change in the minimum distance in the depth information, the more telephoto the image, and the "closer" the sense of distance perceived from the image. On the other hand, the smaller the change in the minimum distance in the depth information, the wider the angle, and the "farther" the sense of distance perceived from the image. Depth information can be acquired using a machine learning model such as a deep neural network (DNN), whose parameters are determined to output a distance image in response to an input RGB image. For example, the distance perception information calculation unit 141 inputs an RGB image captured by the camera 120 into the DNN and acquires the output of the DNN in response to the input RGB image as depth information. Therefore, when the image for remote operation is switched, the change in the sense of distance can be determined based on the change in the minimum distance in the depth information, as in the above-described embodiment, and assistance control according to the sense of distance can be realized.

[0113] (2) Modification 2 In the above embodiment, a case where the remote control image changes as the remotely controlled vehicle 10 changes has been exemplified, but this is not limited to this. The remote control image may change as the camera 120 used to acquire the remote control image changes, for example, when multiple cameras 120 are installed on the same vehicle 10. Alternatively, the remote control image may change as the characteristics of the camera 120 change. In other words, the remote control image may change based on a change in at least one of the vehicle 10 to be remotely controlled and the camera 120 on the vehicle 10.

[0114] Switching of the camera 120 used to acquire the image for remote operation or switching of the characteristics of the camera 120 may mean switching of the FOV of the image for remote operation that is viewed by the operator remotely operating the vehicle 10. In other words, when the image for remote operation is switched, if the sense of distance changes due to a change in at least one of the information related to the camera 120, the threshold value of the second assistance control may be changed.

[0115] According to the configuration of this modified example, even if the vehicle 10 to be operated does not change, if the sense of distance felt by the operator changes, the threshold value can be changed to appropriately assist remote operation.

[0116] (3) Modification 3 In the above-described embodiment, the second assistance control is exemplified as a control for notifying the vehicle 10 of its approach to an object around the vehicle 10, but is not limited to this. The second assistance control may be at least one of a control for notifying the vehicle 10 of its approach to an object around the vehicle 10, a control for braking the vehicle 10, a control for determining whether the vehicle 10 can move and notifying the determination result, and a control for presenting an overhead image showing the area around the vehicle 10.

[0117] For example, the assistance control unit 235 may perform control to brake the vehicle 10 (e.g., perform an emergency stop) when the distance from the remotely controlled vehicle 10 to an object around the vehicle 10 is shorter than, for example, a second assistance control threshold. In this case, the assistance control unit 235 may perform control to display a notification screen (assistance information) on the terminal device 30 notifying the operator of an emergency stop. Furthermore, the assistance control unit 235 may increase the threshold for the second assistance control when the perceived distance becomes "close" and decrease it when the perceived distance becomes "far." Therefore, braking, such as an emergency stop, is performed when the distance between the actual object and the vehicle 10 falls below the second assistance control threshold. Therefore, when the perceived distance becomes "close," the emergency stop is performed at a farther distance, thereby reducing unnecessary braking. Furthermore, when the perceived distance becomes "far," the emergency stop is performed at a closer distance within a range that does not fall below the lower limit. Therefore, the risk of a collision can be reduced even in situations where the distance to the object is unintentionally closed due to a change in the perceived distance.

[0118] For example, when the distance from the remotely controlled vehicle 10 to an object around the vehicle 10 is shorter than the second assistance control threshold, the assistance control unit 235 may determine whether the vehicle 10 can move (e.g., whether it can pass through) and may control the notification of the determination result. In this case, the assistance control unit 235 may control the terminal device 30 to display a notification screen (assistance information) that notifies the operator of the determination result. Furthermore, the assistance control unit 235 may increase the threshold for the second assistance control when the perceived distance becomes "close" and decrease it when the perceived distance becomes "far." Therefore, when the perceived distance to an object such as a wall becomes "close," the notification of whether it can pass through is made at an earlier stage, thereby reducing the operator's anxiety about whether the vehicle 10 can pass through.

[0119] For example, the assistance control unit 235 may perform control to present an overhead image (e.g., an around view) showing the area around the vehicle 10 when the distance from the remotely controlled vehicle 10 to an object around the vehicle 10 is shorter than, for example, a threshold value for the second assistance control. In this case, the assistance control unit 235 may perform control to display a presentation screen (assistance information) presenting the overhead image on the terminal device 30. Furthermore, the assistance control unit 235 may increase the threshold value for the second assistance control when the perceived distance becomes "close" and decrease it when the perceived distance becomes "far." Therefore, when the perceived distance to an object such as a narrow road or a wall becomes "close," the situation around the vehicle 10 can be confirmed using the overhead image at an earlier stage, thereby reducing the operator's anxiety.

[0120] (4) Modification 4 In the remote operation system 1 according to the above embodiment, the vehicle 10 transmits a support request to the remote operation assistance device 20 to request support through remote operation when the vehicle 10 is unable to autonomously travel, for example, when an obstacle is detected on the path. In this situation, the assistance control unit 235 may change the timing at which the support request is transmitted depending on the perceived distance. For example, the assistance control unit 235 may set the distance to the obstacle when switching to remote operation to a farther distance when the perceived distance is "close," and to a closer distance when the perceived distance is "far." In this case, the assistance control unit 235 may control the output of assistance information indicating the changed distance setting to the vehicle 10. Furthermore, the vehicle 10 may change the setting of the distance to the obstacle when switching to remote operation in accordance with the assistance information from the remote operation assistance device 20. Therefore, when the perceived distance becomes "close," the vehicle 10 switches to remote operation with the vehicle 10 stopped at a greater distance, thereby reducing the operator's anxiety at the start of remote operation.

[0121] (5) Variation 5 The assistance control unit 235 may adjust the threshold value of the second assistance control after the image for remote operation is switched depending on at least one of the surrounding environment of the vehicle 10 to be operated after the switch, the type of task, and the type of vehicle 10.

[0122] As an example, the assistance control unit 235 may increase the threshold value of the second assistance control more than usual when the surrounding environment of the vehicle 10 after the switch is an area with many people and objects around, such as a city. For example, when the sense of distance changes from "close" to "far" before and after the switch, the assistance control unit 235 may lower the threshold value of the second assistance control normally, but may increase it instead of lowering it when the area is an area with many people and objects around, such as a city. This allows the alert to be issued from a greater distance, thereby improving safety.

[0123] As an example, the assistance control unit 235 may increase the threshold value of the second assistance control more than usual when the surrounding environment of the vehicle 10 after the switch is dark or in bad weather. For example, when the sense of distance changes from "close" to "far" before and after the switch, the assistance control unit 235 may lower the threshold value of the second assistance control normally, but may increase it instead when it is dark or in bad weather. This accelerates the timing of the second assistance control in situations where it is difficult to see the surroundings, thereby improving safety.

[0124] As an example, the assistance control unit 235 may increase the threshold value of the second assistance control more than usual when the task type of the vehicle 10 after the switch is a task that involves getting close to people, such as a delivery service or a service using a communication robot. For example, when the sense of distance changes from "close" to "far" before and after the switch, the assistance control unit 235 may lower the threshold value of the second assistance control normally, but may increase it instead of lowering it when the task is a task that involves getting close to people. This makes it possible to speed up the timing of issuing an alert or making an emergency stop in the case of a task type that has a high risk of colliding with people if the sense of distance is incorrect, thereby improving safety.

[0125] As an example, if the type of task of the vehicle 10 after switching is a type in which visual recognition of the surroundings of the vehicle 10, such as traffic lights and signs, is important, the assistance control unit 235 may adjust at least one of the focal length, mounting height, imaging range, and the range of the imaging range to be used as the image for remote operation of the camera 120 so that the sense of distance becomes "close." In addition to controlling the sense of distance to be "close," the assistance control unit 235 may also reduce the threshold value of the second assistance control below normal or increase the distance to the obstacle when switching to remote operation. This makes it possible to prevent overlooking traffic lights, signs, etc.

[0126] Alternatively, if the type of task of the vehicle 10 after the switch is a type in which visual recognition of the surroundings of the vehicle 10, such as traffic lights and signs, is important, the assistance control unit 235 may adjust at least one of the focal length, mounting height, imaging range, and the range of the imaging range used for the remote control image, of the camera 120, so as to reduce a change in the sense of distance before and after switching of the image for remote operation. Here, reducing a change in the sense of distance before and after switching of the image for remote operation means keeping the sense of distance constant before and after the switch. This reduces the difference in how objects appear in the image before and after the switch.

[0127] As an example, the assistance control unit 235 may increase the threshold value of the second assistance control more than usual when the type of the vehicle 10 after the switch is a type indicating a large vehicle. For example, when the sense of distance changes from "close" to "far" before and after the switch, the assistance control unit 235 may lower the threshold value of the second assistance control normally, but may increase it instead when the type of the vehicle 10 is a large vehicle. This makes it possible to speed up the timing of alerts and emergency stops when operating a large vehicle that poses a high risk, thereby improving safety. In other words, it is possible to make the operator aware of the surrounding conditions and the like earlier in situations where the risk of an accident is high.

[0128] (6) Modification 6 In the remote control system 1 according to the above embodiment, a case has been exemplified in which control is performed to assist remote control depending on whether the sense of distance indicated by the distance sense information has changed and whether the sense of distance has changed to “far” or “near,” but this is not limiting.

[0129] As an example, after the image for remote control is switched, if the camera 120 is set or positioned so that the sense of distance is perceived as "far," the assistance control unit 235 may perform control to issue a notification (warning) not to get too close, regardless of the change in the sense of distance. In this way, the switching of the image for remote control can be used as an opportunity to call attention to the sense of distance.

[0130] As an example, the assistance control unit 235 may change the threshold value of the second assistance control by a magnitude corresponding to the amount of change in the sense of distance, regardless of whether the sense of distance has changed to "far" or "near." Furthermore, if the amount of change in the sense of distance is small, the assistance control unit 235 may not need to change the threshold value of the second assistance control. This allows the threshold value to be changed by an amount corresponding to the sense of distance, thereby achieving more appropriate assistance.

[0131] As an example, the assistance control unit 235 may set, for each operator, whether to change the threshold value of the second assistance control before and after switching of the remote operation image, based on the selection of, for example, the operator or the operator's manager. In other words, the assistance control unit 235 may not change the threshold value of the second assistance control for a specific operator even if the sense of distance changes. This makes it possible to realize control that does not change the threshold value for operators who may be confused by changing the threshold, such as operators who are skilled in image-based operations or operators who can properly grasp the sense of distance from images.

[0132] Note that, for an operator who has been set not to change the threshold, if it is estimated that emergency braking has occurred due to a misjudgment of distance, such as when sudden braking is performed while an object such as an obstacle is present in the vicinity, the assistance control unit 235 may issue a notification (notification) to the operator urging the operator to change the threshold of the second assistance control in accordance with the sense of distance. This prompts the operator to change the threshold in accordance with the sense of distance when it is estimated that there is a discrepancy in the sense of distance, thereby improving the safety of remote operation.

[0133] As an example, the assistance control unit 235 may not change the threshold value according to the sense of distance if the operation time before the image for remote operation is switched is shorter than a predetermined time, for example, that is determined in advance and stored in the storage unit 220. This reduces the risk of reducing convenience, such as causing confusion among the operator, for example, if the threshold value for the second assistance control is changed according to the sense of distance even though the image is switched before the operator has become accustomed to the sense of distance before the switch, and also reduces the risk of reducing safety, such as causing a discrepancy in the sense of distance due to the change in the threshold value.

[0134] As an example, the assistance control unit 235 does not need to change the threshold value according to the sense of distance for an operator whose experience in operating the vehicle 10 to be operated after the remote operation image is switched is greater than a predetermined standard, for example, that is determined in advance and stored in the storage unit 220. By changing the threshold value according to the sense of distance for an operator who is familiar with the sense of distance in the vehicle 10, it is possible to reduce the risk of a decrease in convenience, such as causing confusion among the operator, and also reduce the risk of a decrease in safety, such as a deviation in the sense of distance, caused by the change in the threshold value.

[0135] As an example, the assistance control unit 235 may restore the threshold value according to the sense of distance to the threshold value before the change if the operation time after the image for remote operation is switched exceeds a predetermined time, for example, that is determined in advance and stored in the storage unit 220. In other words, the assistance control unit 235 may change the threshold value for the second assistance control only for a certain period of time after the image for remote operation is switched. This allows the second assistance control to be performed based on the sense of distance perceived from the image for remote operation once the operator has become accustomed to the sense of distance after the switch, thereby improving convenience for the operator.

[0136] (7) Modification 7 In the remote control system 1 according to the above embodiment, the terminal device 30 notifies the user of closing the distance too much, speeding too fast, or approaching an object by displaying support information, but the present invention is not limited to this.

[0137] As an example, the support control unit 235 may notify the support information by sound rather than by display, thereby preventing a decrease in visibility of the image for remote operation that accompanies the display of the support information.

[0138] As an example, the assistance control unit 235 may perform control to notify (warn) the user not to increase the microphone volume too much so that, for example, when the perceived distance is "far," the volume of the microphone can be prevented from being increased too much so that the sound output from the speaker of the vehicle 10 via the microphone reaches a person who feels that the person is far away.

[0139] As an example, the assistance control unit 235 may perform control to notify (notify) the operator of a margin of safety, indicating that the distance feels close but there is still room to move closer. For example, the margin of safety may be determined by the ratio of the sense of distance information to the actual distance. This allows the operator to easily grasp how far the vehicle 10 can still move, even if the operator feels that the distance is "far."

[0140] As an example, the support control unit 235 may control the display of support information so that the greater the deviation in the sense of distance, i.e., the greater the change in the sense of distance before and after switching the image for remote operation, the stronger the warning, using the size, color, sound, etc. of the characters. This allows the operator to intuitively grasp the magnitude of the change in the sense of distance.

[0141] As an example, the assistance control unit 235 may control the display of assistance information to notify the operator of the difference in perceived distance before and after the remote operation image is switched. This notification may be achieved, for example, by changing the display mode of the background color according to the perceived distance when displayed on the terminal device 30. For example, the assistance control unit 235 may change the display mode so that if the perceived distance is "close," the background color is red, and the greater the difference in perceived distance, the darker the background color. For example, if the perceived distance is "far," the assistance control unit 235 may change the display mode so that the background color is blue, and the greater the difference in perceived distance, the darker the background color. This allows the operator to easily visualize the difference in perceived distance before and after the remote operation image is switched.

[0142] As an example, the assistance control unit 235 may perform control to notify the operator that the sense of distance is the same as the sense of distance of the previously operated vehicle 10 if the operator has experience operating the vehicle 10 after switching, based on the operator's past operation history stored in the storage unit 220, for example. This notification may be information such as the name, characteristics, and color of the previously operated vehicle 10, or information such as the operation location, the movement (travel) location of the vehicle 10 to be operated, and the date and time of the operation. This allows the operator to easily understand the change in the sense of distance based on his or her own experience.

[0143] As an example, the assistance control unit 235 may perform control to notify the operator of information indicating how the sense of distance in the image for remote operation after the change is different from that in the image before the change. For example, the assistance control unit 235 may cause the terminal device 30 to display whether an object at the same distance appears to be "closer" or "farther" than before the change. This allows the operator to easily grasp the trend in the change in the sense of distance.

[0144] (8) Modification 8 In the remote operation system 1 according to the above embodiment, the assistance control unit 235 may adjust at least one of the focal length, the mounting height, the imaging range, and the range of the imaging range used as the image for remote operation of the camera 120 in accordance with a change in the sense of distance before and after switching of the image for remote operation, so as to reduce the change. Here, reducing the change in the sense of distance before and after switching of the image for remote operation means keeping the sense of distance constant before and after the switch.

[0145] As an example, the assistance control unit 235 may change the focal length or the shooting mode of the camera 120 so that the angle of view (imaging range) of the camera 120 remains constant before and after switching the image for remote operation.

[0146] As an example, the assistance control unit 235 may change the angle of view (image capture range) of the camera 120 so that the field of view of the image for remote operation remains constant before and after the image for remote operation is switched.

[0147] As an example, the assistance control unit 235 may crop a portion of the image acquired by the camera 120 or change the crop range so that the field of view of the image for remote control remains constant before and after switching of the image for remote control.

[0148] As an example, the assistance control unit 235 may change the height of the camera 120 or change to a camera 120 with a different mounting height so that the field of view of the image for remote control remains constant before and after switching of the image for remote control, or so that the height of the camera 120 remains constant.

[0149] For example, the support control unit 235 may permit the start of remote control after the adjustment according to the change in the sense of distance is completed. In other words, the support control unit 235 may be configured not to permit the start of remote control during the adjustment according to the change in the sense of distance.

[0150] These configurations can reduce the change in the sense of distance before and after switching of the image for remote operation, thereby improving convenience for the operator and improving the safety of remote operation.

[0151] (9) Variation 9 In the above-described embodiment, the camera 120 is positioned so as to be able to capture an image of the area in front of the vehicle 10, and the tilt information is calculated using the image of the area in front of the vehicle 10. However, this is not limiting, and the tilt information may be calculated using, for example, an image of the area behind the vehicle 10 or an image of either the left or right side of the vehicle 10.

[0152] (10) Modification 10 In the above-described embodiment, each vehicle 10 calculates the tilt information, but this is not limiting, and the tilt information may be calculated by, for example, the remote operation assistance device 20. In this embodiment, the remote operation assistance device 20 receives speed information and an image from each vehicle 10, and can calculate the tilt information for each vehicle 10 based on the received speed information and image.

[0153] (11) Modification 11: As the vehicle 10 used in the remote operation system 1 of the above-described embodiment, various types of moving bodies configured to be movable in response to at least remote operation by an operator can be appropriately used. The vehicle 10 may be, for example, a four-wheeled vehicle or a two-wheeled vehicle. Furthermore, for example, the vehicle 10 may be an automatic guided vehicle (AGV), or various types of robots such as construction machinery, agricultural machinery, or drones.

[0154] These mobile objects are not limited to those that transport people, but may also transport objects other than people, and may also provide specific services other than transportation.

[0155] As an example, the remote control system 1 according to the embodiment may be applied to assist in remote control of an autonomous mobile sales robot. This autonomous mobile sales robot may sell products or services to customers. When the perceived distance during remote control of the autonomous mobile sales robot becomes "far," the assistance control unit 235 may reduce the threshold of the second assistance control within a range not below a lower limit, warn the operator not to speed up too much, change (e.g., speed up) the timing of the stop before switching to remote control, warn the operator not to increase the volume of the voice output too much, or limit the volume of the voice output. This reduces the risk of collisions or the customer feeling pressured or uneasy when the sales robot moves closer to a customer in connection with a sales or sales-related operation, for example, if the perceived distance becomes "far" and the actual distance between the sales robot and the customer becomes too close. Furthermore, for example, the operator's voice or notification sound output from a speaker mounted on the sales robot may be prevented from speaking too loudly to the customer because the perceived distance is "far." Furthermore, when the sense of distance becomes "close," the assistance control unit 235 may increase the threshold value of the second assistance control, notify the driver that there is still room to get closer, automatically adjust the angle of view of the camera 120, or request the customer to move away from the camera 120. This prevents the driver from feeling that the customer is closer than they actually are and stopping unnecessarily far away, or from getting too close and cutting off the customer's face.

[0156] Similarly, the remote control system 1 according to the embodiment may be applied to a communication robot used in nursing care facilities, hospitals, etc., for communication such as conversation between an operator and a target user.

[0157] As an example, the remote operation system 1 according to the embodiment may be applied to assist in remote operation of a robot, such as a humanoid robot, that grasps or carries a target object. Examples of such robots include robots used for loading and unloading in factories and distribution centers, and robots used for agricultural work. The assistance control unit 235 may perform control to convert a remote operation image for remotely operating the robot so that the perceived distance matches the actual distance. This image conversion corresponds to converting the image so that the field of view of the remote operation image remains constant before and after switching the remote operation image. Here, the operational feel refers to, for example, the operator's perception of the amount of movement of the operation target relative to the amount of operation of the operating unit that instructs the remote operation operation content. This can suppress changes in the operational feel caused by changes in the perceived distance.

[0158] (12) Modification 12 In the remote operation system 1 of the above-described embodiment, a sense of distance may be determined in advance for each vehicle 10, and accordingly, a threshold value related to control for assisting remote operation of the vehicle 10 may also be determined in advance. In such a case, when a change in the vehicle 10 and the image for remote operation occurs, the assistance control unit 235 may execute control for assisting remote operation of the vehicle 10 using a threshold value determined in advance for the switched vehicle 10. The predetermined sense of distance and threshold value for each vehicle 10 may be stored in the internal memory of the control device 140 of each vehicle 10, or may be stored in the internal memory of the remote operation assistance device 20. Alternatively, the sense of distance for each vehicle 10 and information indicating the correspondence between the sense of distance and the threshold value may be determined in advance and stored in one of the internal memories.

[0159] (13) Modification 13 In the remote operation system 1 of the above-described embodiment, the threshold value may be determined based on the surrounding environment and the vehicle 10, for example, based on an empirical rule of thumb, such as whether it is easier to operate a delivery robot in a city if an alert is sounded at a certain distance. In such a case, when the vehicle 10 and the remote operation image are switched, the assistance control unit 235 may execute control to assist the remote operation of the vehicle 10 using a predetermined threshold value for the switched vehicle 10, rather than changing the threshold value based on the sense of distance. The predetermined threshold value for each surrounding environment and / or vehicle 10 may be stored in the internal memory of the control device 140 of each vehicle 10 or in the internal memory of the remote operation assistance device 20. Alternatively, information indicating the correspondence between the surrounding environment and / or vehicle 10 and the threshold value may be predetermined and stored in one of the internal memories. Alternatively, parameters may be determined in advance for each surrounding environment and vehicle 10, and these parameters and a predetermined calculation formula for calculating the threshold value using these parameters may be stored in one of the internal memories.

[0160] (14) Modification 14 In the assistance process (see FIG. 12) for the remote operation system 1 according to the embodiment described above, the actual distance when the perceived distance becomes closer or farther may be taken into consideration. FIGS. 15 to 17 are diagrams for explaining an example of the assistance process according to this modification.

[0161] As an example, in the assistance process, if the change in the sense of distance information before and after the switching satisfies the assistance criterion (step S203: Yes), the remote operation assistance device 20 may further determine whether the sense of distance is greater than the actual distance for each of the remote operation images before and after the switching, based on the sense of distance information. In other words, the remote operation assistance device 20 according to this modification may further determine whether the distance to an arbitrary position on the image, as perceived by the operator remotely operating the vehicle 10 while watching the video captured by the camera 120 (the image for remote operation), is greater than the actual distance to the position.

[0162] The determination of whether the perceived distance in the image for remote operation is greater than the actual distance may be performed, for example, by the control device 140 of the vehicle 10. In this case, the determination result of whether the perceived distance is greater than the actual distance may be output from the vehicle 10 to the remote operation assistance device 20 together with the image for remote operation. The determination result of whether the perceived distance is greater than the actual distance may also be information included in the distance perception information.

[0163] For example, the determination of whether the perceived distance of the remote control image is greater than the actual distance may be based on at least one of the following information: a motion vector, tilt information, focus information, mounting height, size of the imaging range (e.g., angle of view), size of the area within the imaging range used as the remote control image (e.g., field of view, crop range), depth information based on the remote control image, and shooting mode. For example, if the motion vector, tilt indicated by the tilt information, or size of the imaging range is equal to or greater than a predetermined threshold, the remote control image is determined to be one in which the perceived distance is greater than the actual distance. For example, if the focal length of the camera 120 indicated by the focus information, the mounting height, or the minimum distance indicated by the depth information is less than a predetermined threshold, the remote control image is determined to be one in which the perceived distance is greater than the actual distance. For example, if the shooting mode of the camera 120 is a wide-angle mode, the remote control image is determined to be one in which the perceived distance is greater than the actual distance. Note that the threshold values ​​of each piece of information used to determine whether the perceived distance of the remote control image is greater than the actual distance may be predetermined and stored in the internal memory of the control device 140 of the vehicle 10 or the remote operation assistance device 20.

[0164] Here, the support process that takes into account the actual distance relative to the perceived distance will be described in more detail.

[0165] The processing described below corresponds to, for example, the processing of step S205, step S206, or step S209 of the assistance processing according to the above-described embodiment. For example, the alert to be strengthened / weakened in this modification is, for example, a warning in the first assistance control (see FIG. 9 ), but it may also be a warning in the second assistance control. Note that when the sense of distance is equivalent to the actual distance, processing similar to step S205, step S206, or step S209 of the assistance processing according to the above-described embodiment may be performed.

[0166] In this modification, strengthening / weakening the alert may mean emphasizing / suppressing the alert, as in the above-described modification 7. As an example, strengthening / weakening the alert may be achieved by changing the display mode of the warning screen (support information), such as changing the display color, size, or presence or absence of a frame. As an example, strengthening / weakening the alert may be achieved by changing the notification mode, such as the presence or absence of a voice notification (support information), or the volume or tone of the notification.

[0167] FIG. 15 illustrates an example of processing when the "actual distance" is smaller than the "perceived distance" for both the images for remote operation before and after switching.

[0168] First, consider a case where the "actual distance" is smaller than the "perceived distance" both before and after switching, and the perceived distance changes from "close" to "far" between the two. For example, consider a case where the "actual distance: 10 m" and "perceived distance: 20 m" before switching change to "actual distance: 20 m" and "perceived distance: 30 m" after switching. As shown in FIG. 15 , if the "actual distance" is smaller than the "perceived distance" both before and after switching, and the perceived distance changes from "close" to "far" between the two, the perceived distance becomes "far" even though the "actual distance" is smaller than the "perceived distance," making it more likely that speeding and / or overtaking will occur after switching. For this reason, the assistance control unit 235 may issue or strengthen an alert (assistance information) warning about speeding and overtaking if the "actual distance" is smaller than the "perceived distance" both before and after switching, and the perceived distance changes from "close" to "far" between the two. In this case, the assistance control unit 235 may set the threshold value for the second assistance control to be smaller than normal, or even smaller, as in the processing of S206.

[0169] Next, consider a case where the "actual distance" is smaller than the "perceived distance" both before and after the switching, and the perceived distance changes from "far" to "close" between the two. For example, the "actual distance: 20 m" and "perceived distance: 30 m" before the switching change to the "actual distance: 10 m" and "perceived distance: 20 m" after the switching. As shown in FIG. 15 , when the "actual distance" is smaller than the "perceived distance" both before and after the switching, and the perceived distance changes from "far" to "close" between the two, the "actual distance" is smaller than the "perceived distance" and the perceived distance becomes "close," which makes the operator feel closer after the switching and increases anxiety. For this reason, the assistance control unit 235 may increase the threshold value of the second assistance control from the normal threshold, as in the processing of S205, when the "actual distance" is smaller than the "perceived distance" both before and after the switching, and the perceived distance changes from "far" to "close" between the two.

[0170] FIG. 16 illustrates an example of processing when the "actual distance" is greater than the "perceived distance" for both the images for remote operation before and after switching.

[0171] First, consider a case where the "actual distance" is greater than the "perceived distance" both before and after switching, and the perceived distance changes from "close" to "far" between the two. For example, consider a case where the "actual distance: 20 m" and "perceived distance: 10 m" before switching change to "actual distance: 30 m" and "perceived distance: 20 m" after switching. As shown in FIG. 16 , when the "actual distance" is greater than the "perceived distance" both before and after switching, and the perceived distance changes from "close" to "far" between the two, the perceived distance becomes "far," making speeding and / or overtaking more likely to occur after switching. However, the "actual distance" is greater than the "perceived distance." Therefore, when the "actual distance" is greater than the "perceived distance" both before and after switching, and the perceived distance changes from "close" to "far" between the two, the assistance control unit 235 may weaken the alert (assistance information) warning about speeding and overtaking, or may not issue an alert at all. In this case, the assistance control unit 235 may not change the threshold value of the second assistance control, or may reduce the threshold value of the second assistance control, as in the processing of S206.

[0172] Next, consider a case where the "actual distance" is greater than the "perceived distance" both before and after the switching, and the perceived distance changes from "far" to "close" between the two. For example, the "actual distance: 30 m" and "perceived distance: 20 m" before the switching change to "actual distance: 20 m" and "perceived distance: 10 m" after the switching. As shown in FIG. 16 , when the "actual distance" is greater than the "perceived distance" both before and after the switching, and the perceived distance changes from "far" to "close" between the two, the perceived distance becomes "closer," which may cause anxiety for the operator. However, the "actual distance" is greater than the "perceived distance." Therefore, when the "actual distance" is greater than the "perceived distance" both before and after the switching, and the perceived distance changes from "far" to "close" between the two, the assistance control unit 235 may display the margin of safety, for example, as in the above-described variant 7. Note that in this case, the assistance control unit 235 may not change the threshold value of the second assistance control, or may increase the threshold value of the second assistance control as in the processing of S205.

[0173] FIG. 17 illustrates an example of processing when the magnitude relationship between the "actual distance" and the "perceived distance" changes before and after switching.

[0174] First, consider a case where the "actual distance" is smaller than the "perceived distance" before switching, and is larger than the "perceived distance" after switching, and the perceived distance changes from "close" to "far" before switching. For example, consider a case where the "actual distance: 10 m" and "perceived distance: 20 m" before switching change to "actual distance: 40 m" and "perceived distance: 30 m" after switching. As shown in FIG. 17 , when the "actual distance" is smaller than the "perceived distance" before switching, and is larger than the "perceived distance" after switching, and the perceived distance changes from "close" to "far" before switching, the perceived distance becomes "far" after switching, making it more likely that speeding and / or overtaking will occur, and the "actual distance" also becomes larger than the "perceived distance." For this reason, if the "actual distance" is smaller than the "perceived distance" before switching and the "actual distance" is larger than the "perceived distance" after switching, and the perceived distance changes from "close" to "far" before and after switching, the assistance control unit 235 may weaken the alert (assistance information) warning about speeding or being too close, or may not issue an alert at all. In this case, the assistance control unit 235 may not change the threshold value of the second assistance control, or may reduce the threshold value of the second assistance control as in the processing of S206.

[0175] Next, consider a case where the "actual distance" is smaller than the "perceived distance" before switching and is larger than the "perceived distance" after switching, and the perceived distance changes from "far" to "close" before switching. For example, consider a case where the "actual distance: 10 m" and "perceived distance: 30 m" before switching change to "actual distance: 40 m" and "perceived distance: 20 m" after switching. As shown in FIG. 17 , when the "actual distance" is smaller than the "perceived distance" before switching and is larger than the "perceived distance" after switching, and the perceived distance changes from "far" to "close" before switching, the perceived distance becomes "close," which can easily make the operator feel uneasy, while the "actual distance" is larger than the "perceived distance." For this reason, if the "actual distance" is smaller than the "perceived distance" before switching and the "actual distance" is larger than the "perceived distance" after switching, and the perceived distance changes from "far" to "close" before and after switching, the assistance control unit 235 may display the margin of safety, for example, as in the above-described modified example 7. In this case, the assistance control unit 235 may not change the threshold value of the second assistance control, or may increase the threshold value of the second assistance control as in the processing of S205.

[0176] Next, consider a case where the "actual distance" is greater than the "perceived distance" before switching and is smaller than the "perceived distance" after switching, and the perceived distance changes from "close" to "far" before switching. For example, consider a case where the "actual distance: 30 m" and "perceived distance: 10 m" before switching change to "actual distance: 20 m" and "perceived distance: 30 m" after switching. As shown in FIG. 17 , when the "actual distance" is greater than the "perceived distance" before switching and is smaller than the "perceived distance" after switching, and the perceived distance changes from "close" to "far" before switching, the perceived distance becomes "far" after switching, making it more likely that speeding and / or overtaking will occur, and the "actual distance" becomes smaller than the "perceived distance." For this reason, if the "actual distance" is greater than the "perceived distance" before switching and the "actual distance" is less than the "perceived distance" after switching, and the perceived distance changes from "close" to "far" before and after switching, the assistance control unit 235 may issue or strengthen an alert (assistance information) warning about speeding or getting too close. Note that in this case, the assistance control unit 235 may make the threshold value of the second assistance control smaller than normal, or even smaller, as in the processing of S206.

[0177] Next, consider a case where the "actual distance" is greater than the "perceived distance" before switching and is smaller than the "perceived distance" after switching, and the perceived distance changes from "far" to "close" before switching. For example, consider a case where the "actual distance: 20 m" and "perceived distance: 30 m" before switching change to "actual distance: 10 m" and "perceived distance: 20 m" after switching. As shown in FIG. 17 , when the "actual distance" is greater than the "perceived distance" before switching and is smaller than the "perceived distance" after switching, and the perceived distance changes from "far" to "close" before switching, the perceived distance becomes "close," which makes the operator more likely to feel uneasy, and the "actual distance" also becomes greater than the "perceived distance." Therefore, if the "actual distance" is greater than the "perceived distance" before the switch and the "actual distance" is less than the "perceived distance" after the switch, and the perceived distance changes from "far" to "close" before and after the switch, the assistance control unit 235 may increase the threshold value of the second assistance control larger than normal, or even larger, as in the processing of S205.

[0178] (15) Modification 15 Although Modification 14 illustrates an example in which the sense of distance is considered based on the actual distance, this is not limiting. The sense of distance of an arbitrary vehicle 10 (reference vehicle) may be used as the reference (reference value) for the sense of distance.

[0179] As an example, the reference vehicle may be the vehicle 10 that was the target of remote operation by the operator immediately before.

[0180] As an example, the reference vehicle may be any vehicle 10 previously determined by an administrator or operator, such as a vehicle 10 that operates in a highly difficult work content or work environment, a large number of vehicles 10, or a vehicle 10 with an average distance size.

[0181] As an example, the reference vehicle may be determined for each operator, and may be, for example, a vehicle 10 that the operator has a lot of experience operating or a vehicle 10 that the operator feels is easy to grasp the sense of distance.

[0182] The reference vehicle setting and / or the sense of distance (reference value) may be determined in advance and stored in the internal memory of the remote operation assistance device 20, for example.

[0183] As an example, when the remote operation support device 20 according to this modification switches the remote operation image, the remote operation support device 20 may display, on the terminal device 30, information (support information) indicating the sense of distance after the switch relative to a reference value. The information indicating the sense of distance after the switch relative to the reference value may be a classification indicating whether or not there is a difference (change) in the sense of distance after the switch relative to the reference value, the amount of change, or the degree of change. For example, the remote operation support device 20 may display, on the terminal device 30, support information such as "The sense of distance is closer than before" or "The sense of distance is farther than before."

[0184] As an example, when the remote operation support device 20 according to this modification switches the image for remote operation, the remote operation support device 20 may adjust at least one of the focal length, the mounting height, the size of the imaging range (e.g., the angle of view), the size of the range within the imaging range to be used as the image for remote operation (e.g., the field of view, the crop range), and the shooting mode so that the sense of distance after the switch matches a reference value. For example, when switching from a "wide-angle" image for remote operation to a "telephoto" image for remote operation, the remote operation support device 20 may generate an image for remote operation by converting the image from the camera 120, such as by cropping, so that the sense of distance after the switch matches or approaches the reference value.

[0185] (16) Modification 16 Here, a modification regarding the remote operation in the remote operation system 1 of the above-described embodiment will be described. In the above-described embodiment, the remote operation system 1 according to the present disclosure has been described using an example in which an operator in charge of multiple vehicles 10 uses the terminal device 30 to simultaneously monitor two or more vehicles 10, or switches between target vehicles 10 to monitor and remotely operate them sequentially.

[0186] In this situation, in the remote control system 1, when a request for assistance through remote control (remote control request) is issued from the vehicle 10, it is possible to assume that an operator will be assigned to respond to the request (remote control) to one of the operators in the remote control room.

[0187] Such operator assignment may be performed by an administrator who manages multiple operators in a remote control room using the terminal device 30, or may be performed automatically by the terminal device 30 or the remote operation support device 20 according to predetermined conditions that are determined in advance and stored in an internal memory. Note that automatically assigning an operator may mean permitting the terminal device 30 of the assigned operator to perform remote operation. Alternatively, automatically assigning an operator may mean proposing permission for remote operation to the terminal device 30 of the administrator or each operator, and / or permitting remote operation if the proposal is approved.

[0188] For example, when a remote control request occurs, the remote operation support device 20 according to this variant may perform operator assignment, in which an operator who satisfies certain conditions among multiple operators in a remote control room is preferentially assigned to the vehicle 10 that issued the request.

[0189] As an example, the predetermined condition includes that, before the current remote control request is generated, another vehicle 10 at the same perceived distance as the vehicle 10 that issued the current remote control request is being remotely controlled. Here, "before the remote control request is generated" may be immediately before switching. In other words, the predetermined condition may be that the perceived distance is the same before and after switching (the perceived distance does not change).

[0190] Alternatively, "before a remote control request is generated" may be a period from the time the target remote control request is generated to a predetermined period that is previously stored in the internal memory. In other words, the predetermined condition may be that the user has had experience operating a remote control of the vehicle 10 at the same distance during the immediately preceding predetermined period.

[0191] Note that "before a remote control request is generated" does not necessarily mean a predetermined period of time before the target remote control request is generated. In other words, the predetermined condition is not limited to a predetermined period of time immediately before the target remote control request, but may also mean having experience of remotely controlling the vehicle 10 at the same distance in the past.

[0192] Note that having such operation experience may mean that the operation time and / or distance of a remote control with the same sense of distance is longer than a predetermined time and / or distance that is predetermined and stored in an internal memory, for example, assuming that the operator becomes accustomed to the sense of distance. Note that the predetermined period and / or the predetermined time may be determined for each vehicle 10, sense of distance, operator, operation experience, or any combination thereof. For example, the higher the risk of an accident occurring, the shorter the predetermined period or the longer the predetermined time or the predetermined distance may be set. For example, the longer the operator's operation experience, the longer the predetermined period may be set.

[0193] Note that the same sense of distance means, for example, that the sense of distance information matches, but is not limited to this. For example, the sense of distance information may be equivalent. Furthermore, there may be cases where the target vehicle 10 cannot be assigned to an operator who satisfies the predetermined condition of the same sense of distance, such as when there is no corresponding operator, the operator is absent, or the operator is currently handling another remote control operation. Therefore, the predetermined condition may be that, before the current remote control request is generated, the operator remotely controlled a vehicle 10 with a sense of distance similar to that of the vehicle 10 that issued the current remote control request. For example, if the remote operation assistance device 20 according to this modification cannot assign the target vehicle 10 to an operator who satisfies the predetermined condition of the same sense of distance, the remote operation assistance device 20 may perform operator assignment by preferentially assigning an operator who remotely controlled a vehicle 10 with similar sense of distance information indicating a sense of distance.

[0194] In this way, the remote operation system 1 according to this modification preferentially assigns an operator who is familiar with the sense of distance of the vehicle 10 that issued the remote control request. This allows the operator to perform remote control in a situation where there is little or no change in the sense of distance before and after switching, or with a sense of distance that is highly familiar to the operator. This improves the safety and efficiency of remote monitoring and control.

[0195] (17) Modification 17 Here, a modification regarding the remote operation in the remote operation system 1 of the above-described embodiment will be described. In the above-described modification 16, an example is given in which a remote control request is assigned to a plurality of operators.

[0196] In this situation, when multiple people provide remote support such as remote monitoring and control of multiple vehicles, the remote operation system 1 may be configured to assign an operator to each operator in the remote control room to provide the vehicle 10 that will be the target of remote support (remote monitoring) regardless of a remote control request.

[0197] For example, in the remote operation support device 20 according to this variant, in operator assignment in which multiple vehicles 10 to be monitored are assigned to multiple operators in a remote control room, the vehicles 10 may be assigned so that the sense of distance for each operator is the same or similar, i.e., so that the discrepancy in the sense of distance is small.

[0198] In this way, in the remote operation system 1 according to this modification, vehicles 10 with the same or similar sense of distance are assigned to an operator in charge of remote support (remote monitoring) of two or more vehicles 10. This allows the operator to target vehicles 10 with the same or similar sense of distance when remotely monitoring and controlling multiple vehicles 10, thereby improving safety and efficiency.

[0199] The above-described embodiment can be arbitrarily combined with at least one of the above-described modified examples. Also, two or more of the above-described modified examples can be arbitrarily combined.

[0200] REFERENCE SIGNS LIST 1 Remote operation system 10 Vehicle 20 Remote operation assistance device 30 Terminal device 141 Distance sense information calculation unit 142 Position information acquisition unit 143 Image acquisition unit 144 Travel control unit 145 Assistance request transmission unit 146 Operation information reception unit 231 Distance sense information reception unit 232 Assistance request reception unit 233 Acquisition unit 234 Determination unit 235 Assistance control unit 236 Remote information transmission / reception unit 341 Information reception unit 342 Display control unit 343 Operation information transmission unit

Claims

1. A remote operation support method executed by a remote operation support device that supports a remote operation in which an operator remotely operates the moving body based on an image for remote operation captured by a camera mounted on the moving body, wherein when the distance from the moving body that is the target of the remote operation to an object around the moving body is shorter than a threshold value, support control for supporting the remote operation is executed; when the image for remote operation is switched, distance information indicating the operator's sense of distance with respect to the distance to an object in the image for remote operation before and after the switching is compared; when the comparison result of the distance information satisfies a predetermined support criterion, the threshold value in the remote operation before and after the switching is changed. Remote operation support method.

2. When the sense of distance indicated by the distance information becomes closer before and after the switching, the threshold value is increased. The remote operation support method according to claim 1.

3. When the sense of distance indicated by the distance information becomes farther before and after the switching, the threshold value is decreased within a range exceeding a lower limit value predetermined based on safety. The remote operation support method according to claim 1.

4. The threshold value after the switching is adjusted according to at least any one of the surrounding environment of the moving body that is the target of the remote operation after the switching, the type of task, and the type of the moving body. The remote operation support method according to any one of claims 1 to 3.

5. For each operator, it is set whether to change the threshold value before and after the switching. The remote operation support method according to any one of claims 1 to 3.

6. When the moving body performs emergency braking and the setting is such that the threshold value is not changed before and after the switching, a notification is given to select a setting for changing the threshold value before and after the switching. The remote operation support method according to claim 5.

7. When the operation time of the remote operation before the switching is shorter than a predetermined time, or when the operation experience regarding the moving body that is the target of the remote operation after the switching is more than a predetermined criterion, the threshold value is not changed before and after the switching. The remote operation support method according to any one of claims 1 to 3.

8. The change of the threshold value is performed only for a certain period of time. The remote operation support method according to any one of claims 1 to 3.

9. The distance feeling information is: inclination information indicating an inclination which is a ratio of a change in the speed of the moving body and a change in a motion vector of a peripheral region of the moving body imaged by the camera, information regarding the camera, or depth information based on an image for remote operation. The remote operation support method according to any one of claims 1 to 3.

10. The support control includes control for notifying approach to surrounding objects, control for braking the moving body, control for determining whether the moving body can move and notifying the determination result, or control for presenting an overhead image showing an overhead view of the periphery of the moving body. The remote operation support method according to any one of claims 1 to 3.

11. When the comparison result of the distance feeling information satisfies a predetermined support criterion, at least one of the focal length of the camera, the mounting height, the imaging range, and the range used as the image for remote operation among the imaging range is adjusted so that the distance feeling becomes constant before and after the switching. The remote operation support method according to any one of claims 1 to 3.

12. The switching of the image for remote operation is based on switching of at least one of the moving body which is the object of the remote operation and the camera in the moving body. The remote operation support method according to any one of claims 1 to 3.

13. When the comparison result of the distance feeling information satisfies a predetermined support criterion, support information for notifying that the distance feeling changes before and after the switching is output. The remote operation support method according to any one of claims 1 to 3.

14. A remote operation support device includes: a support control unit that executes support control for assisting the remote operation when the distance from a moving body, which is the target of the remote operation that an operator remotely operates based on an image for remote operation captured by a camera mounted on the moving body, to an object around the moving body is shorter than a predetermined threshold; and a determination unit that compares distance information indicating the operator's sense of distance with respect to the distance to an object in the image for remote operation before and after the switching when the image for remote operation is switched. The support control unit changes the threshold in the remote operation before and after the switching when the comparison result of the distance information satisfies a predetermined support criterion.

15. A program for causing a computer to execute: executing support control for assisting the remote operation when the distance from a moving body, which is the target of the remote operation that an operator remotely operates based on an image for remote operation captured by a camera mounted on the moving body, to an object around the moving body is shorter than a predetermined threshold; comparing distance information indicating the operator's sense of distance with respect to the distance to an object in the image for remote operation before and after the switching when the image for remote operation is switched; and changing the threshold in the remote operation before and after the switching when the comparison result of the distance information satisfies a predetermined support criterion.

Images