Remote operation assistance method, remote operation assistance device, and program

Abstract

Provided are a remote operation assistance method, a remote operation assistance device, and a program with which it is possible to appropriately assist remote operation by an operator. A remote operation assistance method of the present disclosure is a remote operation assistance method performed by a remote operation assistance apparatus that assists remote operation of a vehicle, and includes an acquisition step and an assistance control step. In the acquisition step, relationship information indicating a relationship between a speed of a vehicle and a motion vector of a peripheral region of the vehicle captured by a camera mounted on the vehicle is acquired. In the assist control step, control for assisting the remote operation of the vehicle is performed on the basis of the relationship information acquired by the acquisition step.

Description

Technical Field

[0001] This disclosure relates to a remote operation assistance method, a remote operation assistance device, and a program. Background Technology

[0002] In recent years, various services based on autonomous vehicles have been put into practical use, and the development of remote control systems capable of remotely monitoring or operating these vehicles is also progressing. In these remote control systems, when an autonomous vehicle makes a request for remote operation assistance, the operator in the remote control room remotely operates the vehicle while observing images captured by a camera mounted on the autonomous vehicle, thereby enabling assistance such as moving the autonomous vehicle.

[0003] For example, a technique is known in which the visual effects of an image representing the surrounding area of ​​a vehicle are altered according to the vehicle's speed, thereby giving the operator a sense of speed corresponding to the vehicle's speed.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2014-71776 Summary of the Invention

[0007] The problem the invention aims to solve

[0008] However, in existing technologies, such as when remotely operating different types of vehicles, the range of the surrounding area captured by the camera will vary depending on factors such as the field of view or installation position of the camera mounted on the vehicle, thus changing the perceived speed. Therefore, this could potentially affect the operator's remote operation.

[0009] The purpose of this disclosure is to provide a remote operation assistance method, remote operation assistance device, and program that can appropriately assist an operator in remote operation.

[0010] Solution for solving the problem

[0011] To achieve the above objectives, the remote operation assistance method disclosed herein is a remote operation assistance method executed by a remote operation assistance device for assisting remote operation of a vehicle. The remote operation assistance method includes: an acquisition step, acquiring relational information representing the relationship between the speed of the vehicle and motion vectors of the vehicle's surrounding area captured by a camera mounted on the vehicle; and an assistance control step, performing control to assist remote operation of the vehicle based on the relational information acquired through the acquisition step.

[0012] The effects of the invention

[0013] According to this disclosure, remote operation by the operator can be appropriately assisted. Furthermore, the effects described herein are not necessarily limited and may be any of the effects described in this specification. Attached Figure Description

[0014] Figure 1 This is a diagram illustrating an example of the outline structure of a remote operating system in an implementation.

[0015] Figure 2 This is a diagram illustrating an example of the structure of each of the multiple devices included in a remote operating system of an embodiment.

[0016] Figure 3 This is a diagram illustrating an example of the hardware structure of the control device according to an embodiment.

[0017] Figure 4 This is an example of an image captured by a camera mounted on a vehicle in an embodiment.

[0018] Figure 5 This is a diagram illustrating an example of the correspondence between the speed of a vehicle in an embodiment and the magnitude of the motion vector of the surrounding area.

[0019] Figure 6 This is a diagram illustrating an example of slope information corresponding to each vehicle ID in the implementation.

[0020] Figure 7 This is a diagram illustrating an example of the peripheral area in an image captured by a camera in wide-angle mode according to an embodiment.

[0021] Figure 8 This is a diagram illustrating an example of the peripheral region in an image captured by a camera in telephoto mode according to an embodiment.

[0022] Figure 9 This is a diagram illustrating an example of the peripheral area in an image captured by a camera at a downward angle according to the embodiment.

[0023] Figure 10 This is a diagram illustrating an example of the peripheral area in an image captured by a camera at an elevation angle according to the embodiment.

[0024] Figure 11 This is a diagram illustrating an example of the correspondence between the speeds of the two vehicles and the magnitudes of the motion vectors of the surrounding area in an embodiment.

[0025] Figure 12 This is an example of a warning screen shown in an implementation method.

[0026] Figure 13This is a timing diagram illustrating an example of the operation process of a remote operating system when a vehicle that is the object of remote operation in an embodiment is switched.

[0027] Figure 14 This is a flowchart illustrating an example of vehicle movement when calculating slope information in an implementation method.

[0028] Figure 15 This is a flowchart illustrating an example of the operation of a remote operation assistance device when an assistance request is received in an embodiment.

[0029] Figure 16 This is a diagram illustrating an example of slope information corresponding to each combination of vehicle ID and camera field of view for the same variant.

[0030] Figure 17 This is a diagram illustrating an example of slope information corresponding to each combination of vehicle ID and location for the same variant.

[0031] Figure 18 This is a diagram showing an example of the motion vectors of the surrounding area corresponding to each vehicle ID in the variant example.

[0032] Figure 19 This is a diagram illustrating an example of the correspondence between the magnitude of the motion vectors of the respective peripheral regions of two vehicles and the vehicle's speed. Detailed Implementation

[0033] The remote operation assistance method, remote operation assistance device, and program involved in the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0034] Figure 1 This is a diagram illustrating an example of the outline structure of the remote operating system 1 according to this embodiment. Figure 1 As shown, the remote operating system 1 includes multiple (in Figure 1 In this example, there are two vehicles, 10A and 10B, a remote operation assistance device 20, and a terminal device 30. Figure 1 The diagram illustrates two vehicles, 10A and 10B, but is not limited to these. The number of vehicles 10 included in the remote operating system 1 can be arbitrarily changed according to design conditions, etc. In the following description, vehicles 10A and 10B are simply referred to as "vehicle 10" without distinguishing them from each other. The same reference numerals are used to label the same elements constituting vehicle 10.

[0035] exist Figure 1 In the example, the vehicle 10, the remote operation assistance device 20, and the terminal device 30 can be interconnected, for example, via a network 40 such as the Internet.

[0036] Vehicle 10 is an autonomous vehicle used for providing various services. Remote operation assistance device 20 is a device that assists in the remote operation of vehicle 10. Terminal device 30 is a device operated by an operator located in a remote control room. In the remote operating system 1 of this embodiment, when vehicle 10 becomes unable to drive autonomously (e.g., when an obstacle is detected on the road), it sends an assistance request to remote operation assistance device 20 requesting remote operation assistance. Upon receiving the assistance request from vehicle 10, remote operation assistance device 20 sends a remote operation request to terminal device 30 requesting remote operation from the operator. The operator of terminal device 30, upon receiving the remote operation request, can operate terminal device 30 while observing images captured by a camera mounted on vehicle 10, thereby remotely operating vehicle 10 and providing assistance such as moving vehicle 10.

[0037] Figure 2 This is a diagram illustrating an example of the structure of each device included in the remote operating system 1, such as the vehicle 10, the remote operation assistance device 20, and the terminal device 30. Below, referring to... Figure 2 To explain the structure of each device in vehicle 10, remote operation assistance device 20, and terminal device 30.

[0038] First, the structure of vehicle 10 will be described. The following description uses the structure of one vehicle 10 as an example, but the structures of other vehicles 10 (other vehicles 10 included in the remote operating system 1) are the same. For example... Figure 2 As shown, vehicle 10 includes a communication device 110, a camera 120, a drive unit 130, and a control unit 140 as hardware components. Furthermore, the hardware components of vehicle 10 are not limited to... Figure 2 The illustrated structure can also be implemented with other hardware elements.

[0039] The communication device 110 is a device that communicates with external devices (such as the remote operation assistance device 20) via the network 40. The camera 120 is mounted on the vehicle 10 and configured to capture images of the front of the vehicle 10.

[0040] The drive unit 130 is a device that drives the vehicle 10. The drive unit 130 may include, for example, a wheel drive device that provides rotational driving force to the wheels, a steering drive device that steers the wheels, etc.

[0041] The control device 140 is a device that comprehensively controls the movement of the vehicle 10. Figure 3This diagram illustrates an example of the hardware structure of the control device 140. In this embodiment, the control device 140 is composed of a computer device. Furthermore, the hardware structures of the control device 230 included in the remote operation assistance device 20 (described later) and the control device 340 included in the terminal device 30 are also similar. Figure 3 It's the same.

[0042] like Figure 3 As shown, the control device 140 includes a processor 150, a ROM (Read Only Memory) 160, a RAM (Random Access Memory) 170, and a device I / F (interface) unit 180.

[0043] Processor 150 is, for example, a CPU (Central Processing Unit). Processor 150 executes programs to comprehensively control the operation of control device 140, realizing the various functions of control device 140. The various functions of control device 140 will be described later.

[0044] ROM 160 is a non-volatile memory that stores various information, including programs executed by processor 150. RAM 170 is a volatile memory that has the working area of ​​processor 150. Device I / F section 180 is an interface for connecting to other devices (communication device 110, camera 120, drive device 130, etc.).

[0045] return Figure 2 The functions of the control device 140 will be explained. For example... Figure 2 As shown, the control device 140 includes a slope information calculation unit 141, a position information acquisition unit 142, an image acquisition unit 143, a driving control unit 144, an assistance request sending unit 145, and an operation information receiving unit 146. Furthermore, in Figure 2 In the example shown, only the functions required to explain the main parts of this embodiment are illustrated, but the functions of the control device 140 are not limited to this. In this embodiment, the functions of the above-mentioned parts are implemented by the processor 150 executing the program stored in the ROM 160. However, it is not limited to this, and some or all of these functions may also be implemented by dedicated hardware circuits (semiconductor integrated circuits, etc.).

[0046] The slope information calculation unit 141 calculates slope information, which represents the ratio of the change in the speed of the vehicle 10 to the change in the motion vector of the surrounding area of ​​the vehicle 10. In this embodiment, the slope information is information representing the ratio of the change in the motion vector to the change in the speed of the vehicle 10. The slope information is an example of "relationship information" that represents the relationship between the speed of the vehicle 10 and the motion vector of the surrounding area of ​​the vehicle 10 captured by the camera 120 mounted on the vehicle 10.

[0047] In this embodiment, before using the service, the vehicle 10 is driven in a pre-defined location. The slope information calculation unit 141 calculates the slope of the corresponding linear equation representing the speed of the vehicle 10 and the motion vector of the surrounding area of ​​the vehicle 10, based on the speed of the vehicle 10 and images captured by the camera 120, and sends slope information representing the calculated slope to the remote operation assistance device 20. Furthermore, the timing of slope information calculation is not limited to before using the service and can be arbitrarily set. For example, the slope information calculation unit 141 can calculate and update the slope information during service use.

[0048] The following is an example of how to calculate the motion vector of the surrounding area of ​​vehicle 10. While vehicle 10 is moving, in the camera 120 that captures images in front of vehicle 10, such as... Figure 4 An image (moving image) is captured as shown, depicting the road surface flowing in the opposite direction to the direction of travel of vehicle 10. In this image, even at the same speed, the visual movement is greater closer to the edge (periphery), thus appearing to move faster. That is, as vehicle 10 moves, the subject moves faster closer to the edge of the image, and the movement becomes smoother as it moves from the edge towards the inside of the image.

[0049] Therefore, for example, the slope information calculation unit 141 can also calculate motion vectors (block matching method) by searching for similar blocks between frames for dividing the image captured by the camera 120 into units, i.e., blocks, and setting blocks with motion vectors of magnitude greater than or equal to a reference value as peripheral regions. Thus, the motion vectors of each block contained within the peripheral region can be calculated simultaneously with setting the peripheral region. Furthermore, the method for calculating motion vectors is not limited to the block matching method described above; for example, the gradient method can be used to calculate motion vectors for each pixel of the image, and pixels with motion vectors of magnitude greater than or equal to a reference value can be set as peripheral regions. Additionally, for example, the motion vectors of the peripheral region can be calculated using the motion vectors output by the codec used in video transmission. Alternatively, for example, a predetermined area at the periphery of the image captured by each camera 120 can be pre-defined as a peripheral region.

[0050] Next, an example of a method for calculating slope information will be explained. In this embodiment, as... Figure 5 As shown, a coordinate system is established with the speed of vehicle 10 as the horizontal axis and the average magnitude of the motion vectors of all blocks contained in the surrounding area as the vertical axis. The slope information calculation unit 141 calculates the average magnitude of the motion vectors of the surrounding area calculated as above and the speed of vehicle 10 at this time (in... Figure 5 The coordinates of the corresponding points between the calculated coordinates and the origin of the coordinate system (referred to as "the specified speed" in the text) are calculated. Furthermore, the slope information calculation unit 141 calculates the equation of a straight line connecting the calculated coordinates to the origin of the coordinate system (an example of a function representing the correspondence between the speed of the vehicle 10 and the motion vector of the surrounding area), thereby calculating the slope of this straight line equation. The information representing the slope calculated as described above is called slope information.

[0051] In addition, Figure 5 In the example, the average value of the motion vectors of each block contained in the surrounding area is set as the vertical axis, but it is not limited to this. For example, the cumulative value of the motion vectors of each block contained in the surrounding area can also be set as the vertical axis. In this case, the slope information calculation unit 141 can also calculate the slope information in the same way as described above.

[0052] The slope information calculation unit 141 sends the slope information calculated as described above and a vehicle ID representing information used to identify the vehicle 10 to the remote operation assistance device 20. The remote operation assistance device 20 stores the slope information received from the vehicle 10 in correspondence with the vehicle ID. The specific structure of the remote operation assistance device 20 will be described later. Furthermore, in this embodiment, the slope information calculation unit 141 has both the function of calculating slope information and the function of sending slope information, but it is not limited to this. For example, the functions of calculating slope information and sending slope information can be set separately.

[0053] return Figure 2 The functions of the control device 140 of the vehicle 10 will now be explained. The position information acquisition unit 142 acquires position information indicating the position of the vehicle 10. As a method for acquiring position information, various known technologies can be used. For example, the position information acquisition unit 142 can also receive GPS signals representing radio waves transmitted from each of the multiple GPS satellites, and calculate the position of the vehicle 10 based on the received GPS signals through three-dimensional positioning, thereby acquiring position information indicating that position.

[0054] The image acquisition unit 143 acquires images captured by the camera 120. The images acquired by the image acquisition unit 143 are also used in the aforementioned slope information calculation. Furthermore, during the operation of the vehicle 10 after the service commences, the images acquired by the image acquisition unit 143 are sent to the remote operation assistance device 20.

[0055] In the autonomous driving mode, where the vehicle 10 is driving autonomously, the driving control unit 144 controls the vehicle 10 to move closer to the target position (controlling the drive unit 130) based on the target position and the position information acquired by the position information acquisition unit 142. On the other hand, in the remote operation mode, where an operator remotely operates the vehicle 10, the driving control unit 144 controls the vehicle 10 to move based on operation information input to the terminal device 30 in accordance with the operator's operation. In this embodiment, the vehicle 10 has both the autonomous driving mode and the remote operation mode; the vehicle 10 is driven primarily in the autonomous driving mode, and in the remote operation mode when it cannot drive autonomously.

[0056] The assistance request sending unit 145 sends an assistance request to the remote operation assistance device 20 for remote operation-based assistance, for example, when the vehicle 10 is in autonomous driving mode and cannot drive autonomously. An example of a state where the vehicle 10 cannot drive autonomously is a state where an object of a predetermined size (determined to be too large to travel directly in) exists in the direction of travel of the vehicle 10. In this embodiment, the assistance request sending unit 145 sends an assistance request to the remote operation assistance device 20 when it detects an object of a predetermined size or larger in the direction of travel of the vehicle 10 based on an image acquired by the image acquisition unit 143. The assistance request in this embodiment includes at least information requesting remote operation-based assistance and a vehicle ID indicating information for identifying the vehicle 10.

[0057] The operation information receiving unit 146 receives operation information sent from the terminal device 30 via the remote operation assistance device 20. After the assistance request sending unit 145 sends an assistance request, the driving control unit 144 controls the vehicle 10 to drive based on the operation information received by the operation information receiving unit 146, without performing driving control based on the target position and position information. In other words, the driving mode of the vehicle 10 switches from autonomous driving mode to remote operation mode.

[0058] Next, the structure of the remote operation assistance device 20 will be described. For example... Figure 2As shown, the remote operation assistance device 20 includes a communication device 210, a storage unit 220, and a control device 230 as hardware components. Furthermore, the hardware components of the remote operation assistance device 20 are not limited to... Figure 2 The illustrated structure can also be implemented with other hardware elements.

[0059] The communication device 210 is a device that communicates with external devices (such as vehicle 10, terminal device 30, etc.) via network 40.

[0060] The storage unit 220 stores slope information corresponding to each vehicle 10. In this embodiment, as... Figure 6 As shown, the storage unit 220 stores slope information corresponding to each vehicle ID representing information used to identify the vehicle. Furthermore, the storage format of the slope information is not limited to... Figure 6 In this way.

[0061] The control device 230 is a device that comprehensively controls the operation of the remote operation assistance device 20. In this embodiment, the control device 230 is composed of a computer device and has... Figure 3 Same hardware structure.

[0062] Next, the functions of the control device 230 of the remote operation assistance device 20 will be explained. For example... Figure 2 As shown, the control device 230 includes a slope information receiving unit 231, an auxiliary request receiving unit 232, an acquisition unit 233, a determination unit 234, an auxiliary control unit 235, and a remote information transmission and receiving unit 236. Furthermore, in Figure 2 In this example, only the functions required to describe the main parts of this embodiment are illustrated, but the functions of the control device 230 are not limited to this. In this embodiment, the functions of the above-mentioned parts are implemented by the processor 150 executing the program stored in the ROM 160. However, it is not limited to this, and some or all of these functions may be implemented by dedicated hardware circuitry.

[0063] The slope information receiving unit 231 receives the slope information and vehicle ID sent from the vehicle 10 as described above. Furthermore, the slope information receiving unit 231 stores the slope information received from the vehicle 10 in the storage unit 220 (see reference 220) corresponding to the vehicle ID. Figure 6 Furthermore, in this embodiment, the slope information receiving unit 231 has the function of receiving slope information and storing slope information in the storage unit 220, but it is not limited to this. For example, the functions of receiving slope information and storing slope information in the storage unit 220 may be set separately.

[0064] The assistance request receiving unit 232 receives the aforementioned assistance request sent from the vehicle 10.

[0065] The acquisition unit 233 acquires the slope information described above. More specifically, in this embodiment, when the assistance request receiving unit 232 receives an assistance request, the acquisition unit 233 determines the vehicle ID included in the received assistance request. Furthermore, the acquisition unit 233 acquires the slope information corresponding to the determined vehicle ID from the storage unit 220. In this embodiment, the acquisition unit 233 acquires the slope information whenever the assistance request receiving unit 232 receives an assistance request. As described above, the assistance request is sent per vehicle 10. If an assistance request is received from one vehicle 10 and then from another vehicle 10, it can also be considered that a switch has occurred in the vehicle 10 that is the target of remote operation.

[0066] The determination unit 234 determines whether the slope represented by the slope information acquired by the acquisition unit 233 is decreasing. More specifically, if the acquisition unit 233 acquires first slope information representing the slope corresponding to the first vehicle 10 and then acquires second slope information representing the slope corresponding to the second vehicle 10, the determination unit 234 determines whether the slope represented by the second slope information is smaller than the slope represented by the first slope information.

[0067] Here, it can be assumed that "second vehicle 10" corresponds to the vehicle 10 that sent the latest assistance request, and "first vehicle 10" corresponds to the vehicle 10 that sent the assistance request immediately before it. Therefore, "second slope information" corresponds to the latest slope information obtained by the acquisition unit 233, and "first slope information" corresponds to the slope information obtained by the acquisition unit 233 immediately before the second slope information. In this embodiment, whenever the acquisition unit 233 obtains slope information, the determination unit 234 compares the slope represented by the latest slope information (second slope information) with the slope represented by the slope information (first slope information) immediately before it, and determines whether the slope represented by the slope information is decreasing.

[0068] The auxiliary control unit 235 controls the remote operation of the auxiliary vehicle 10 based on the slope information acquired by the acquisition unit 233. More specifically, the auxiliary control unit 235 controls the remote operation of the auxiliary vehicle 10 based on changes in the slope represented by the slope information acquired by the acquisition unit 233. In this embodiment, the auxiliary control unit 235 controls the remote operation of the auxiliary vehicle 10 when the slope represented by the slope information acquired by the acquisition unit 233 decreases. Furthermore, the auxiliary control unit 235 controls the remote operation of the auxiliary vehicle 10 when the determination unit 234 determines that the slope represented by the second slope information is smaller than the slope represented by the first slope information.

[0069] Here, the change in perceived speed corresponding to changes in the surrounding area of ​​vehicle 10 will be explained. For example, assume the following situation: vehicle 10, which is the object of remote operation (vehicle 10 that sent an assistance request), switches from vehicle 10 equipped with a wide-angle mode camera 120 with a large field of view to vehicle 10 equipped with a telephoto mode camera 120 with a small field of view. Here, the camera 120 equipped on vehicle 10 can switch between wide-angle mode and telephoto mode depending on the purpose.

[0070] Figure 7 This is a diagram showing the vehicle 10 equipped with a wide-angle camera 120 and the surrounding area in an image captured by the camera 120. Figure 8 This is a diagram showing a vehicle 10 equipped with a camera 120 in telescopic mode and the surrounding area in an image captured by the camera 120. According to... Figure 7 and Figure 8 It can be seen that the peripheral area of ​​the image captured by camera 120 in telephoto mode becomes smaller. When the peripheral area becomes smaller, the average magnitude of the motion vector in the peripheral area also decreases, and from the operator's perspective, the vehicle 10 will feel slower (the perceived speed decreases). Therefore, when from... Figure 7 The state is switched to Figure 8 In this state, even if the speed of vehicle 10 remains unchanged, the operator will perceive that the speed of vehicle 10 has slowed down. Therefore, the operator may over-increase the speed of vehicle 10.

[0071] Additionally, for example, sometimes even if the size of the surrounding area does not change much, the perceived speed can change depending on the difference in the range of areas with large motion vectors (such as the road surface). Figure 9 This is a diagram showing a vehicle 10 equipped with a camera 120 whose angle θ between a vertical line extending in the vertical direction and the optical axis of the camera 120 is less than a reference value (equivalent to a downward angle), and the surrounding area in an image captured by the camera 120. Figure 10 This is a diagram showing a vehicle 10 equipped with a camera 120 whose angle θ between the vertical line and the optical axis of the camera 120 is above a reference value (equivalent to an elevation angle), and the surrounding area in an image captured by the camera 120. According to... Figure 9 and Figure 10 It can be seen that in the image captured by camera 120 at an elevation angle, the area of ​​the surrounding region, including the road surface area with large motion vectors, is smaller. Therefore, the average magnitude of the motion vectors in the surrounding region is also smaller. Thus, when from... Figure 9 The state is switched to Figure 10In this state, even if the speed of vehicle 10 remains unchanged, the operator will perceive that the speed of vehicle 10 has slowed down. Therefore, similarly to the above, the operator may excessively increase the speed of vehicle 10.

[0072] When the average magnitude of the motion vectors in the surrounding area decreases, the slope represented by the slope information also decreases, and in this case, it is assumed that the operator's perceived speed will decrease. Therefore, the remote operation assistance device 20 of this embodiment determines whether the slope represented by the slope information is decreasing, and if it determines that the slope represented by the slope information is decreasing, it performs remote operation control to assist the operator. For example, when the slope represented by the slope information acquired by the acquisition unit 233 decreases from... Figure 11 When the slope of line A corresponding to vehicle 10A is reduced to the slope of line B corresponding to vehicle 10B, the auxiliary control unit 235 implements control to suppress the operator from excessively increasing the speed of vehicle 10 as a control to assist the remote operation of the vehicle 10.

[0073] In this embodiment, the auxiliary control unit 235 controls the reduction of the notification slope as a means of assisting in the remote operation of the vehicle 10. As an example of controlling the reduction of the notification slope, it can also control the possibility of speeding. For example, the auxiliary control unit 235 can also display on the terminal device 30. Figure 12 The control of the warning screen shown is not limited to this. For example, the auxiliary control unit 235 can also perform the following control: instead of outputting an audible warning about the possibility of speeding from the terminal device 30, it can output information about the possibility of speeding from the terminal device 30. Figure 12 The warning screen shown. In addition, for example, the auxiliary control unit 235 can also refer to past historical records and, if the type of vehicle 10 corresponding to the reduced slope information is the same as the type of vehicle 10 previously operated, perform control by outputting information (either image output or sound output) from the terminal device 30 to notify that the speed is the same as the vehicle 10 previously operated by the operator.

[0074] When the slope represented by the slope information acquired by the acquisition unit 233 decreases, the auxiliary control unit 235 of this embodiment sends auxiliary information (such as information representing the aforementioned warning screen) and a remote operation request to the terminal device 30, indicating information for remote operation of the vehicle 10. On the other hand, when the slope represented by the slope information acquired by the acquisition unit 233 does not decrease, the auxiliary control unit 235 sends a remote operation request to the terminal device 30 without sending auxiliary information.

[0075] return Figure 2The functions of the control device 230 of the remote operation assistance device 20 will continue to be described. The remote information transmission and reception unit 236 transmits and receives remote information representing information used in the remote operation of the vehicle 10. Remote information includes, for example, operation information transmitted from the terminal device 30, images captured by the camera 120 of the vehicle 10, and other such information. For example, the remote information transmission and reception unit 236 can receive operation information transmitted from the terminal device 30 and transmit the received operation information to the vehicle 10. Furthermore, as described above, during the operation of the vehicle 10 after the service has commenced, images captured by the camera 120 of the vehicle 10 are transmitted to the remote operation assistance device 20; therefore, the remote information transmission and reception unit 236 can also transmit images received from the vehicle 10 to the terminal device 30.

[0076] Next, the structure of the terminal device 30 will be described. For example... Figure 2 As shown, the terminal device 30 includes a communication device 310, a display device 320, an operating device 330, and a control device 340 as hardware elements. Furthermore, the hardware elements of the terminal device 30 are not limited to... Figure 2 The illustrated structure can also be implemented with other hardware elements.

[0077] The communication device 310 is a device that communicates with external devices (such as the remote operation assistance device 20) via the network 40. The display device 320 is a device that displays various information, such as a liquid crystal display. The operating device 330 is a device for the operator to perform various operations.

[0078] The control device 340 is a device that comprehensively controls the operation of the terminal device 30. In this embodiment, the control device 340 is composed of a computer device and has... Figure 3 Same hardware structure.

[0079] Next, the functions of the control device 340 will be explained. For example... Figure 2 As shown, the control device 340 includes an information receiving unit 341, a display control unit 342, and an operation information transmitting unit 343. Furthermore, in Figure 2 In this example, only the functions required to describe the main parts of this embodiment are illustrated, but the functions of the control device 340 are not limited to this. In this embodiment, the functions of the above-mentioned parts are implemented by the processor 150 executing the program stored in the ROM 160. However, it is not limited to this, and some or all of these functions may be implemented by dedicated hardware circuitry.

[0080] The information receiving unit 341 receives various types of information sent from the remote operation assistance device 20. For example, the information receiving unit 341 can receive information such as the aforementioned remote operation request, the aforementioned assistance information, and images captured by the camera 120 of the vehicle 10 from the remote operation assistance device 20.

[0081] The display control unit 342 controls the display of various information on the display device 320. For example, the display control unit 342 can control the display of the aforementioned remote operation requests (such as messages requesting remote operation) on the display device 320, as well as the display of the aforementioned auxiliary information (such as warning screens) on the display device 320, and can also control the display of images captured by the camera 120 of the vehicle 10 on the display device 320. For example, in the aforementioned remote operation mode, the images captured by the camera 120 of the vehicle 10 are sent to the terminal device 30 via the remote operation assistance device 20, and the display control unit 342 displays the images received from the remote operation assistance device 20 on the display device 320, allowing the operator to remotely operate the vehicle 10 while checking its driving status.

[0082] The operation information sending unit 343 sends operation information input by the operator corresponding to the operation of the operation device 330 to the remote operation assistance device 20. For example, after the operator confirms the remote operation request displayed on the display device 320, the operator operates the operation device 330 to start the remote operation of the vehicle 10, and the operation information sending unit 343 can send operation information input by the operator corresponding to the operation of the operation device 330 to the remote operation assistance device 20.

[0083] Next, refer to Figure 13 Here is an example of the operation process of the remote operating system 1 when the remotely operated object is switched from vehicle 10A to vehicle 10B.

[0084] First, before using the service, vehicle 10A calculates the slope information and sends its vehicle ID and the calculated slope information to the remote operation assistance device 20 (step S1). The remote operation assistance device 20 stores the slope information received from vehicle 10A in the storage unit 220 corresponding to the vehicle ID (step S2). Similarly, vehicle 10B sends the slope information and vehicle ID to the remote operation assistance device 20 (step S3), and the remote operation assistance device 20 stores the slope information received from vehicle 10B in the storage unit 220 corresponding to the vehicle ID (step S4).

[0085] The following describes the steps after starting to use the service. Figure 13In this example, firstly, vehicle 10A becomes unable to drive autonomously and sends the aforementioned assistance request to the remote operation assistance device 20 (step S5). Upon receiving the assistance request from vehicle 10A, the remote operation assistance device 20 retrieves slope information corresponding to the vehicle ID of vehicle 10A included in the assistance request from the storage unit 220, and determines whether the slope represented by the retrieved slope information is smaller than the slope represented by the previously retrieved slope information (step S6). In this example, the premise is that no assistance request was received from vehicle 10A, therefore the determination result of step S6 is negative. Since the determination result of step S6 is negative, remote operation control of vehicle 10A is not required, and the remote operation assistance device 20 sends the aforementioned remote operation request to the terminal device 30 without sending the aforementioned assistance information (step S7).

[0086] Terminal device 30 displays the remote operation request received from remote operation assistance device 20 (step S8). The operator, having confirmed the remote operation request, begins remote operation of vehicle 10A, and terminal device 30 sends operation information corresponding to the operator's operation to remote operation assistance device 20 (step S9). Remote operation assistance device 20 sends the operation information received from terminal device 30 to vehicle 10A (step S10), and vehicle 10A drives according to the operation information received from remote operation assistance device 20. In other words, vehicle 10A drives according to the operator's remote operation.

[0087] Next, in Figure 13 In the example, vehicle 10B becomes unable to drive autonomously and sends the aforementioned assistance request to the remote operation assistance device 20 (step S11). Upon receiving the assistance request from vehicle 10B, the remote operation assistance device 20 retrieves the slope information corresponding to the vehicle ID of vehicle 10B included in the assistance request from the storage unit 220, and determines whether the slope represented by the retrieved slope information is smaller than the slope represented by the previously retrieved slope information corresponding to vehicle 10A (step S12). Here, as... Figure 11 As shown, the slope of the straight line corresponding to vehicle 10B (representing the straight line between the speed of vehicle 10 and the magnitude of the motion vector of the surrounding area) is smaller than the slope of the straight line corresponding to vehicle 10A. Therefore, the determination result of step S12 is positive. Since the determination result of step S12 is positive, remote operation control of vehicle 10B is required. The remote operation assistance device 20 sends the aforementioned remote operation request and the aforementioned assistance information to the terminal device 30 (step S13).

[0088] The terminal device 30 displays the remote operation request and assistance information received from the remote operation assistance device 20 (step S14). The operator, having confirmed the remote operation request and assistance information, begins remote operation of the vehicle 10B. At this time, the display device 320 of the terminal device 30 displays, as shown below. Figure 12 The warning screen (auxiliary information) shown can prevent the operator from excessively increasing the speed of vehicle 10B due to a decreased perceived speed. Terminal device 30 sends operation information corresponding to the operator's operation to remote operation assistance device 20 (step S15). Remote operation assistance device 20 sends the operation information received from terminal device 30 to vehicle 10B (step S16), and vehicle 10B drives according to the operation information received from remote operation assistance device 20. In other words, vehicle 10B drives according to the operator's remote operation.

[0089] Figure 14 This is a flowchart illustrating an example of the actions of vehicle 10 when calculating the aforementioned slope information. For example... Figure 14 As shown, firstly, the slope information calculation unit 141 acquires speed information representing the speed of the vehicle 10 (step S101). Next, the slope information calculation unit 141 calculates the motion vector of the surrounding area of ​​the vehicle 10 based on the image captured by the camera 120 (step S102). Next, the slope information calculation unit 141 calculates the corresponding linear equation representing the speed of the vehicle 10 and the motion vector of the surrounding area based on the speed information acquired in step S101 and the motion vector calculated in step S102, and calculates slope information representing the slope of the linear equation (step S103). Next, the slope information calculation unit 141 sends the vehicle ID and the slope information calculated in step S103 to the remote operation assistance device 20 (step S104). The remote operation assistance device 20 (slope information receiving unit 231) stores the slope information received from the vehicle 10 (slope information calculation unit 141) in the storage unit 220 corresponding to the vehicle ID.

[0090] Figure 15 This is a flowchart illustrating an example of the operation of the remote operation assistance device 20 when the aforementioned assistance request is received. For example... Figure 15As shown, firstly, the assistance request receiving unit 232 receives an assistance request from vehicle 10 (step S201). Next, the acquisition unit 233 acquires the slope information corresponding to the vehicle ID included in the assistance request received in step S201 from the storage unit 220 (step S202). Next, the determination unit 234 determines whether the slope represented by the slope information acquired in step S202 (the second slope information corresponding to the second vehicle 10 that sent the latest assistance request) is smaller than the slope represented by the previously acquired slope information (the first slope information corresponding to the first vehicle 10 that sent the assistance request immediately after it) (step S203).

[0091] If the result of step S203 is positive (step S203: "Yes"), the auxiliary control unit 235 performs remote operation control of the auxiliary vehicle 10 (step S204). Specifically, as described above, the auxiliary control unit 235 sends auxiliary information (e.g., information indicating the aforementioned warning screen) and a remote operation request to the terminal device 30, indicating a request for remote operation of the vehicle 10. On the other hand, if the result of step S203 is negative (step S203: "No"), the auxiliary control unit 235 only sends the aforementioned remote operation request to the terminal device 30, without performing remote operation control of the auxiliary vehicle 10 (step S205).

[0092] As explained above, the remote operation assistance device 20 of this embodiment controls the remote operation of the vehicle 10 based on slope information (an example of "relationship information"), which represents the ratio of the change in the motion vector of the surrounding area of ​​the vehicle 10 to the change in the speed of the vehicle 10. By considering the slope information in the control of the remote operation of the vehicle 10, the remote operation of the operator can be appropriately assisted. More specifically, the remote operation assistance device 20 of this embodiment determines whether the slope represented by the slope information is decreasing. If it determines that the slope represented by the slope information is decreasing, it assumes that the operator's perceived speed will decrease (the operator will feel that the speed of the vehicle 10 is slow), and therefore implements control to suppress the operator from excessively increasing the speed of the vehicle 10 as a control to assist the remote operation of the vehicle 10. As described above, the remote operation assistance device 20 of this embodiment controls the control of notifying the decrease in slope as a control to suppress the operator from excessively increasing the speed of the vehicle 10. Thus, it is possible to suppress the operator from excessively increasing the speed of the vehicle 10. Therefore, according to this embodiment, the remote operation of the operator can be appropriately assisted.

[0093] The embodiments of this disclosure have been described above, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways and can be omitted, substituted, or modified in various ways without departing from the spirit of the invention. These new embodiments and their variations are included within the scope and spirit of the invention, and are included within the scope of the invention as described in the claims and its equivalents.

[0094] Furthermore, the effects described in the embodiments in this specification are merely illustrative and not limiting, and may have other effects as well.

[0095] The following are examples of variations.

[0096] (1) Variation Example 1

[0097] In the above-described embodiment, the storage unit 220 of the remote operation assistance device 20 stores slope information corresponding to each vehicle ID, but is not limited thereto. For example, the storage unit 220 can also be configured to store slope information corresponding to each combination of the field of view of the vehicle 10 and the camera 120. Even for the same vehicle 10, there may be cases where the field of view differs depending on the shooting mode of the camera 120. For example, the field of view may be set to 90 degrees in telephoto mode and 150 degrees in wide-angle mode. As mentioned above, in wide-angle mode, the peripheral area becomes larger, and therefore the motion vector of the peripheral area also becomes larger, and the perceived speed also increases. It is also possible to assume such a situation in advance and calculate the slope information, for example, as Figure 16 As shown, the storage unit 220 stores slope information corresponding to each combination of the vehicle ID used to identify the vehicle 10 and the field of view of the camera 120.

[0098] In this method, the aforementioned assistance request includes, in addition to the vehicle ID, information representing the field of view of the camera 120. The acquisition unit 233 acquires slope information corresponding to the combination of the vehicle ID and the field of view of the camera 120 included in the assistance request from the storage unit 220. The determination unit 234 determines whether the slope represented by the slope information acquired by the acquisition unit 233 is decreasing. Similarly to the above embodiment, when the acquisition unit 233 acquires second slope information corresponding to the combination of the first vehicle 10 and the field of view after acquiring first slope information corresponding to the combination of the first vehicle 10 and the field of view, the determination unit 234 can determine whether the slope represented by the second slope information is smaller than the slope represented by the first slope information.

[0099] Furthermore, remote operation of the auxiliary vehicle 10 can be controlled without using the slope information described above. For example, when the target of remote operation changes from a vehicle 10 equipped with a wide-angle camera 120 (first vehicle 10) to a vehicle 10 equipped with a telephoto camera 120 (second vehicle 10), the auxiliary control unit 235 can perform the remote operation control of the auxiliary vehicle 10 described above (output of warning screens, etc.). In this case, the auxiliary request may include, for example, camera information that can determine the mode of the camera 120, in addition to the vehicle ID (for example, information indicating either wide-angle or telephoto mode, or information indicating the field of view). When the acquisition unit 233 acquires the camera information included in the auxiliary request of the first vehicle 10 and then acquires the camera information included in the auxiliary request of the second vehicle 10, the determination unit 234 can determine, based on the camera information, whether the target of remote operation has changed from a vehicle 10 equipped with a wide-angle camera 120 to a vehicle 10 equipped with a telephoto camera 120. Furthermore, the auxiliary control unit 235 can control the remote operation of the auxiliary vehicle 10 based on the determination result of the determination unit 234. As described above, as a means of controlling the remote operation of the auxiliary vehicle 10, the auxiliary control unit 235 can, for example, control the possibility of speeding warnings (e.g., control the display of the aforementioned warning screen on the terminal device 30). In this method, it is not necessary to store the slope information of each vehicle ID in the storage unit 220, and the memory capacity can be reduced.

[0100] (2) Variation Example 2

[0101] For example, the storage unit 220 could also store slope information corresponding to each combination of vehicle 10 and location (area). For example, in locations with fewer surrounding buildings, the motion vector of the surrounding area is smaller, thus the perceived speed tends to be smaller. Alternatively, this situation could be assumed in advance and the slope information calculated, for example... Figure 17 As shown, the storage unit 220 stores slope information corresponding to each combination of vehicle ID and location.

[0102] In this method, the aforementioned assistance request includes not only the vehicle ID but also location information indicating the current location of vehicle 10. The acquisition unit 233 acquires slope information corresponding to the combination of the vehicle ID and the location indicated by the location information in the assistance request from the storage unit 220. Similarly to the above embodiment, when the acquisition unit 233 acquires second slope information corresponding to the combination of the first vehicle 10 and the location after acquiring the first slope information corresponding to the combination of the first vehicle 10 and the location, the determination unit 234 can determine whether the slope indicated by the second slope information is smaller than the slope indicated by the first slope information.

[0103] Furthermore, for example, the remote operation of the auxiliary vehicle 10 can be controlled without using the slope information described above. For example, when the target of remote operation changes from a vehicle 10 (first vehicle 10) located in an area with many surrounding buildings to a vehicle 10 (second vehicle 10) located in an area with few surrounding buildings, the auxiliary control unit 235 can control the remote operation of the auxiliary vehicle 10 as described above. When the acquisition unit 233 acquires the location information included in the assistance request of the first vehicle 10 and then acquires the location information included in the assistance request of the second vehicle 10, the determination unit 234 can determine, based on each location information, whether the target of remote operation has changed from a vehicle 10 located in an area with many surrounding buildings to a vehicle 10 located in an area with few surrounding buildings. Moreover, the auxiliary control unit 235 can control the remote operation of the auxiliary vehicle 10 based on the determination result of the determination unit 234. In this method, it is not necessary to store the slope information of each vehicle ID in the storage unit 220, and the memory capacity can be reduced.

[0104] (3) Variation Example 3

[0105] For example, the installation angle θ of camera 120 (the angle θ between the vertical line and the optical axis of camera 120) can be used instead of the field of view of camera 120 in the above-described variation 1. As mentioned above, when the installation angle θ of camera 120 is a downward angle, the area of ​​the surrounding region containing the road surface with large movement becomes larger, and therefore the motion vector of the surrounding region also becomes larger, and the perceived speed also becomes larger. Alternatively, such a situation can be assumed in advance and slope information can be calculated, and the storage unit 220 can store the slope information corresponding to each combination of vehicle ID and installation angle θ of camera 120.

[0106] In this method, the aforementioned auxiliary request includes, in addition to the vehicle ID, information indicating the mounting angle θ of the camera 120. The vehicle 10 may also have the function of detecting the mounting angle θ of the camera 120 in real time using sensors or the like. Similarly to the above embodiment, when the acquisition unit 233 acquires first slope information corresponding to the combination of the mounting angle θ of the first vehicle 10 and the camera 120, and then acquires second slope information corresponding to the combination of the mounting angle θ of the second vehicle 10 and the camera 120, the determination unit 234 can determine whether the slope represented by the second slope information is smaller than the slope represented by the first slope information.

[0107] Furthermore, remote operation of the auxiliary vehicle 10 can be controlled without using the aforementioned slope information. For example, when the object of remote operation changes from a vehicle 10 (first vehicle 10) with a camera 120 mounting angle θ at a downward angle to a vehicle 10 (second vehicle 10) with a camera 120 mounting angle θ at an upward angle, the auxiliary control unit 235 can perform the aforementioned remote operation control of the auxiliary vehicle 10. When the acquisition unit 233 acquires information indicating the camera 120 mounting angle θ included in the auxiliary request of the first vehicle 10, and then acquires information indicating the camera 120 mounting angle θ included in the auxiliary request of the second vehicle 10, the determination unit 234 can determine, based on the information indicating the mounting angle θ of each camera 120, whether the object of remote operation has changed from a vehicle 10 with a camera 120 mounting angle θ at a downward angle to a vehicle 10 with a camera 120 mounting angle θ at an upward angle. Moreover, the auxiliary control unit 235 can perform remote operation control of the auxiliary vehicle 10 based on the determination result of the determination unit 234. In this method, it is not necessary to store the slope information of each vehicle ID in the storage unit 220, and the memory capacity can be reduced.

[0108] (4) Variation Example 4

[0109] For example, the storage unit 220 may store slope information corresponding to each combination of vehicle 10 and environment (weather, daytime, nighttime, etc.). In rainy weather or at night, the motion vector in the surrounding area decreases due to poor visibility, thus the perceived speed tends to be smaller. Alternatively, the storage unit 220 may pre-assume such conditions and calculate the slope information, storing the slope information corresponding to each combination of vehicle ID and environment.

[0110] In this method, the aforementioned assistance request includes not only the vehicle ID but also environmental information representing the current external environment. The acquisition unit 233 acquires slope information corresponding to the combination of the vehicle ID and the environment represented by the environmental information in the assistance request from the storage unit 220. Similarly to the above embodiment, when the acquisition unit 233 acquires second slope information corresponding to the combination of the first vehicle 10 and the environment after acquiring the first slope information corresponding to the combination of the first vehicle 10 and the environment, the determination unit 234 can determine whether the slope represented by the second slope information is smaller than the slope represented by the first slope information.

[0111] Furthermore, control of the remote operation of the auxiliary vehicle 10 can also be performed without using the slope information described above. For example, when the target of remote operation changes from a vehicle 10 (first vehicle 10) in a sunny or daytime environment to a vehicle 10 (second vehicle 10) in a rainy or nighttime environment (an environment where the motion vector of the surrounding area is smaller compared to a sunny or daytime environment), the auxiliary control unit 235 can perform the remote operation control of the auxiliary vehicle 10 as described above. When the acquisition unit 233 acquires the environmental information contained in the assistance request of the first vehicle 10 and then acquires the environmental information contained in the assistance request of the second vehicle 10, the determination unit 234 can determine, based on each environmental information, whether the target of remote operation has changed from a vehicle 10 in a sunny or daytime environment to a vehicle 10 in a rainy or nighttime environment. Here, for example, it can be assumed that there is a time difference between the location of the first vehicle 10 and the location of the second vehicle 10, and that the target of remote operation has changed from the first vehicle 10 in a daytime environment to the second vehicle 10 in a nighttime environment. Furthermore, the auxiliary control unit 235 can remotely control the operation of the auxiliary vehicle 10 based on the determination result of the determination unit 234. In this method, it is not necessary to store the slope information of each vehicle ID in the storage unit 220, and the memory capacity can be reduced.

[0112] (5) Variation Example 5

[0113] In the above-described embodiment, slope information is used as an example of "relationship information" representing the relationship between the speed of vehicle 10 and the motion vector of the surrounding area of ​​vehicle 10 captured by camera 120 mounted on vehicle 10. Remote operation control of vehicle 10 is performed based on the slope information. However, this is not a limitation; for example, the motion vector corresponding to the speed of vehicle 10 can also be used as the relationship information. In other words, the auxiliary control unit 235 can also perform remote operation control of vehicle 10 based on the motion vector corresponding to the speed of vehicle 10.

[0114] For example, similar to the above-described implementation, before using the service, the vehicle 10 is driven in a pre-defined location. The slope information calculation unit 141 can calculate the magnitude (which can be an average or a cumulative value) of the motion vector of the surrounding area corresponding to a predetermined speed of the vehicle 10 based on the image captured by the camera 120, and send the vehicle ID and the calculated magnitude of the motion vector of the surrounding area to the remote operation assistance device 20 and store them. In this case, the speed of the vehicle 10 corresponding to the magnitude of the motion vector of the surrounding area calculated for each vehicle 10 is a common speed (predetermined speed). That is, regardless of which vehicle 10, the magnitude of the motion vector of the surrounding area is calculated when it is driving in a pre-defined location at a predetermined speed before using the service. For example, as Figure 18 As shown, the storage unit 220 can store the magnitude of the motion vector of the surrounding area (the magnitude of the motion vector corresponding to the specified speed) for each vehicle ID.

[0115] In this modified example, when the assistance request receiving unit 232 receives an assistance request, the acquisition unit 233 determines the vehicle ID contained in the received assistance request. Furthermore, the acquisition unit 233 retrieves the motion vector corresponding to the determined vehicle ID from the storage unit 220. Similarly to the embodiment described above, the acquisition unit 233 retrieves the motion vector whenever the assistance request receiving unit 232 receives an assistance request.

[0116] The determination unit 234 determines whether the motion vector acquired by the acquisition unit 233 is decreasing. More specifically, when the acquisition unit 233 acquires a second motion vector corresponding to the second vehicle 10 after acquiring a first motion vector corresponding to the motion vector corresponding to the first vehicle 10, the determination unit 234 determines whether the second motion vector is smaller than the first motion vector.

[0117] Similarly to the above-described implementation, it can be assumed that "second vehicle 10" corresponds to the vehicle 10 that sent the latest assistance request, and "first vehicle 10" corresponds to the vehicle 10 that sent the assistance request immediately before it. Therefore, "second motion vector" corresponds to the latest motion vector obtained by the acquisition unit 233, and "first motion vector" corresponds to the motion vector obtained by the acquisition unit 233 immediately before the second motion vector.

[0118] Whenever the acquisition unit 233 acquires a motion vector, the determination unit 234 compares the acquired motion vector with the motion vector acquired by the acquisition unit 233 immediately before (the previous time), and determines whether the motion vector is decreasing.

[0119] When the motion vector acquired by the acquisition unit 233 decreases, the auxiliary control unit 235 controls the remote operation of the auxiliary vehicle 10. Furthermore, when the determination unit 234 determines that the second motion vector is smaller than the first motion vector, the auxiliary control unit 235 controls the remote operation of the auxiliary vehicle 10. The content of this control is the same as in the embodiment described above. Even with this modified structure, the same effects as in the embodiment described above can be achieved.

[0120] (6) Variation Example 6

[0121] In the above-described embodiment, slope information is pre-calculated for each vehicle 10. When a vehicle 10 that is the object of remote operation (the vehicle 10 that sent the assistance request) switches, the slope represented by the slope information (first slope information) corresponding to the vehicle 10 before the switch is compared with the slope represented by the slope information (second slope information) corresponding to the vehicle 10 after the switch. Based on the comparison result, it is determined whether remote operation control of the assisted vehicle 10 can be performed. Not limited to this, for example, it could be as follows: even if the vehicle 10 that is the object of remote operation does not switch but remains the same vehicle 10, the remote operation assistance device 20 calculates the slope information while the vehicle 10 is moving, and performs remote operation control of the assisted vehicle 10 when the slope represented by the calculated slope information decreases. For example, when the driving scenario of the vehicle 10 changes from a location with many buildings to a location with fewer buildings, the motion vector of the surrounding area will decrease, therefore it is assumed that the operator's perceived speed also decreases.

[0122] Furthermore, for example, when the camera 120 switches from wide-angle mode to telephoto mode, or when the camera 120's mounting angle θ changes from a downward angle to an upward angle, it is assumed, as mentioned above, that the operator's perceived speed will decrease. Similarly, for example, when the environment of vehicle 10 changes drastically from sunny to rainy weather, it is assumed, as mentioned above, that the operator's perceived speed will decrease. Moreover, the operator's perceived speed also changes depending on the road conditions on which vehicle 10 is traveling. For example, when the number of vehicles parallel to vehicle 10 (surrounding vehicles) increases, the area containing surrounding vehicles with small motion vectors in the surrounding region of vehicle 10 increases, resulting in a decrease in the average magnitude of the motion vectors in the surrounding region. Therefore, the operator's perceived speed will also decrease, and the operator may excessively increase the speed of vehicle 10.

[0123] In this situation, the remote operation assistance device 20 also performs remote operation control of the aforementioned auxiliary vehicle 10, thereby preventing the operator from excessively increasing the speed of the vehicle 10.

[0124] This modification can also be applied to the above-described modification 5. That is, even if the vehicle 10 that is the object of remote operation does not change but remains the same vehicle 10, the remote operation assistance device 20 can calculate the motion vector corresponding to the specified speed while the vehicle 10 is in motion, and perform remote operation control of the vehicle 10 when the calculated motion vector decreases.

[0125] (7) Variation Example 7

[0126] In the above-described embodiment, as a control for the remote operation of the auxiliary vehicle 10, the auxiliary control unit 235 controls the reduction of the notification slope. However, it is not limited to this. As an example of the control for the remote operation of the auxiliary vehicle 10, the auxiliary control unit 235 may also control the speed of the vehicle 10.

[0127] For example, the auxiliary control unit 235 can also set a maximum speed without exceeding the speed limit and control the vehicle 10 to travel at a speed below the set maximum speed. Additionally, the auxiliary control unit 235 can also control the application of a reaction force to the accelerator of the vehicle 10 to prevent speeding. Furthermore, the auxiliary control unit 235 can also control the terminal device 30 to guide the operator to prevent speeding. As an example of this control, the auxiliary control unit 235 can also control the terminal device 30 to display visual guidance (vection).

[0128] Furthermore, as an example of controlling the remote operation of the aforementioned auxiliary vehicle 10, the auxiliary control unit 235 may also control the field of view of the camera 120 to be constant. For example, the auxiliary control unit 235 may also adjust the field of view of the camera 120 of the vehicle 10 (second vehicle) that is now the target of remote operation to be the same as the field of view of the camera 120 of the vehicle 10 (first vehicle) that was previously remotely operated by the operator.

[0129] For example, when the control of switching the mode of camera 120 (wide-angle mode → telephoto mode or telephoto mode → wide-angle mode) is possible, the following method may also be used: when the object of remote operation is switched from vehicle 10 (first vehicle 10) equipped with wide-angle mode camera 120 to vehicle 10 (second vehicle 10) equipped with telephoto mode camera 120, the auxiliary control unit 235 starts (allows) the control of auxiliary remote operation based on switching the mode of camera 120 of the second vehicle 10 from telephoto mode to wide-angle mode.

[0130] Alternatively, similarly to the above, for example, if the mounting angle θ of the camera 120 can be adjusted, the following method can be used: when the remotely operated object is switched from vehicle 10 (first vehicle 10) where the mounting angle θ of the camera 120 is at a downward angle to vehicle 10 (second vehicle 10) where the mounting angle θ of the camera 120 is at an upward angle, the auxiliary control unit 235 starts (allows) control for auxiliary remote operation based on adjusting the mounting angle θ of the camera 120 of the second vehicle 10 from a downward angle to an upward angle.

[0131] Alternatively, the auxiliary control unit 235 may be configured to control the remote operation of the auxiliary vehicle 10 for a limited time. For example, the auxiliary control unit 235 may be configured to control the remote operation of the auxiliary vehicle 10 for a limited time after the vehicle 10 becomes the target of remote operation is switched. This is because immediately after the switch to the vehicle 10, the operator's speed perception is most skewed, thus requiring assistance. However, as time passes, the operator gradually becomes accustomed to the new speed perception of the vehicle 10, and therefore the necessity for assistance decreases after a certain period.

[0132] Furthermore, the starting point for the timing of the switch of the vehicle 10 that becomes the object of remote operation can be arbitrarily set according to design conditions, etc. For example, the timing of receiving the auxiliary request by the auxiliary request receiving unit 232 can be set as the starting point, the timing of the completion of the determination by the determination unit 234 can be set as the starting point, or the timing of sending a remote operation request to the terminal device 30 can be set as the starting point.

[0133] (8) Variation Example 8

[0134] The slope information in the above embodiment represents the slope as the ratio of the change in the motion vector of the surrounding area to the change in the speed of the vehicle 10, but it is not limited to this. For example, the slope information could also represent the ratio of the change in the speed of the vehicle 10 to the change in the motion vector of the surrounding area. That is to say, it could also be set to... Figure 5 The slope information is calculated using a coordinate system where the horizontal axis is set as the motion vector of the area surrounding the vehicle 10 and the vertical axis is set as the velocity of the vehicle 10. In this case, contrary to the above embodiment, the auxiliary control unit 235 performs remote operation control of the auxiliary vehicle 10 when the slope represented by the slope information acquired by the acquisition unit 233 increases.

[0135] In this variation, contrary to the embodiment described above, when the average magnitude of the motion vectors in the surrounding area decreases, the slope represented by the slope information increases. In this case, it is assumed that the operator's perceived speed decreases. Examples of situations where the average magnitude of the motion vectors in the surrounding area decreases are the same as those described previously. For example, it is assumed that the vehicle 10 being remotely operated (the vehicle 10 that sent the assistance request) switches from a vehicle 10 equipped with a wide-angle camera 120 to a vehicle 10 equipped with a telephoto camera 120. In addition to the above, it is also assumed that the vehicle 10 being remotely operated switches from a vehicle 10 with a camera 120 mounted at a downward angle θ to a vehicle 10 with a camera 120 mounted at an upward angle θ, from a vehicle 10 in a sunny or daytime environment to a vehicle 10 in a rainy or nighttime environment, or from a vehicle 10 in an area with many surrounding buildings to a vehicle 10 in an area with few surrounding buildings, etc.

[0136] In the remote operation assistance device 20 of this modified example, the determination unit 234 determines whether the slope represented by the slope information acquired by the acquisition unit 233 is increasing. When the acquisition unit 233 acquires second slope information corresponding to the second vehicle 10 after acquiring first slope information corresponding to the first vehicle 10, the determination unit 234 determines whether the slope represented by the second slope information is greater than the slope represented by the first slope information. Furthermore, if the determination unit 234 determines that the slope represented by the second slope information is greater than the slope represented by the first slope information, the assistance control unit 235 performs remote operation control of the assistance vehicle 10. For example, when the slope represented by the slope information acquired by the acquisition unit 233 decreases from the first slope information, the determination unit determines whether the slope represented by the second slope information is greater than the slope represented by the first slope information. Figure 19 When the slope of line A corresponding to vehicle 10A increases to the slope of line B corresponding to vehicle 10B, the auxiliary control unit 235 can implement control to suppress the operator from excessively increasing the speed of vehicle 10, thus providing control for the remote operation of the auxiliary vehicle 10. In short, the auxiliary control unit 235 can simply perform remote operation control of the auxiliary vehicle 10 based on the slope change indicated by the slope information acquired by the acquisition unit 233.

[0137] (9) Variation Example 9

[0138] The camera 120 described above is configured to capture images of the front of the vehicle 10 and use the images of the front of the vehicle 10 to calculate slope information. However, it is not limited to this. For example, images of the rear of the vehicle 10 or images of either the left or right side of the vehicle 10 can also be used to calculate slope information.

[0139] (10) Variation Example 10

[0140] The vehicle 10 used in the remote operating system 1 of the above-described embodiment can be, for example, a four-wheeled vehicle or a two-wheeled vehicle. Alternatively, the vehicle 10 can be, for example, an automated guided vehicle (AGV), construction machinery, agricultural machinery, drones, etc.

[0141] (11) Variation Example 11

[0142] In the above-described embodiment, each vehicle 10 calculates slope information, but this is not limited to this; for example, the remote operation assistance device 20 can also calculate the slope information. In this method, the remote operation assistance device 20 can receive speed information and images from each vehicle 10, and calculate the slope information of each vehicle 10 based on the received speed information and images.

[0143] (12) Variation Example 12

[0144] For example, the auxiliary control unit 235 may also be a control method that does not perform auxiliary remote operation even if the control conditions for performing remote operation of the auxiliary vehicle 10 (hereinafter referred to as "auxiliary conditions") are met.

[0145] For example, the auxiliary control unit 235 may also be configured such that, if an auxiliary request is received from the second vehicle 10 after a certain period of time has elapsed since the remote operation to stop the first vehicle 10, no auxiliary remote operation control is performed even if the aforementioned auxiliary conditions are met. If a certain period of time has elapsed since the remote operation to stop the first vehicle 10, it is assumed that the perceived speed of the first vehicle 10 has disappeared or decreased, therefore, in this case, auxiliary remote operation control is not required.

[0146] Alternatively, the auxiliary control unit 235 may be configured such that, if the time for remotely operating the first vehicle 10 (remote control time) is less than a predetermined time, auxiliary remote operation control is not performed even if the aforementioned auxiliary conditions are met. When the time for remotely operating the first vehicle 10 is short, it is assumed that the speed of the first vehicle 10 is almost imperceptible; therefore, auxiliary remote operation control is unnecessary in this case.

[0147] Alternatively, the auxiliary control unit 235 may be configured such that, if the maximum speed of the first vehicle 10 during remote operation is less than a predetermined value, no auxiliary remote operation control is performed even if the aforementioned auxiliary conditions are met. When the maximum speed of the first vehicle 10 during remote operation is low (e.g., slow driving), it is assumed that the speed perception of the first vehicle 10 is almost negligible, and therefore the impact caused by the deviation between the speed perception of the first vehicle 10 and that of the second vehicle 10 is minimal. Therefore, in this case, auxiliary remote operation control is not required.

[0148] Alternatively, the auxiliary control unit 235 may be configured such that even if the above-mentioned auxiliary conditions are met, the control of the auxiliary remote operation is not performed, and when the predetermined speed of the first vehicle 10 at the time of remote operation is reached, the output of the notification is made (for example, the output of displaying an image on the terminal device 30 or the output of sound emitted from the terminal device 30).

[0149] The aforementioned "prescribed speed" could be, for example, the maximum speed. By notifying the operator of the maximum speed of the first vehicle 10 during remote operation, the operator can grasp the maximum speed of the first vehicle 10 during remote operation of the second vehicle 10, thus enabling them to be aware of the speed sensation and safely remotely operate the second vehicle 10. Furthermore, the aforementioned "prescribed speed" is not limited to the maximum speed; for example, it could be the speed of the first vehicle 10 during straight-line travel (the speed with constant acceleration) during remote operation, or the speed at which the travel time is longest. In short, the "prescribed speed" is any speed that evokes the feeling of speed of the first vehicle 10 during remote operation (a speed that influences the feeling of speed).

[0150] Alternatively, the auxiliary control unit 235 can be configured such that, even if the aforementioned auxiliary conditions are met, no auxiliary remote operation is performed; however, when the second vehicle 10 is remotely operated, if the accelerator pedal pressure exceeds a predetermined value, a notification is output. This prevents speeding from occurring.

[0151] Here, it can be assumed that "second vehicle 10" corresponds to the vehicle 10 that sent the latest assistance request, and "first vehicle 10" corresponds to the vehicle 10 that sent the assistance request immediately before. Therefore, "second slope information" corresponds to the latest slope information obtained by the acquisition unit 233, and "first slope information" corresponds to the slope information obtained by the acquisition unit 233 immediately before the second slope information. In this embodiment, whenever the acquisition unit 233 obtains slope information, the determination unit 234 compares the slope represented by the latest slope information (second slope information) with the slope represented by the previously obtained slope information (first slope information) and determines whether the slope represented by the slope information is decreasing.

[0152] The auxiliary control unit 235 controls the remote operation of the auxiliary vehicle 10 based on the slope information acquired by the acquisition unit 233. More specifically, the auxiliary control unit 235 controls the remote operation of the auxiliary vehicle 10 based on changes in the slope represented by the slope information acquired by the acquisition unit 233. In this embodiment, the auxiliary control unit 235 controls the remote operation of the auxiliary vehicle 10 when the slope represented by the slope information acquired by the acquisition unit 233 decreases. Furthermore, the auxiliary control unit 235 controls the remote operation of the auxiliary vehicle 10 when the determination unit 234 determines that the slope represented by the second slope information is smaller than the slope represented by the first slope information.

[0153] The above-described embodiments can be arbitrarily combined with the above-described modifications, and the above-described modifications can also be arbitrarily combined with each other.

[0154] Explanation of reference numerals in the attached figures

[0155] 1: Remote operating system; 10: Vehicle; 20: Remote operation assistance device; 30: Terminal device; 141: Slope information calculation unit; 142: Position information acquisition unit; 143: Image acquisition unit; 144: Driving control unit; 145: Assist request sending unit; 146: Operation information receiving unit; 231: Slope information receiving unit; 232: Assist request receiving unit; 233: Acquisition unit; 234: Judgment unit; 235: Assist control unit; 236: Remote information sending and receiving unit; 341: Information receiving unit; 342: Display control unit; 343: Operation information sending unit.

Claims

1. A remote operation assistance method, executed by a remote operation assistance device for remote operation of an auxiliary vehicle, the remote operation assistance method comprising: The acquisition step involves acquiring relationship information representing the relationship between the vehicle's speed and the motion vectors of the vehicle's surrounding area captured by a camera mounted on the vehicle. as well as The auxiliary control step, based on the relationship information obtained through the acquisition step, performs remote operation control of the vehicle.

2. The remote operation assistance method according to claim 1, wherein, The relationship information is slope information, which represents the ratio of the change in the vehicle's speed to the change in the motion vector. In the auxiliary control step, remote operation of the vehicle is assisted by controlling the change of the slope represented by the slope information.

3. The remote operation assistance method according to claim 2, wherein, The slope is the ratio of the change in the motion vector to the change in the vehicle's speed. In the auxiliary control step, when the slope represented by the slope information decreases, remote operation control is performed to assist the vehicle.

4. The remote operation assistance method according to claim 3, wherein, The remote operation assistance method further includes a determination step, in which, if, after obtaining first slope information representing the slope corresponding to the first vehicle through the acquisition step, second slope information representing the slope corresponding to the second vehicle is obtained, it is determined whether the slope represented by the second slope information is smaller than the slope represented by the first slope information. In the auxiliary control step, if the determination step determines that the slope represented by the second slope information is smaller than the slope represented by the first slope information, remote operation control of the vehicle is performed.

5. The remote operation assistance method according to claim 4, wherein, The slope information corresponds to each combination of the field of view angles of the vehicle and the camera. In the determination step, when the second slope information corresponding to the combination of the first vehicle and the field of view is obtained after the first slope information corresponding to the combination of the first vehicle and the field of view is obtained through the acquisition step, it is determined whether the slope represented by the second slope information is smaller than the slope represented by the first slope information.

6. The remote operation assistance method according to claim 4, wherein, The slope information corresponds to each combination of the vehicle and the location. In the determination step, when the second slope information corresponding to the combination of the first vehicle and the location is obtained after the first slope information corresponding to the combination of the first vehicle and the location is obtained through the acquisition step, it is determined whether the slope represented by the second slope information is smaller than the slope represented by the first slope information.

7. The remote operation assistance method according to claim 1, wherein, The relationship information is information representing the motion vector corresponding to the speed of the vehicle. In the auxiliary control step, when the motion vector corresponding to the vehicle's speed decreases, remote control is performed to assist in the operation of the vehicle.

8. The remote operation assistance method according to claim 7, wherein, The remote operation assistance method further includes a determination step, in which, if a second motion vector corresponding to a second vehicle is obtained after obtaining a first motion vector representing the motion vector corresponding to the first vehicle through the acquisition step, it is determined whether the second motion vector is smaller than the first motion vector. In the auxiliary control step, if the determination step determines that the second motion vector is smaller than the first motion vector, remote control of the vehicle is performed to assist in the operation of the vehicle.

9. The remote operation assistance method according to claim 3 or 7, wherein, The control that assists in the remote operation of the vehicle includes control that notifies a reduction in the slope or the motion vector.

10. The remote operation assistance method according to claim 3 or 7, wherein, The remote control assisting in the operation of the vehicle includes controlling the speed of the vehicle.

11. The remote operation assistance method according to claim 3 or 7, wherein, The control that assists in the remote operation of the vehicle includes the control of adjusting the field of view of the camera to a constant.

12. The remote operation assistance method according to claim 3 or 7, wherein, The remote control of the vehicle is only performed for a certain period of time.

13. A remote operation assistance device, comprising: The acquisition unit acquires relational information representing the relationship between the speed of the vehicle, which is a remotely operated object, and motion vectors of the surrounding area of ​​the vehicle captured by a camera mounted on the vehicle; and An auxiliary control unit performs remote control of the vehicle based on the relationship information obtained by the acquisition unit.

14. A program for causing a computer to perform the following steps: The acquisition step involves acquiring relationship information representing the relationship between the speed of the vehicle being remotely operated and the motion vectors of the area surrounding the vehicle captured by a camera mounted on the vehicle; and The auxiliary control step, based on the relationship information obtained through the acquisition step, performs remote operation control of the vehicle.

Images