Information processing method, information processing device, and program

Abstract

To guarantee safety even when a malfunction occurs due to a failure of hardware concerning remote control with respect to an automatic operation mobility.SOLUTION: An information processing method is executed by an information processing system for supporting remote control of an operator with respect to a plurality of mobile bodies. The method includes: performing one of transmission modes among the first to the third transmission modes in response to an operation of the operator acquired by a frontend; allowing the frontend to periodically transmit to a backend, a notification signal different from a travel instruction signal and a control signal in the second transmission mode for transmitting a single-shot travel instruction signal to the object mobile body so as to remotely instruct control of the object mobile body and in the third transmission mode for transmitting continuous control signals to the object mobile body so as to remotely control the object mobile body; allowing the backend to monitor the notification signal from the frontend; and stopping the object mobile body when delay occurs in the period of the notification signal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, an information processing device, and a program. [Background technology]

[0002] In recent years, in order to alleviate labor shortages, the introduction of autonomous mobility has been promoted in various places such as public roads, factories, airports, and ports. However, from a safety perspective, achieving completely unmanned operation without human intervention is extremely difficult, and remote human intervention is necessary in the event of hazards or abnormalities.

[0003] For example, Patent Document 1 discloses a technique for assigning a remote operator to a vehicle that is a remote monitoring target. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-156078 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when remote control of autonomous driving mobility is performed, such as remote driving instructions or remote operation (remote control), there is a risk of safety being reduced, for example, due to a hardware malfunction in the system, which could result in instructions or control being given unintended by the remote operator.For this reason, when remotely controlling autonomous driving mobility, safety measures must be taken to ensure safety even if a malfunction occurs due to a hardware malfunction in the system.

[0006] One of the objectives of the present disclosure is to ensure safety in remote control of autonomous driving mobility even when malfunction occurs due to a hardware failure. [Means for solving the problem]

[0007] An information processing method according to the present disclosure is executed by an information processing system that supports at least one operator in remotely controlling a plurality of mobile objects, each of which is configured to move autonomously and perform a predetermined task. The information processing system includes at least one front end that acquires an operation of the at least one operator, a plurality of edges including the plurality of mobile objects, and a back end that is communicatively connected to the at least one front end and each of the plurality of edges. The information processing method executes one of a first transmission mode, a second transmission mode, and a third transmission mode depending on the operation of the operator acquired by the front end. The first transmission mode remotely monitors a target mobile object among the plurality of mobile objects by transmitting an image of the target mobile object from the corresponding edge to the front end. The second transmission mode remotely instructs the target mobile object to be controlled by transmitting a single driving instruction signal to the target mobile object. The third transmission mode remotely controls the target mobile object by transmitting continuous control signals to the target mobile object. In the information processing method, in the second transmission mode and the third transmission mode, the front end periodically transmits a notification signal different from the driving instruction signal and the control signal to the back end, and the back end monitors the notification signal from the front end, and stops the target moving object when a delay occurs in the period of the notification signal. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a remote control system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of each device included in the remote control system according to the embodiment. [Figure 3]FIG. 3 is a diagram illustrating an example of a hardware configuration of an information processing device that realizes the functions of each device included in the remote control system according to the embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of the flow of remote assistance in remote control according to the embodiment. [Figure 5] FIG. 5 is a diagram for explaining an example of a scene of the remote assistance in FIG. [Figure 6] FIG. 6 is a flowchart showing an example of the flow of remote operation in the remote control according to the embodiment. [Figure 7] FIG. 7 is a diagram for explaining an example of a scene of the remote control of FIG. [Figure 8] FIG. 8 is a diagram for explaining hazards that are expected in the remote control according to the embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of the flow of information processing executed by the operator terminal according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a screen display on the operator terminal according to the first embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of the flow of information processing executed by the operator terminal according to the second embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of the configuration of a controller according to the second embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of the flow of information processing executed by the operator terminal according to the third embodiment. [Figure 14] FIG. 14 is a sequence diagram illustrating an example of the flow of information processing executed by the operator terminal according to the fourth embodiment. [Figure 15] FIG. 15 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system according to the fourth embodiment. [Figure 16] FIG. 16 is a sequence diagram showing an example of the flow of information processing executed by each unit of a remote control system according to a modified example of the fourth embodiment. [Figure 17] FIG. 17 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system according to the fifth embodiment. [Figure 18] FIG. 18 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system according to the sixth embodiment. [Figure 19] FIG. 19 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system according to the seventh embodiment. [Figure 20] FIG. 20 is a diagram showing an example of a screen display on an operator terminal according to the eighth embodiment. [Figure 21A] FIG. 21A is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system according to the ninth embodiment. [Figure 21B] FIG. 21B is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system according to the ninth embodiment. [Figure 22] FIG. 22 is a diagram for explaining state transitions assumed in the remote control system according to the tenth embodiment. [Figure 23] FIG. 23 is a diagram for explaining mask control in the remote control system according to the tenth embodiment. [Figure 24] FIG. 24 is a flowchart showing an example of the flow of information processing executed by the edge device according to the tenth embodiment. [Figure 25] FIG. 25 is a flowchart showing an example of the flow of information processing executed by the edge device according to the tenth embodiment. [Figure 26] FIG. 26 is a flowchart illustrating an example of the flow of information processing executed by an edge device according to the first modified example of the tenth embodiment. [Figure 27] FIG. 27 is a flowchart illustrating an example of the flow of information processing executed by an edge device according to the first modified example of the tenth embodiment. [Figure 28]FIG. 28 is a diagram showing an example of a screen display on an operator terminal according to a first modified example of the tenth embodiment. [Figure 29] FIG. 29 is a diagram for explaining state transitions assumed in a remote control system according to a second modification of the tenth embodiment. [Figure 30] FIG. 30 is a diagram for explaining mask control in a remote control system according to a second modified example of the tenth embodiment. [Figure 31] FIG. 31 is a flowchart illustrating an example of the flow of information processing executed by an edge device according to the second modification of the tenth embodiment. [Figure 32] FIG. 32 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system according to the eleventh embodiment. [Figure 33] FIG. 33 is a flowchart showing an example of the flow of information processing executed by the operator terminal according to the eleventh embodiment. [Figure 34] FIG. 34 is a diagram illustrating an example of a data configuration of delay information according to the eleventh embodiment. [Figure 35] FIG. 35 is a diagram showing an example of a data configuration of quality information according to the eleventh embodiment. [Figure 36] FIG. 36 is a diagram showing an example of a screen display on an operator terminal according to the eleventh embodiment. [Figure 37] FIG. 37 is a diagram showing an example of a screen display on an operator terminal according to a first modified example of the eleventh embodiment. [Figure 38] FIG. 38 is a sequence diagram showing an example of the flow of information processing executed by each unit of a remote control system according to a second modification of the eleventh embodiment. [Figure 39] FIG. 39 is a flowchart illustrating an example of the flow of information processing executed by an edge device according to the second modification of the eleventh embodiment. [Figure 40] FIG. 40 is a flowchart showing an example of the flow of information processing executed by an edge device according to the twelfth embodiment. [Figure 41] FIG. 41 is a flowchart showing an example of the flow of information processing executed by an edge device according to the thirteenth embodiment. [Figure 42] FIG. 42 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system according to the fourteenth embodiment. [Figure 43] FIG. 43 is a flowchart showing an example of the flow of information processing executed by the operator terminal according to the fifteenth embodiment. [Figure 44] FIG. 44 is a diagram showing an example of a data structure of required video quality information according to the fifteenth embodiment. [Figure 45] FIG. 45 is a diagram illustrating an example of the configuration of a controller according to the sixteenth embodiment. [Figure 46] FIG. 46 is a flowchart showing an example of the flow of information processing executed by the server according to the sixteenth embodiment. [Figure 47] FIG. 47 is a flowchart showing an example of the flow of information processing executed by the server according to the sixteenth embodiment. [Figure 48] FIG. 48 is a flowchart showing an example of the flow of information processing executed by the operator terminal according to the seventeenth embodiment. [Figure 49] FIG. 49 is a flowchart showing an example of the flow of information processing executed by the operator terminal according to the eighteenth embodiment. [Figure 50] FIG. 50 is a flowchart showing an example of the flow of information processing executed by a server according to the nineteenth embodiment. [Figure 51] FIG. 51 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system according to the twentieth embodiment. [Figure 52] FIG. 52 is a sequence diagram showing an example of the flow of information processing executed by each unit of a remote control system according to a modified example of the twentieth embodiment. [Figure 53]FIG. 53 is a diagram illustrating an example of the configuration of a controller according to the twenty-first embodiment. [Figure 54] FIG. 54 is a flowchart showing an example of the flow of information processing executed by the operator terminal according to the twenty-first embodiment. [Figure 55] FIG. 55 is a flowchart showing an example of the flow of information processing executed by the operator terminal according to the 22nd embodiment. [Figure 56] FIG. 56 is a flowchart showing an example of the flow of information processing executed by the operator terminal according to the twenty-second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of an information processing method, an information processing device, an information processing system, a mobile object, a program, and a recording medium according to the present disclosure will be described in detail with reference to the accompanying drawings.

[0010] In the description of the present disclosure, components having the same or substantially the same functions as those described above with respect to the previously-mentioned drawings may be given the same reference numerals, and descriptions thereof may be omitted as appropriate. Furthermore, even when the same or substantially the same parts are shown, the dimensions and proportions may be different depending on the drawing. Furthermore, for example, in order to ensure the visibility of the drawings, reference numerals may be given to only the main components in the description of each drawing, and reference numerals may not be given to components having the same or substantially the same functions as those described above with respect to the previously-mentioned drawings.

[0011] In the description of the present disclosure, components having the same or substantially the same functions may be distinguished by adding alphanumeric characters and / or symbols to the end of the reference numeral. Alternatively, when multiple components having the same or substantially the same functions are not distinguished, they may be collectively described by omitting the alphanumeric characters and / or symbols added to the end of the reference numeral.

[0012] In this disclosure, "remote control" is classified into three modes: "remote monitoring," "remote assistance," and "remote operation," which differ in responsibility, possible operations, safety function activation conditions, etc. depending on the form of control over the target mobile object. Here, "remote control" refers to at least one of "remote monitoring," "remote assistance," and "remote operation" for at least one mobile object that is configured to move autonomously and perform predetermined tasks.

[0013] In the present disclosure, "remote monitoring" refers to remotely monitoring the operation of a target moving body among at least one moving body based on an image of the target moving body. Here, "remote monitoring" is an example of a first transmission mode in which a target moving body is remotely monitored by transmitting an image of the target moving body. Also, "remote assistance" refers to remotely instructing the control of a target moving body among at least one moving body by transmitting a driving instruction signal to the target moving body. Here, "remote assistance" is an example of a second transmission mode in which a target moving body is controlled by a driving instruction signal transmitted as a single instruction request. Also, "remote operation" refers to remotely controlling a target moving body among at least one moving body by transmitting a control signal to the target moving body. Here, "remote operation" is an example of a third transmission mode in which a target moving body is controlled by a control signal transmitted as a continuous instruction request.

[0014] The display on the UI (user interface) includes the display of information about the moving object and the surrounding environment in the remote control modes of "remote monitoring," "remote assistance," and "remote operation."

[0015] The operator's control responsibility is "none" in the "remote monitoring" remote control mode, but "yes" in the "remote assistance" and "remote operation" remote control modes.

[0016] The operations that the operator can perform are "not available" in the "remote monitoring" remote control mode, but "available" in the "remote assistance" and "remote operation" remote control modes. For example, the operations that the operator can perform in the "remote assistance" remote control mode include the safe stopping of the vehicle, stopping automatic driving, resuming automatic driving, and sending commands to change route. For example, the operations that the operator can perform in the "remote operation" remote control mode include steering, such as longitudinal and lateral control of the vehicle, as well as operating the accelerator and brake.

[0017] The simultaneous control constraint is "none" in the "remote monitoring" remote control mode, but "yes" in the "remote assistance" and "remote operation" remote control modes. For example, the simultaneous control constraint in the "remote assistance" and "remote operation" remote control modes is that only one operator can be in the "remote assistance" or "remote operation" mode for one vehicle.

[0018] The safety function activation condition is "absent" in the "remote monitoring" remote control mode, but "present" in the "remote assistance" and "remote operation" remote control modes. For example, the safety function activation condition in the "remote assistance" and "remote operation" remote control modes is that the transmission and display function for video and vehicle information is operating correctly without delay, and / or that the transmission and execution of instruction commands is performed correctly without delay.

[0019] As such, in the "remote control" of the present disclosure, "remote assistance" and "remote operation" allow the operator to assume control responsibility, but the level of autonomous driving of the mobile object can be lowered. In other words, during "remote assistance" and "remote operation" in the "remote control" of the present disclosure, the remote system ensures that the operator can fulfill control responsibility. Note that in the present disclosure, operation in each of the "remote monitoring," "remote assistance," and "remote operation" modes may be referred to as the "(remote) monitoring state," the "(remote) assistance state," and the "(remote) operation state," respectively. Also, in the present disclosure, "remote operation" and / or "remote assistance" may be combined and referred to as "remote operation assistance."

[0020] (Example of remote control system configuration) 1 is a diagram illustrating an example of a schematic configuration of a remote control system 1 according to an embodiment. As illustrated in FIG. 1, the remote control system 1 includes a front end 2, a back end 3, and an edge 4.

[0021] In the remote control system 1, a plurality of front ends 2 and a plurality of edges 4 are connected to a back end 3. In the remote control system 1, a plurality of remote operators simultaneously control a plurality of vehicles 46 (moving bodies).

[0022] The front end 2 forms the remote operator's side of the remote control system 1. The front end 2 includes an operator terminal 20 and a controller 25. The operator terminal 20 and the controller 25 may be configured integrally.

[0023] The back end 3 relays the transmission and reception of information between the corresponding pair of front end 2 and edge 4. For example, the back end 3 receives video and vehicle information transmitted from the edge 4, and transmits the received video and vehicle information to the corresponding front end 2. For example, the back end 3 receives various control signals, including mode switching and driving instructions, transmitted from the front end 2, and transmits the received control signals to the corresponding edge 4. The back end 3 includes a server 30. Note that the back end 3 may be realized by the cooperation of multiple servers 30.

[0024] The edge 4 forms the side that is the target of remote control by the remote control system 1. The edge 4 includes an edge device 40, a camera 45, and a vehicle 46. Note that at least two of the edge device 40, the camera 45, and the vehicle 46 may be configured integrally. For example, the camera 45 may include an in-vehicle camera of the vehicle 46. For example, the edge device 40 may be realized by a computer installed in the vehicle 46.

[0025] The front end 2, back end 3, and edge 4 cooperate with each other through communication, but strictly speaking, they are asynchronous. For example, the front end 2 and back end 3 are connected to each other so that they can communicate with each other via any telecommunication line such as a LAN (Local Area Network). Note that communication between the front end 2 and back end 3 may be realized via the Internet. The back end 3 and edge 4 are connected to each other so that they can communicate with each other via any telecommunication line such as the Internet.

[0026] Here, the remote control system 1 according to the embodiment is an information processing system (remote monitoring and operation assistance system) that performs remote control in which an operator remotely monitors, assists, and / or operates an autonomously traveling vehicle 46 (mobile body). The remote control system 1 provides various services, such as delivery, security, cleaning, childcare, nursing care, sales, farm work, manufacturing, loading and unloading, transportation, and construction, through its remote control. In this remote control system 1, an operator is assigned in response to a request for assistance (remote request) from a vehicle 46 that requires remote control, and the assigned operator intervenes in the control of the vehicle 46, such as remote operation. The server 30 according to the embodiment is an information processing device (remote control assistance device) that supports the remote control of the vehicle 46. Furthermore, the operator terminal 20 according to the embodiment is an information processing device (terminal device) operated by an operator in a remote control room installed in a control center or the like. Furthermore, the vehicle 46 according to the embodiment is an example of an autonomously traveling vehicle, and is used to provide various services.

[0027] As an example, the remote control system 1 according to the embodiment can be constructed by applying edge computing. In this case, for example, the edge 4 is used as the network edge portion, but other devices may also be used as the edge.

[0028] As an example, the vehicle 46 is an example of a mobile object that performs various tasks, including autonomous driving, related to various services provided by the remote control system 1, such as delivery, security, cleaning, childcare, nursing care, sales, farm work, manufacturing, loading and unloading, transportation, and construction. For example, the vehicle 46 is a mobile object configured to be able to move autonomously and perform predetermined tasks. For example, the vehicle 46 is a mobile object configured to be able to move and perform predetermined tasks according to remote control by an operator who monitors multiple vehicles 46.

[0029] The mobile body is not limited to a vehicle, and various mobile bodies configured to be movable at least in response to remote operation by an operator can be used as appropriate. The mobile body may be, for example, a four-wheeled vehicle or a two-wheeled vehicle. Furthermore, the mobile body may be, for example, an automatic guided vehicle (AGV), or various robots such as construction machinery, agricultural machinery, or drones. Furthermore, these mobile bodies are not limited to those that transport people, but may also transport objects other than people, or may provide specific services other than transportation.

[0030] As an example, when a vehicle 46 detects an obstacle, such as a fallen object or a vehicle parked on the road, in its path and becomes unable to autonomously travel, the vehicle 46 transmits to the server 30 a support request (remote request) requesting remote control, i.e., support through remote assistance or operation. As an example, the server 30 transmits an image captured by a camera mounted on the vehicle 46 to the corresponding operator terminal 20. As an example, when the server 30 receives a support request from the vehicle 46, the server 30 transmits a remote operation request to the operator terminal 20, requesting remote operation by an operator. As an example, the operator terminal 20 displays a display screen including an image (video) captured by a camera 45, such as a camera mounted on the vehicle 46. The display screen includes at least one image (video) of at least one vehicle 46 managed by the operator operating the operator terminal 20. The operator monitors the status of each of the at least one vehicle 46 under his / her care while viewing the display screen of the operator terminal 20. This monitor is an example of an operator's remote monitoring of multiple vehicles 46 based on video data transmitted from each of the multiple vehicles 46. As an example, when the operator terminal 20 receives a remote operation request (remote request) from the server 30, it notifies the operator operating the terminal that assistance through remote operation has been requested. The operator operates the controller 25 connected to the operator terminal 20 while viewing the display screen, and provides assistance such as moving the vehicle 46 by remotely assisting or operating the vehicle 46 that made the assistance request. This assistance of the vehicle 46 through remote control is an example of an operator's remote operation of a remotely operated vehicle 46 out of the multiple vehicles 46 based on video data transmitted from the remotely operated vehicle 46.

[0031] The display screen of the operator terminal 20 may be a screen or image generated by the operator terminal 20 based on display information from the server 30, or may display an image (display information) generated in the server 30.

[0032] In this way, the remote control system 1 of the embodiment is configured to be able to execute an information processing method (remote control support method) that supports remote control, which is the operator's remote monitoring, assistance and / or operation of the vehicle 46, based on remote control images captured by the camera 45.

[0033] Fig. 2 is a diagram showing an example of the configuration of each device included in the remote control system 1 according to the embodiment. Fig. 2 illustrates one operator terminal 20 and one edge device 40, but the configurations of the other operator terminals 20 and edge devices 40 included in the remote control system 1 are similar. Below, the configurations of each of the operator terminal 20, server 30, and edge device 40 will be described with reference to Fig. 2.

[0034] (Example of operator terminal configuration) The operator terminal 20 has a screen transition management unit 201, a mode switching signal transmission unit 202, a driving instruction signal transmission unit 203, a control signal transmission unit 204, an emergency stop signal transmission unit 205, an operator terminal status transmission unit 206, an operator terminal application monitoring unit 207, a vehicle information display unit 208, a vehicle information monitoring unit 209, an input reception unit 210, an input device monitoring unit 211, an emergency stop input unit 212, a vehicle information receiving unit 213, and a mode switching management unit 214.

[0035] The screen transition management unit 201 monitors whether the vehicle is in a remote control state or a remote assistance state, and manages screen transitions to prohibit screen transitions, including logout, when the vehicle is in a remote control state or a remote assistance state. The mode switching signal transmission unit 202 transmits a mode switching signal to the server 30 in response to a switching input from the controller 25. The driving instruction signal transmission unit 203 transmits an automatic driving start signal or an automatic driving stop signal to the server 30 in response to a driving start instruction input to the controller 25 during remote assistance. The driving instruction signal transmission unit 203 assigns the transmission time of the latest vehicle information received by the vehicle information receiving unit 213 to the automatic driving start signal or the automatic driving stop signal. The control signal transmission unit 204 transmits a control signal to the server 30 in response to a control-related input to the controller 25 during remote operation. The control signal transmission unit 204 transmits a time signal, described below, during remote assistance. The control signal transmission unit 204 assigns the transmission time of the latest vehicle information received by the vehicle information receiving unit 213 to the control signal. The emergency stop signal transmitting unit 205 transmits an emergency stop signal to the server 30 .

[0036] The operator terminal status transmission unit 206 monitors whether the processing within the operator terminal 20 and the reception of input from the controller 25 are loop-executed within a specified time period, and periodically notifies the server 30 that the processing is normal. The operator terminal application monitoring unit 207 monitors whether multiple software applications are running within the operator terminal 20 and prohibits the start of a second application. The vehicle information display unit 208 displays information (e.g., camera footage) received by the vehicle information receiving unit 213. The vehicle information monitoring unit 209 verifies the integrity of the vehicle information and monitors delays. The vehicle information monitoring unit 209 performs an emergency stop if the vehicle information is irregular or if the delay exceeds a specified value. Here, performing an emergency stop is an example of restricting the travel of the target vehicle 46 (movement of a moving object). Note that restricting travel may involve bringing the vehicle 46 to a halt (not allowing it to travel) through an emergency stop, restricting its speed, or restricting the execution of various other travel-related functions. The input receiving unit 210 receives a signal from the controller 25.

[0037] The input device monitoring unit 211 monitors whether the controller 25 is disconnected or whether multiple controllers 25 are connected. If the input device monitoring unit 211 detects an abnormality in the controller connection during remote operation and / or remote assistance, it performs an emergency stop. The emergency stop input unit 212 receives a signal from a second emergency stop button 252b (see FIG. 12) provided in addition to the steering wheel 251 of the controller 25. The vehicle information receiving unit 213 requests vehicle information such as camera footage from the server 30 and receives vehicle information transmitted from the vehicle 46 via the server 30. The mode switching management unit 214 monitors whether vehicle information about the vehicle 46 for which remote assistance and / or remote operation is being performed is being received. The mode switching management unit 214 determines whether mode switching is possible based on the vehicle information reception status. As an example, the mode switching management unit 214 retains information indicating whether the edge device 40 is in a remote assistance state or a remote operation state, and determines whether control signals and driving instruction signals can be transmitted based on this information.

[0038] (Controller configuration example) The controller 25 is an input device connected to the operator terminal 20. The controller 25 receives an operation from an operator. Specifically, the controller 25 generates an operation signal in response to the operation from the operator and outputs the generated operation signal to the operator terminal 20.

[0039] (Server configuration example) As shown in FIG. 2 , the server 30 includes a vehicle information transmission unit 301, an operator terminal connection monitoring unit 302, a mode switching signal transmission unit 303, a driving instruction signal transmission unit 304, a control signal transmission unit 305, an emergency stop signal transmission unit 306, an operator terminal status monitoring unit 307, and a server status transmission unit 308.

[0040] The vehicle information transmission unit 301 transmits information (video, etc.) sent from the vehicle 46 to the operator terminal 20, which performs remote control of monitoring, operation, and assistance. The operator terminal connection monitoring unit 302 monitors whether multiple operator terminals 20 are sending control signals or driving instructions to a single vehicle 46. The operator terminal connection monitoring unit 302 processes the signal that started transmission first to be sent to the target vehicle 46. The mode switching signal transmission unit 303 transmits a mode switching signal from the operator terminal 20 to the edge device 40. The driving instruction signal transmission unit 304 transmits a driving start instruction signal and / or a driving stop instruction signal from the paired operator terminal 20 to the edge device 40 during remote assistance.

[0041] The control signal transmission unit 305 transmits a control signal from the paired operator terminal 20 to the edge device 40 during remote operation. Also, during remote assistance, it transmits time information from the paired operator terminal 20 to the edge device 40. The emergency stop signal transmission unit 306 transmits an emergency stop signal from the operator terminal 20 to the edge device 40. The operator terminal status monitoring unit 307 receives a normal information signal from the operator terminal status transmission unit 206 of the operator terminal 20. If the operator terminal status monitoring unit 307 does not receive a normal information signal for a certain period of time during remote operation or remote assistance, it considers this to be an abnormality and performs an emergency stop. The server status transmission unit 308 monitors whether the processing within the server 30 is looping within a specified time. The server status transmission unit 308 periodically notifies the edge device 40 that it is normal.

[0042] Here, "time information" refers to information used to monitor communication delays, such as delay measurement during remote operation assistance. Unlike remote operation, remote assistance does not transmit continuous signals during normal operation. For this reason, remote assistance periodically transmits time information to monitor delays. On the other hand, remote operation does not require separate transmission of "time information" as in remote assistance, since time information is included in the control signals transmitted periodically. As an example, in the remote control system 1 according to the present disclosure, delay measurement may be performed on the side receiving time information included in transmission information such as control signals or vehicle information, or transmitted separately from the transmission information, by comparing the transmission time indicated by the received time information with the reception time.

[0043] In the remote control system 1 according to the present disclosure, delay measurement may be performed by transmitting and receiving, instead of or in addition to "time information," the transmitted information itself, or an "identifier (identification information)" for uniquely identifying the source or destination of the transmitted information or the controlled object. As an example, the identifier transmitting side may manage the identifier and the transmission time by linking them together using, for example, a table (linking table) stored in an internal memory, and when an identifier is received, the table may be referenced to obtain the transmission time, and the obtained transmission time may be compared with the reception time at which the identifier was received, thereby performing delay measurement.

[0044] Furthermore, a "normal information" signal is a signal containing time information and / or an identifier in a specified format that arrives (is received) without delay, or a control signal, i.e., a signal that is received in a state that can be considered normal.

[0045] (Edge device configuration example) As shown in Figure 2, the edge device 40 has a vehicle information transmission unit 401, a vehicle information input unit 402, an instruction control management unit 403, a behavior monitoring unit 404, a driving instruction unit 405, a control unit 406, a signal monitoring unit 407, an emergency stop unit 408, a server status monitoring unit 409, and an edge status transmission unit 410.

[0046] The vehicle information transmission unit 401 transmits vehicle information, such as camera images from the camera 45 and / or the vehicle 46, to the operator terminal 20. The vehicle information transmission unit 401 also assigns an acquisition time to the vehicle information. The vehicle information input unit 402 receives camera images and vehicle body information from the camera 45 and / or the vehicle 46. This vehicle body information may include sensor information from a Light Detection and Ranging (LiDAR), radar, sonar, or a Global Navigation Satellite System (GNSS) system, such as a Global Positioning System (GPS), attached to the vehicle 46, as well as sensing information such as target information, position information, and map information processed from the sensor information. The instruction control management unit 403 determines whether to accept a mode switching signal from the operator terminal 20 based on the vehicle information (vehicle speed). Note that the vehicle information used for this determination is not limited to vehicle speed, but may also include the status of an automated driving system, such as whether the vehicle 46 is driving or stopped (including temporarily stopped), or the manual driving status, such as whether a person is driving the vehicle 46. The instruction control management unit 403 switches whether to send a control signal (during remote operation) or a driving instruction signal (during remote assistance) to the vehicle 46 depending on the remote control mode. The behavior monitoring unit 404 compares the details of the instructions and control given to the vehicle 46 by the control unit 406 and the driving instruction unit 405 (instruction details) with the information from the vehicle 46 (execution details). If the comparison results in a mismatch, the behavior monitoring unit 404 determines that an abnormality has occurred and makes an emergency stop of the vehicle 46. The driving instruction unit 405 sends a driving start instruction and / or a driving stop instruction to the vehicle 46 during remote assistance.

[0047] The control unit 406 controls the operation of the vehicle 46 by transmitting a control signal to the vehicle 46 during remote operation. The signal monitoring unit 407 checks the integrity of the control signal and / or driving instruction signal from the server 30 and monitors delays. If the signal monitoring unit 407 detects irregular information or a delay exceeding a specified value, the signal monitoring unit 407 performs an emergency stop of the vehicle 46. The emergency stop unit 408 issues an emergency stop instruction to the vehicle 46. Specifically, the emergency stop unit 408 transmits a control signal instructing the vehicle 46 to perform an emergency stop. The server status monitoring unit 409 receives normality information from the server status transmission unit 308. If a normality information signal is not received for a certain period of time during remote operation or remote assistance, the server status monitoring unit 409 considers this to be an abnormality and performs an emergency stop. The edge status transmission unit 410 monitors whether the processing within the edge device 40 is loop-executed within a specified time. The edge status transmission unit 410 periodically notifies the vehicle 46 that the processing is normal.

[0048] (Example of camera configuration) The camera 45 is an input device connected to the edge device 40. The camera 45 acquires images for remote control and outputs the acquired images to the edge device 40. Note that this image may be a still image or a moving image (video). Furthermore, this image may be the same as or different from the image used for the autonomous driving of the vehicle 46.

[0049] (Example of vehicle configuration) The vehicle 46 is a mobile object connected to the edge device 40. As described above, the vehicle 46 is a mobile object that performs various tasks, including autonomous driving, related to the various services provided by the remote control system 1.

[0050] 3 is a diagram showing an example of the hardware configuration of an information processing device 8 that realizes the functions of each device included in the remote control system 1 according to the embodiment. The information processing device 8 is a computer that performs overall control of the operation of each device included in the remote control system 1.

[0051] The information processing device 8 that realizes each function of the vehicle 46 may be a computer such as an ECU (Electronic Control Unit) installed inside the vehicle 46, a DCU (Domain Control Unit) such as a CDC (Cockpit Domain Controller) that integrates multiple ECUs, or an OBU (On Board Unit).

[0052] As shown in FIG. 3, the information processing device 8 includes a processor 81, a ROM (Read Only Memory) 82, a RAM (Random Access Memory) 83, and a device I / F (Interface) unit 84.

[0053] The processor 81 is, for example, a CPU (Central Processing Unit), but in addition to or instead of a CPU, at least one of various processors such as a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc. may be used as appropriate. Here, the processor 81 according to the embodiment is an example of at least one processor in the information processing device 8.

[0054] As an example, the processor 81 of the operator terminal 20 executes a program stored in the ROM 82, for example, to realize the functions of a screen transition management unit 201, a mode switching signal transmission unit 202, a driving instruction signal transmission unit 203, a control signal transmission unit 204, an emergency stop signal transmission unit 205, an operator terminal status transmission unit 206, an operator terminal application monitoring unit 207, a vehicle information display unit 208, a vehicle information monitoring unit 209, an input acceptance unit 210, an input device monitoring unit 211, an emergency stop input unit 212, a vehicle information receiving unit 213, and a mode switching management unit 214. As an example, the processor 81 of the server 30 executes a program stored in the ROM 82, for example, to realize the functions of a vehicle information transmission unit 301, an operator terminal connection monitoring unit 302, a mode switching signal transmission unit 303, a driving instruction signal transmission unit 304, a control signal transmission unit 305, an emergency stop signal transmission unit 306, an operator terminal status monitoring unit 307, and a server status transmission unit 308. As an example, the processor 81 of the edge device 40 executes a program stored in, for example, the ROM 82, thereby realizing the functions of a vehicle information transmission unit 401, a vehicle information input unit 402, an instruction control management unit 403, a behavior monitoring unit 404, a driving instruction unit 405, a control unit 406, a signal monitoring unit 407, an emergency stop unit 408, a server status monitoring unit 409, and an edge status transmission unit 410.

[0055] Note that, in the example of FIG. 2, only functions necessary for explaining the main parts of the embodiment are illustrated, but the functions possessed by each device included in the remote control system 1 are not limited to these. In the embodiment, the processor 81 executes a program stored in the ROM 42 to realize each function possessed by each device, including the functions of each part described above. However, this is not limited to this, and some or all of these functions may be realized by dedicated hardware circuits. Furthermore, in each device of the remote control system 1, two or more functions may be integrated and realized as a single function. Similarly, in each device of the remote control system 1, one function may be divided and realized as two or more functions. Furthermore, in the remote control system 1, the functions of two or more devices may be integrated and realized as at least one function of one of the devices. Similarly, in the remote control system 1, the function of one device may be divided and realized as two or more functions of two or more devices.

[0056] The ROM 82 is a non-volatile memory that stores various information including programs executed by the processor 81. The memory of the information processing device 8 is not limited to the ROM 82, and various recording media and recording devices such as a hard disk drive (HDD), a solid state drive (SSD), and flash memory can be used as appropriate. The RAM 83 is a volatile memory that has a working area for the processor 81. The device I / F unit 84 is an interface for connecting each device included in the remote control system 1 to other devices of the information processing device 8, such as a communication device (not shown), a display device (not shown), and an input device (not shown).

[0057] As an example, the device I / F unit 84 of the operator terminal 20 connects to the controller 25. As an example, the device I / F unit 84 of the edge device 40 connects to the camera 45 and the vehicle 46.

[0058] (About the expected use case) Here, a description will be given of a use case of the remote control system 1 assumed in the embodiment of the present disclosure. The remote control system 1 according to the embodiment monitors communication delays, video quality, and system normality during remote assistance (S16a to S21a in FIG. 4) and remote operation (S16b to S21b in FIG. 7).

[0059] (Remote assistance use cases) Fig. 4 is a flowchart showing an example of a flow of remote assistance in remote control according to the embodiment Fig. 5 is a diagram for explaining an example of a scene of the remote assistance in Fig. 4.

[0060] First, assume that a vehicle 46 becomes stuck while autonomously traveling (S11). FIG. 5 illustrates a vehicle 46 that has stopped autonomous traveling (become stuck) at an intersection 601 due to the presence of a stop line 603 on the vehicle's path 602. At this time, an operator may be remotely monitoring the vehicle 46. Then, a remote assistance request is transmitted from the stuck vehicle 46 to the server 30 via the edge device 40 (S12a). Thereafter, the operator terminal 20 selects the vehicle 46 that issued the remote assistance request by operating the controller 25 by the operator in response to a notification from the server 30 (S13a). Furthermore, an image for remote assistance of the vehicle 46 that issued the remote assistance request is displayed on the operator terminal 20 (S14).

[0061] If remote assistance is to be performed (S15a: Yes), the operator presses an assistance start button provided on controller 25 (S16a), and a safety check is performed around target vehicle 46 based on images such as camera footage (S17a). If, as a result of the safety check, it is determined that remote travel cannot be started (S18: No), the operator presses an assistance end button provided on controller 25 (S19a), and the system transitions to a standby state for remote assistance (S20a). For example, in the example of FIG. 5, the operator determines, based on camera footage or the like, that there is another traveling vehicle 604 crossing vehicle path 602 of vehicle 46, and that it is not possible to remotely start travel. Thereafter, the flow of FIG. 4 returns to step S15a.

[0062] On the other hand, if it is determined that remote driving can be started as a result of the operator's safety check (S18: Yes), then when the operator presses the driving start button provided on the controller 25 (S21a), automatic driving by the vehicle 46 begins (S22a). If remote assistance is not provided (S15a: No), or after remote assistance (S16a to S22a), the selection of the target vehicle 46 is cancelled (S23), and the flow in FIG. 4 ends.

[0063] (Remote control use cases) Fig. 6 is a flowchart showing an example of the flow of remote operation in remote control according to the embodiment. Here, differences from the flow in Fig. 4 will be mainly explained, and overlapping explanations will be omitted as appropriate. Fig. 7 is a diagram for explaining an example of a scene of the remote operation in Fig. 4.

[0064] 7 illustrates a vehicle 46 that has stopped autonomous driving (becomes stuck) due to the presence of a parked vehicle 606 on the vehicle's path 602 on a travel path 605 such as a road. At this time, an operator may be remotely monitoring the vehicle. Then, a remote control request is transmitted from the stuck vehicle 46 to the server 30 via the edge device 40 (S12b). Thereafter, the vehicle 46 that issued the remote control request is selected (S13a), and an image for remotely controlling the vehicle 46 is displayed (S14).

[0065] If remote control is to be performed (S15b: Yes), the operator presses an operation start button provided on controller 25 (S16b), and remotely controls the target vehicle 46 based on images such as camera footage to avoid getting stuck (avoiding roadside parking in the example of FIG. 7) (S17b). If it is determined that the vehicle cannot be remotely started to travel as a result of avoiding getting stuck (S18: No), the operator presses a remote control end button provided on controller 25 (S19b), and the system transitions to a standby state (S20b). Thereafter, the flow in FIG. 6 returns to step S15b.

[0066] On the other hand, if it is determined that it is OK to start traveling remotely as a result of the operator's efforts to avoid the cause of the stuck vehicle (S18: Yes), and the operator presses a traveling restart button provided on the controller 25 (S21b), the vehicle 46 starts traveling automatically (S22b). If no remote control is performed (S15b: No), or after remote control (S16b to S22b), the selection of the target vehicle 46 is cancelled (S23), and the flow in FIG. 6 ends.

[0067] (Regarding anticipated hazards) Here, a description will be given of hazards that are expected in the remote operation and remote assistance of the remote control according to the embodiment of the present disclosure. Fig. 8 is a diagram for explaining hazards that are expected in the remote control according to the embodiment.

[0068] As an example, a possible anomaly on the front end 2 side (FE anomaly: H1) could occur, such as an operator accidentally logging out during remote operation or remote assistance, causing the system to suddenly become inoperable. As an example, a possible anomaly on the front end 2 side (FE anomaly: H2) could occur, such as an operator terminal 20 failing during remote operation or remote assistance, causing the front end 2 to stop, causing the system to suddenly become inoperable. As an example, a possible anomaly on the back end 3 side (BE anomaly: H3) could occur, such as an server 30 failing during remote operation or remote assistance, causing the system to suddenly become inoperable. As an example, a possible anomaly on the edge 4 side (ED anomaly: H4) could occur, such as an edge device 40 failing during remote operation or remote assistance, causing the system to suddenly become inoperable.

[0069] As an example, an abnormality (stop system duplication: H5) in which the controller 25 does not accept operations can be assumed, such as when the controller 25 suddenly breaks down and becomes inoperable during remote operation or remote assistance.

[0070] As an example, a possible anomaly is that the operator terminal 20 and / or server 30 malfunctions during remote operation or remote assistance, causing the same control signal to be repeatedly transmitted, resulting in the control signal continuing to be transmitted even after the remote operation is stopped (vehicle information / control signal anomaly: H6).As an example, a possible anomaly is that the control signal is delayed during remote operation, resulting in the communication system malfunctioning during remote operation or remote assistance, causing a delay in the transmission of the control signal (vehicle information / control signal anomaly: H7).

[0071] As an example, an abnormality (control abnormality, vehicle information / control signal abnormality: H8) may occur in which an unintended control signal is sent, such as a malfunction of the operator terminal 20 and / or server 30 during remote operation or remote assistance, causing an incorrect control signal to be sent, or a malfunction of the operator terminal 20 and / or server 30 causing a control signal to be sent even when the state is other than remote operation or remote assistance.

[0072] As an example, a possible anomaly could occur during remote operation where a control signal cannot be transmitted due to a failure of the operator terminal 20, server 30, and / or communication system during remote operation or remote assistance (vehicle information / control signal anomaly: H9). As an example, a possible anomaly could occur during remote operation or remote assistance where an emergency stop signal cannot be sent due to a deterioration in communication conditions, such as a malfunction in the communication system during remote operation or remote assistance (vehicle information / control signal anomaly: H10). As an example, a possible anomaly could occur where the operator does not notice a slight delay in the video (video / control signal anomaly: H11), which could lead to an incorrect decision if the operator does not correctly recognize the magnitude of the delay during remote operation or remote assistance. As an example, a possible anomaly could occur during remote operation assistance where a video is delayed due to a large delay during remote operation or remote assistance, which could lead to an incorrect decision or operation (video / control signal anomaly: H12).

[0073] As an example, if the image freezes during remote operation or remote assistance, the operator may make an incorrect operation or judgment, which could be an abnormality where the image freezes (image abnormality: H13). As an example, if the image quality (image quality, etc.) is low during remote operation or remote assistance, the operator may make an incorrect operation or judgment, which could be an abnormality where the image quality (image quality, etc.) deteriorates and the operator is unable to correctly recognize the surroundings from the image (image abnormality: H14). As an example, if the operator is unable to see the surrounding conditions on the image during remote operation or remote assistance, the operator may make an incorrect operation or judgment, which could be an abnormality where the operator is unable to correctly recognize the surroundings from the image due to setting sun, backlighting, camera abnormalities, etc. (image abnormality: H15).

[0074] As an example, during remote operation or remote assistance, a malfunction in the operator terminal 20 and / or server 30 may occur, causing the image displayed on the screen and the destination of the control signal to differ, resulting in the operator operating a vehicle 46 different from the one he or she recognizes, resulting in an abnormality (image linking: H16) in which the vehicle image displayed on the front end 2 and the destination of the control signal differ.

[0075] As one example, a possible abnormality (terminal restriction: H17) could occur where multiple remote control applications are started and remotely controlled or remotely assisted simultaneously for multiple vehicles 46, such as when multiple apps are launched in the operator terminal 20 during remote operation or remote assistance, causing the operator to unintentionally operate other vehicles 46. As another example, a possible abnormality (terminal restriction: H18) could occur where the same vehicle 46 is remotely controlled or remotely assisted from multiple terminals, such as when multiple operators remotely control or remotely assist the same vehicle 46 during remote operation or remote assistance, causing the vehicle 46 to run out of control.

[0076] As an example, an abnormality (notification multiplexing: H19) can be expected in which a vehicle can be remotely operated even when the device that issues warnings to people around the vehicle is broken, such as when the vehicle is operated without any means of communication with people around the vehicle during remote operation or remote assistance.

[0077] As an example, an abnormality may occur in which an unintended mode switching request is sent (mode switching abnormality: H20), such as a malfunction of the operator terminal 20 and / or server 30 during remote operation or remote assistance, causing an unintended request to switch between remote operation, remote assistance, or automatic driving, resulting in a mode switch (for example, the vehicle 46 suddenly starting to move).

[0078] As an example, an abnormality (mode switching abnormality, vehicle information / control signal abnormality: H21) may occur in which a state (mode) discrepancy occurs between the front end 2, back end 3, and edge 4, such as a malfunction of the operator terminal 20 and / or server 30 during remote operation or remote assistance, causing the original mode to differ from the mode displayed on the screen of the operator terminal 20.

[0079] If any device or part of the device in the remote control system fails, it is expected that the above-mentioned hazards will occur in relation to remote operation and remote assistance. Below, we will explain each embodiment of the remote control system 1 that can ensure the safety of remote control of autonomous driving mobility regardless of these hazards. Note that in each embodiment of the present disclosure, the movement of the vehicle 46 is considered a risk, and a state in which the vehicle is stopped is considered safe.

[0080] Next, an example of the operation of each embodiment of the remote control system 1 configured as described above will be described with reference to the drawings. Note that the operation procedures and processing flows described below are examples, and the order of steps can be changed, some steps can be deleted, and other steps can be added as desired.

[0081] In the following description of each embodiment, the commonalities with the above-described embodiment will be omitted as appropriate, and the differences will be mainly described. Furthermore, the following embodiments can be combined as appropriate.

[0082] (First embodiment) FIG. 9 is a flowchart showing an example of the flow of information processing executed by the operator terminal 20 according to the first embodiment. In the operator terminal 20, the screen transition management unit 201 receives mode information from the mode switching management unit 214 (S101). Here, the mode information includes information indicating whether the remote control mode is "remote assistance" or "remote operation." If the remote control mode is other than "remote assistance" or "remote operation" (S102: No), the screen transition management unit 201 enables an operation button (see FIG. 10) that causes a screen transition when pressed, and sets it to an active state (S103). On the other hand, if the remote control mode is "remote assistance" or "remote operation" (S102: Yes), the screen transition management unit 201 disables an operation button that causes a screen transition when pressed, and sets it to an inactive state (S104).

[0083] FIG. 10 is a diagram illustrating an example of a screen display on the operator terminal 20 according to the first embodiment. The display screen 610 in FIG. 10 is an example of a screen display presented to an operator performing remote control. The display screen 610 in FIG. 10 includes at least one image 611 for remote control and a map 613 of the area surrounding the remotely controlled vehicle 46. Note that, in the example of FIG. 10, a forward image 611a, a leftward image 611b, a rightward image 611c, and a rearward image 611d are illustrated as examples, but are not limited thereto. The image 611 on the display screen 610 may be any one of these images, or may further include other images. The display screen 610 also illustrates a help button 615, a settings button 616, a logout button 617, and buttons 618a to 618d for transitioning to other screens. The help button 615 is an operation button for instructing transition to a help screen, such as a screen for displaying reference information. The settings button 616 is an operation button for instructing transition to a screen for performing various settings. The logout button 617 is an operation button for instructing logout or transition to a screen for logout processing. The transition buttons 618a to 618d to other screens are operation buttons for instructing transition to pre-assigned transition destination screens.

[0084] The help button 615, the settings button 616, the logout button 617, and the transition buttons 618a to 618d to other screens are examples of operation buttons that cause a screen transition when pressed. In other words, these operation buttons can be expressed as operation buttons that trigger a screen transition. In the present disclosure, a "screen transition" triggered by an operation button that is switched between active and inactive is not limited to switching a currently displayed screen to another screen, but also includes superimposing another screen or information on a part or all of a currently displayed screen. Examples of such displays include a display that enlarges or expands a part of the screen before the transition, or a pop-up display. The operation button that is switched between active and inactive may be an operation button on all display screens that cause a "screen transition" when pressed, or may be an operation button on only some of those display screens. The operation button that triggers a screen transition may be predetermined and stored in the internal memory of the operator terminal 20, for example.

[0085] As described above, in the remote control according to this embodiment, screen transitions and logout during remote operation and / or remote assistance are prohibited. This configuration ensures safety in the event of an abnormality on the front end 2 side (FE abnormality: H1) in which the operator logs out of the system, such as when the operator accidentally logs out and suddenly becomes unable to operate the system. Furthermore, the configuration that prohibits screen transitions ensures safety in the event of an abnormality (terminal restriction: H17) in which the operator unintentionally operates other vehicles 46, such as when an operation is accepted on a screen that is displayed behind the operator and is not recognized by the operator.

[0086] (Second embodiment) Fig. 11 is a flowchart showing an example of the flow of information processing executed by the operator terminal 20 according to the second embodiment. The flow in Fig. 11 illustrates a case where the operator presses the emergency stop button 252 (see Fig. 12) of the controller 25. In the operator terminal 20, the input receiving unit 210 or the emergency stop input unit 212 acquires the operator's operation on the emergency stop button 252 (S201). Then, the emergency stop signal transmitting unit 205 transmits an emergency stop signal to the server 30 (S202).

[0087] Fig. 12 is a diagram showing an example of the configuration of a controller 25 according to the second embodiment. As shown in Fig. 12, the controller 25 has a first emergency stop button 252a provided on the hub of a steering wheel 251 that is gripped by an operator, for example. An operation by the operator on this first emergency stop button 252a is acquired by the input receiving unit 210. The controller 25 also has a second emergency stop button 252b provided separately and in a different location from the first emergency stop button 252a. An operation by the operator on this second emergency stop button 252b is acquired by the emergency stop input unit 212.

[0088] As described above, in the remote control system 1 according to this embodiment, the emergency stop button 252 is multiplexed. This multiplexing may be achieved by using three or more emergency stop buttons. The multiplexing of the emergency stop button 252 may also be achieved by providing an emergency stop button on a screen displayed on the operator terminal 20 for receiving an operator operation, such as a click or a touch. In the remote control according to this embodiment, when an operation on at least one of the emergency stop buttons 252 is received, an emergency stop signal is transmitted from the operator terminal 20 to the server 30. This configuration ensures safety in the event of an abnormality in which the controller 25 does not accept an operation (dual-redundant stop system: H5), such as when the controller 25 suddenly breaks down and becomes inoperable during remote operation or remote assistance.

[0089] (Third embodiment) 13 is a flowchart showing an example of the flow of information processing executed by the operator terminal 20 according to the third embodiment. In the operator terminal 20, the mode switching management unit 214 receives mode information stored, for example, in an internal memory (S301). As an example, the mode information indicating whether the vehicle 46 is in a remotely controlled state or a remotely assisted state is transmitted from the vehicle information input unit 402 in the edge device 40 to the server 30 via the vehicle information transmission unit 401, and then transmitted to the corresponding operator terminal 20 by the vehicle information transmission unit 301 of the server 30 that has received the mode information. The vehicle information reception unit 213 in the operator terminal 20 receives the mode information and transmits it to the mode switching management unit 214, where it is stored.

[0090] If the remote control mode indicated by the mode information is “remote operation” or “remote assistance” (S302: Yes), the input receiving unit 210 acquires the connection status of the controller 25 to the operator terminal 20 (S303). Then, the input device monitoring unit 211 determines whether the controller 25 is not connected or whether multiple controllers 25 are connected (S304). As an example, the input device monitoring unit 211 determines the connection status of the controller 25 by monitoring the transmission cycle of a control signal periodically transmitted from the connected controller 25. For example, if the control signal is not received or if the transmission cycle is delayed by more than a predetermined threshold value stored in the internal memory, the controller 25 is determined to be not connected (disconnected). If the controller 25 is not connected or multiple controllers 25 are connected (S304: Yes), the emergency stop signal transmitting unit 205 transmits an emergency stop signal to the server 30. After the emergency stop signal is transmitted, or if one controller 25 is connected (S304: No), the flow in FIG. 13 ends. In the present disclosure, when an emergency stop signal is transmitted, the vehicle 46 is brought to an emergency stop. At this time, output from the source and / or destination of the emergency stop signal may also be stopped.

[0091] On the other hand, if the remote control mode indicated by the mode information is other than "remote operation" and "remote assistance" (S302: No), the input receiving unit 210 acquires the connection status of the controller 25 to the operator terminal 20 (S306). Then, if the controller 25 is not connected or multiple controllers 25 are connected (S304: Yes), the input device monitoring unit 211 disables the remote operation switching button that switches the remote control mode to "remote operation" when pressed and the remote operation switching button that switches the remote control mode to "remote assistance" when pressed, and sets them to inactive states (S308-S309). Then, the flow in FIG. 13 ends.

[0092] As described above, in the remote control according to this embodiment, during remote operation and / or remote assistance, monitoring is performed in the direction of controller 25←front end 2←back end 3←edge 4←vehicle 46, and an emergency stop is performed if the monitored object does not operate correctly. Specifically, the operator terminal 20 monitors the connection status of the controller 25 during remote operation assistance, and performs an emergency stop if a disconnection or multiple connections is detected. This configuration ensures safety in the event of an abnormality on the front end 2 side (FE abnormality: H2) that causes the front end 2 to stop during remote operation or remote assistance.

[0093] (Fourth embodiment) First, a description will be given of internal processing of the operator terminal 20 in the remote control system 1 according to this embodiment. Fig. 14 is a sequence diagram showing an example of the flow of information processing executed by the operator terminal according to the fourth embodiment.

[0094] The operator terminal 20 is configured to execute two predetermined internal processes, each of which outputs an execution signal, in the remote operation assistance. While FIG. 14 illustrates an example in which there are two internal processes to be monitored, this is not limiting. The internal processes to be monitored may be one or three or more processes. When a first process is being executed in the operator terminal 20, the operator terminal status transmission unit 206 of the operator terminal 20 receives a first execution signal transmitted from the first process (S401) and retains the time of acquisition (S402). Similarly, when a second process is being executed in the operator terminal 20, the operator terminal status transmission unit 206 receives a second execution signal transmitted from the second process (S403) and retains the time of acquisition (S404).

[0095] The operator terminal status transmitting unit 206 then periodically executes the processes of S405 to S409. It is assumed that information defining the cycle and timing of this periodic execution is predetermined and stored, for example, in the internal memory of the operator terminal 20. The operator terminal status transmitting unit 206 compares the acquisition time of each of the first execution signal and the second execution signal held therein with the current time (S405 to S406). If the difference between the acquisition time of both the first execution signal and the second execution signal and the current time is within a specified time, the operator terminal status transmitting unit 206 increments the counter value by a specified amount, assigns an error check code to the count value (S408), and periodically transmits this as a normal status notification to the server 30 (S409). This normal status notification is an example of a notification signal that is different from the driving instruction signal and the control signal, and is also an example of a heartbeat signal. The specified time, which is the threshold value of the difference between the acquisition time and the current time, and the specified amount, which is the increment (or decrement) per count of the counter value, are assumed to be predetermined and stored in, for example, the internal memory of the operator terminal 20.

[0096] Next, the flow of processing during remote operation assistance between the operator terminal 20, server 30, and edge device 40 according to this embodiment will be described. FIG. 15 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system 1 according to the fourth embodiment. First, the operator terminal status monitoring unit 307 acquires a normal status notification from the operator terminal 20 (S501), and performs a count value check (S502) and an error check (S503). In other words, the operator terminal status monitoring unit 307 checks whether the normal status notification from the operator terminal 20 is valid data.

[0097] If a normal state notification is not received or if the normal state notification is invalid data, the operator terminal state monitoring unit 307 issues an emergency stop instruction (S504). In this case, the emergency stop signal transmission unit 306 transmits an emergency stop signal to the edge device 40 (S505). In addition, in the edge device 40 that has received this, the emergency stop unit 408 executes an emergency stop (S506). Thereafter, the vehicle information transmission unit 401 and the vehicle information input unit 402 transmit a stop notification to the server 30 (S507). In addition, in the server 30 that has received this, the driving instruction signal transmission unit 304 and the control signal transmission unit 305 stop transmitting the control signal and the driving instruction signal (S508). In addition, the vehicle information transmission unit 301 transmits a remote operation assist mode cancellation notification to the operator terminal 20 (S509). In addition, in the operator terminal 20 that has received this, the vehicle information display unit 208 displays the remote operation assist mode cancellation notification on the screen (S510). Furthermore, the driving instruction signal transmitting unit 203 and the control signal transmitting unit 204 stop transmitting the control signal and driving instruction signal (S511).

[0098] As described above, the remote control according to this embodiment is provided with a watchdog that monitors the controller 25 ← front end 2 ← back end 3 ← edge 4 ← vehicle 46 during remote operation and / or remote assistance. An emergency stop is performed if the monitored object does not operate correctly. Specifically, the server 30 monitors the operator terminal 20 during remote assistance, and performs an emergency stop if a disconnection or partial processing stop is detected. That is, the vehicle 46 is brought to an emergency stop, and output from the server 30 or edge device 40 is stopped. Furthermore, an abnormality in the operator terminal 20 can be notified on the operator terminal 20 screen. This configuration ensures safety in the event of an abnormality on the front end 2 side (FE abnormality: H2), in which the front end 2 stops during remote operation or remote assistance. The configuration for notifying the operator terminal 20 on the operator terminal 20 screen that an abnormality in the operator terminal 20 occurs is not essential and may not be provided.

[0099] (Modification of the fourth embodiment) 16 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system 1 according to a modified example of the fourth embodiment. Here, a case where an emergency stop is performed on the first operator terminal 20a in the flow of FIG. 15 will be described.

[0100] In the server 30, the vehicle information transmission unit 301 transmits a notification of cancellation of the remote operation assist mode to the first operator terminal 20a (S509), and then transmits a response notification requesting a response to the stopped remote control to the second operator terminal 20b other than the first operator terminal 20a. In the second operator terminal 20b that receives this, the vehicle information display unit 208 presents the response notification requesting a response to the stopped remote control to the operator in the first operator terminal 20a, for example, by displaying it on a screen.

[0101] As described above, in the remote control according to this modification, if a disconnection or a partial processing stop is detected by monitoring the operator terminal 20, an emergency stop is performed and a response notification is sent to another operator terminal 20. According to this configuration, the emergency stop ensures safety in the event of an abnormality on the front end 2 side (FE abnormality: H2), and remote control of the target vehicle 46 can be continued.

[0102] (Fifth embodiment) The following describes the flow of processing during remote operation assistance among the operator terminal 20, the server 30, and the edge device 40 according to this embodiment. Fig. 17 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system 1 according to the fifth embodiment.

[0103] First, the server status transmission unit 308 in the server 30 executes at least one server internal process similar to the operator terminal internal process of Fig. 14, and periodically transmits a normal status notification (heartbeat signal) with an error check code added to the count value to the edge device 40 (S601). Furthermore, in the edge device 40 that receives this, the server status monitoring unit 409 checks whether the normal status notification from the server 30 is valid as data, similar to the process of S502 to S503 of Fig. 14 (S602 to S603). If the normal status notification is not received, or if the normal status notification is invalid as data, the emergency stop unit 408 executes an emergency stop (S506). Thereafter, the processes of S507 to S511 are executed similar to the flow of Fig. 14.

[0104] As described above, the remote control according to this embodiment is provided with a watchdog that monitors the controller 25 ← front end 2 ← back end 3 ← edge 4 ← vehicle 46 during remote operation and / or remote assistance. An emergency stop is performed if the monitored object does not operate correctly. Specifically, the edge device 40 monitors the server 30 during remote assistance, and performs an emergency stop if a disconnection or partial processing stop is detected. Furthermore, an abnormality in the server 30 can be notified on the screen of the operator terminal 20. This configuration ensures safety in the event of an abnormality on the back end 3 side (BE abnormality: H3) that causes the back end 3 to stop, such as when the server 30 suddenly breaks down and becomes inoperable during remote assistance. The configuration for notifying the operator terminal 20 of an abnormality in the server 30 on its screen is not essential and may be omitted.

[0105] (Sixth embodiment) The following describes the flow of processing during remote operation assistance among the operator terminal 20, server 30, edge device 40, and vehicle 46 according to this embodiment. Fig. 18 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system 1 according to the sixth embodiment.

[0106] First, the edge status transmission unit 410 in the edge device 40 executes at least one edge device internal process similar to the operator terminal internal process of Fig. 14 and periodically transmits a normal status notification (heartbeat signal) with an error check code added to a count value to, for example, the vehicle 46 that is the target of the remote operation assistance (S701). Here, in the on-board computer (information processing device 8) of the vehicle 46 according to this embodiment, the processor 81 is configured to be able to realize the function of an edge device status monitoring unit that checks whether the normal status notification from the edge device 40 is legitimate data. This edge device status monitoring unit is a functional unit similar to the operator terminal status monitoring unit 307 or server status monitoring unit 409 described above.

[0107] Furthermore, the edge device status monitoring unit of the vehicle 46 that has received the normal status notification from the edge device 40 checks whether the normal status notification from the edge device 40 is valid as data, similar to the processing of S502 to S503 in FIG. 14 (S702 to S703). If the normal status notification is not received, or if the normal status notification is invalid as data, the vehicle 46 performs an emergency stop in the same manner as when the emergency stop unit 408 of the edge device 40 performs an emergency stop (S704). The vehicle 46 also transmits a stop notification to the edge device 40 (S705). In the edge device 40 that has received this, the vehicle information transmission unit 401 and the vehicle information input unit 402 transmit the stop notification to the server 30 (S507). Thereafter, the processing of S508 to S511 is performed in the same manner as the flow of FIG. 14.

[0108] As described above, the remote control according to this embodiment is provided with a watchdog that monitors the controller 25 ← the front end 2 ← the back end 3 ← the edge 4 ← the vehicle 46 during remote operation and / or remote assistance. An emergency stop is performed if the monitored object does not operate correctly. Specifically, the vehicle 46 monitors the edge device 40 during remote assistance, and performs an emergency stop if a disconnection or partial processing stop is detected. Furthermore, an abnormality in the edge device 40 can be notified on the screen of the operator terminal 20. This configuration ensures safety in the event of an abnormality on the edge 4 side (ED abnormality: H4) that causes the edge 4 to stop, such as when the edge device 40 suddenly breaks down and becomes inoperable during remote assistance. The configuration for notifying the operator terminal 20 of an abnormality in the edge device 40 on the screen is not essential and may not be provided.

[0109] (Seventh embodiment) A description will be given of the flow of processing during remote operation assistance among the operator terminal 20, the server 30, and the edge device 40 according to this embodiment. Fig. 19 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system 1 according to the seventh embodiment.

[0110] First, the vehicle information input unit 402 in the edge device 40 acquires an image (camera video) from the camera 45 (S801), and adds the acquisition time to the acquired camera video (S802). The vehicle information transmission unit 401 then transmits the camera video with the added time information to the server 30. The vehicle information transmission unit 301 in the server 30 that receives the camera video transmits the received camera video to the operator terminal 20 (S804).

[0111] In the remote operation assistance state, the vehicle information monitoring unit 209 in the operator terminal 20 compares the current time with the video acquisition time in the edge device 40 (S805). Then, if the comparison result, i.e., the video transmission delay, exceeds a predetermined threshold value stored in the internal memory, the emergency stop signal transmitting unit 205 transmits an emergency stop signal to the server 30 (S806). In the server 30 that receives this, the emergency stop signal transmitting unit 306 transmits the emergency stop signal to the edge device 40 (S807). In the edge device 40 that receives this, the emergency stop unit 408 performs an emergency stop of the vehicle 56 (S808).

[0112] The threshold for video transmission delay, i.e., the allowable delay amount, differs depending on the remote control mode. For example, the allowable delay amount for "remote assistance" is greater than the allowable delay amount for "remote operation." As an example, the allowable delay amount for "remote assistance" is 500 ms, and the allowable delay amount for "remote operation" is 300 ms. The allowable delay amount may also be changed depending on, for example, the speed of the vehicle 46.

[0113] In this way, in the remote control according to this embodiment, if there is a delay in the camera image during remote operation assistance, an emergency stop is performed. This configuration reduces the possibility of the operator making an error in operation or judgment when an abnormality occurs during remote operation assistance, such as an image delay (image / control signal abnormality: H12) or an image freeze (image abnormality: H13), thereby ensuring safety.

[0114] (Eighth embodiment) FIG. 20 is a diagram showing an example of a screen display on the operator terminal 20 according to the eighth embodiment.

[0115] FIG. 20 illustrates a display screen 620 during remote operation assistance. As shown in FIG. 20, on the display screen 620 during remote operation assistance, at least one image 621 for remote control displays a video transmission delay time 622. Furthermore, if the delay time 622 exceeds a predetermined threshold stored in an internal memory, an alert 623 such as a "Caution Alert" may be displayed on the display screen 620. The video transmission delay time threshold for this alert display may be smaller than or equal to the delay time threshold for an emergency stop, for example. The delay time may be displayed continuously or may be displayed at the timing when the alert 623 is displayed, for example, to reduce oversight due to familiarity. Furthermore, when displaying the delay time, the color of the text and background color of the delay display may be changed depending on the magnitude of the delay. For example, a delay time of 0-800 ms may be displayed in blue, a delay time of 800-1500 ms may be displayed in yellow, and an abnormality such as exceeding 1500 ms may be displayed in red.

[0116] In this way, in the remote control according to this embodiment, the delay time is displayed along with the camera image during remote operation assistance. This configuration reduces the possibility of the operator making an error in operation or judgment in the event of an abnormality where the operator does not notice a slight delay in the image during remote operation assistance (image / control signal abnormality: H11), thereby ensuring safety.

[0117] (Ninth embodiment) The following describes the flow of processing during remote operation assistance among the controller 25, operator terminal 20, server 30, and edge device 40 according to this embodiment. Figures 21A and 21B are sequence diagrams showing an example of the flow of information processing executed by each unit of the remote control system 1 according to the ninth embodiment.

[0118] (When remotely controlled) 21A illustrates a flow during remote control. First, in the edge device 40, the vehicle information input unit 402 acquires vehicle information from the camera 45 and / or the vehicle 46, and assigns the time of acquisition, "time A," to the acquired vehicle information (S901). Then, the vehicle information sending unit 401 sends the vehicle information with the time information assigned to it to the server 30 (S902). In the server 30 that receives this, the vehicle information transmission unit 301 transmits the received vehicle information to the operator terminal 20 (S903). In the operator terminal 20 that receives this, the vehicle information receiving unit 213 acquires and holds the "time A" assigned to the received vehicle information (S904). Then, the vehicle information display unit 208 displays information about the vehicle 46 that is the target of remote control based on the received vehicle information (S905).

[0119] In the operator terminal 20, the input receiving unit 210 acquires an operation signal related to remote operation output from the controller 25 in response to an operation by the operator (S906a). The control signal transmitting unit 204 assigns "time A," the count value of the counter, and an error check code to a control signal corresponding to the acquired operation signal (S907a), and transmits the signal to the server 30 (S908a). In the server 30 that receives this, the control signal transmitting unit 305 transmits the received signal to the edge device 40 (S909a). In the edge device 40 that receives this, the signal monitoring unit 407 extracts "time A," the count value of the counter, and the error check code from the received signal (S910a).

[0120] During remote operation, the remote control system 1 repeatedly executes the processes of S901 to S910a. During remote operation, the remote control system 1 repeatedly executes the following processes of S911 to S915 in addition to the processes of S901 to S910. The processes of S911 to S915 may be executed each time the processes of S901 to S910a are executed, or may be executed each time the processes of S901 to S910a are executed multiple times.

[0121] During remote control, the signal monitoring unit 407 verifies the extracted counter value and error check code (S911) and performs a delay determination for the received control signal (S912a). In the delay determination, the signal monitoring unit 407 determines whether the delay time, which is the difference between the current time and "time A," exceeds a specified value. This specified value is assumed to be, for example, predetermined and stored in an internal memory. Note that this specified value may be the same as or different from the threshold value (allowable delay amount) related to the delay in video transmission according to the seventh embodiment.

[0122] If the delay time of the received control signal is equal to or longer than a specified value, if there is a counter abnormality, or if the message based on the error check code is irregular, the emergency stop unit 408 performs an emergency stop (S913). Here, a counter abnormality is an abnormality in which, when verifying the count value of the extracted counter, the extracted count value is equal to or less than the count value held in the edge device 40. For example, if there is an abnormality in which the same signal is continuously output from the operator terminal 20, the extracted count value matches the count value held in the edge device 40.

[0123] On the other hand, if the delay time of the received control signal is less than the specified value, the counter is normal, and the message based on the error check code is valid, the driving instruction unit 405 and the control unit 406 execute control based on the received control signal (S914). Furthermore, the signal monitoring unit 407 stores the count value of the counter extracted from the received signal (S915). The stored count value is used for verification in the next processing of S911.

[0124] (When using remote assistance) Figure 21B illustrates an example of a flow during remote assistance. Here, differences from the flow in Figure 21A will be mainly explained. During remote assistance, the remote control system 1 repeatedly executes the processes of S901 to S910c. During remote assistance, the remote control system 1 further repeatedly executes the processes of S911 to S915 in addition to the processes of S901 to S910c. The processes of S911 to S915 may be executed each time the processes of S901 to S910c are executed, or may be executed each time the processes of S901 to S910c are executed multiple times.

[0125] When a driving instruction operation (instruction operation) is performed during remote assistance, the input receiving unit 210 in the operator terminal 20 acquires an operation signal (instruction signal) related to the driving instruction output from the controller 25 in response to the operator's operation (S906b). The control signal transmitting unit 204 assigns "time A," the count value of the counter, and an error check code to a driving instruction signal (driving start instruction) corresponding to the acquired instruction signal (S907b), and transmits the signal to the server 30 (S908b). The control signal transmitting unit 305 in the server 30 that receives this signal transmits the received signal to the edge device 40 (S909b). Furthermore, the signal monitoring unit 407 in the edge device 40 that receives this signal extracts "time A," the count value of the counter, and the error check code from the received signal (S910b).

[0126] Furthermore, each time the processes of S901 to S910c are repeatedly executed, the control signal transmitting unit 204 assigns "Time A", the counter count value, and an error check code to the time signal (time information) (S907c), and transmits the signal to the server 30 (S908c). In the server 30 that receives this signal, the control signal transmitting unit 305 transmits the received signal to the edge device 40 (S909c). In the edge device 40 that receives this signal, the signal monitoring unit 407 extracts "Time A", the counter count value, and the error check code from the received signal (S910c).

[0127] Furthermore, under remote control, the signal monitoring unit 407 verifies the extracted counter value and error check code (S911) and executes a delay determination for the received time signal and / or driving instruction signal (S912b). In the delay determination, the signal monitoring unit 407 determines whether the delay time, which is the difference between the current time and "Time A," exceeds a specified value. This specified value is assumed to be, for example, predetermined and stored in an internal memory. Note that this specified value may be the same as or different from the threshold value (allowable delay amount) for video transmission delay according to the seventh embodiment. Furthermore, it may be the same as or different from the specified value for the control signal in FIG. 21A.

[0128] If the delay time of the received time signal and / or driving instruction signal is equal to or greater than a specified value, if the counter is abnormal, or if the message based on the error check code is irregular, the emergency stop unit 408 performs an emergency stop (S913). On the other hand, if the delay time of the received time signal and / or driving instruction signal is less than a specified value, if the counter is normal, and if the message based on the error check code is regular, the driving instruction unit 405 and the control unit 406 execute control based on the received control signal (S914). In addition, the signal monitoring unit 407 holds the count value of the counter extracted from the received signal (S915). The held count value is used for verification in the next processing of S911.

[0129] As described above, in the remote control according to this embodiment, if there is a delay in a control command or instruction command, the edge device 4 issues an emergency stop. Specifically, the edge device 40 checks the counters and error check codes attached to the control signal, driving instruction signal, and / or time signal in the operator terminal 20, and performs an emergency stop if the count value of the same counter or the like is abnormal or the signal is irregular. This configuration ensures safety in the event of an abnormality where the same control signal is repeatedly transmitted during remote operation, or where a control signal continues to be transmitted even after the remote operation has been stopped (vehicle information / control signal abnormality: H6). Furthermore, the edge device 40 transmits the vehicle information with an acquisition time attached, and compares the time information attached to the control signal, driving instruction signal, and / or time signal in the operator terminal 20 with the current time to determine a delay. If the delay is significant, an emergency stop is performed. This configuration ensures safety in the event of an abnormality where signal transmission or video is delayed during remote operation assistance (vehicle information (video) / control signal abnormality: H7, H12) or an abnormality where a control signal, driving instruction signal, and / or emergency stop signal cannot be transmitted (vehicle information / control signal abnormality: H9, H10). Note that instead of or in addition to the time information ("time A"), the delay may be measured using, for example, identification information (e.g., "identifier A") for uniquely identifying vehicle information sent from the vehicle 46. In this case, for example, when transmitting vehicle information, the edge device 40 may associate the identifier with the transmission time of the vehicle information and store them in a table, and when receiving the identifier or information including the identifier, may refer to the table to obtain the transmission time, and measure the delay based on the obtained transmission time and the reception time of the identifier.

[0130] (Tenth embodiment) FIG. 22 is a diagram for explaining state transitions assumed in the remote control system 1 according to the tenth embodiment. In a state transition diagram 630 illustrated in FIG. 22, a "stopped" state 631 can transition between a "remote control" state 632 and a "remote assistance" state 633. The "remote control" state 632 is a state in which a control signal is received remotely. The "remote assistance" state 633 is a state in which a driving start instruction (driving instruction signal) is received remotely. The "remote assistance" state 633 can transition to an "automatic driving" state 634 in response to the driving start instruction. The "automatic driving" state 634 can transition to a "stopped" state 631 or an "emergency stop" state 635. The "emergency stop" state 635 can transition to the "stopped" state 631. The "remote control" state 632 and the "remote assistance" state 633 can transition to the "emergency stop" state 635.

[0131] FIG. 23 is a diagram illustrating mask control in the remote control system 1 according to the tenth embodiment. As shown in FIG. 23, the mask control executed in the remote control according to this embodiment enables transmission of a control signal from the server 30 to the vehicle 46 via the edge device 40 in the "remote operation" state 632. On the other hand, the mask control masks the control signal in the edge device 40 and does not transmit it in the other states 633 to 635 of the "remote operation" state 632. FIG. 24 is a flowchart showing an example of the flow of information processing executed by the edge device 40 according to the tenth embodiment. After receiving a control signal from the server 30 (S1101a), the instruction control management unit 403 in the edge device 40 checks the remote control mode (S1102). If the mode is "remote operation" (state 632) (S1103a: Yes), the instruction control management unit 403 transmits a control signal or a signal corresponding to the control signal to the vehicle 46, thereby controlling the vehicle 46 according to the control signal (S1104a). On the other hand, if the mode is not "remote control" (state 632) (S1103a: No), the flow in FIG. 24 ends.

[0132] 23, the mask control executed in the remote control according to this embodiment enables the server 30 to transmit a driving instruction signal indicating a driving instruction, such as a driving start instruction or a driving stop instruction, and time information to the vehicle 46 via the edge device 40 in the "remote assistance" state 633. On the other hand, the mask control masks the driving instruction signal and the time information in the edge device 40 in the states 631 to 632 and 634 to 635 other than the "remote assistance" state 633, and does not transmit them. FIG. 25 is a flowchart showing an example of the flow of information processing executed by the edge device 40 according to the tenth embodiment. After receiving a driving instruction signal from the server 30 (S1101b), the instruction control management unit 403 in the edge device 40 checks the remote control mode (S1102). If the mode is "remote assistance" (state 633) (S1103b: Yes), the instruction control management unit 403 starts or stops the vehicle 46 from traveling by transmitting a traveling instruction signal or an instruction (signal) corresponding to the traveling instruction signal to the vehicle 46 (S1104b). On the other hand, if the mode is not "remote assistance" (state 633) (S1103b: No), the flow in FIG. 25 ends.

[0133] As described above, in the remote control according to this embodiment, control signals received other than during remote operation and driving instruction requests received other than during remote assistance are ignored by the edge 4. Specifically, the edge device 40 accepts control signals only during remote operation, accepts driving instruction signals with time information added and periodically transmitted time information only during remote assistance, and executes masking control to mask these other signals. This configuration ensures safety in the event of an abnormality in which an unintended control signal is transmitted during remote operation or remote assistance (control abnormality, vehicle information / control signal abnormality: H8), an abnormality in which an unintended mode switching request is transmitted (mode switching abnormality: H20), or an abnormality in which a state (mode) mismatch occurs between the front end 2, back end 3, and edge 4 (mode switching abnormality, vehicle information / control signal abnormality: H21).

[0134] (First modified example of the tenth embodiment) When the mask control is performed, a notification of an abnormality may be given.

[0135] Fig. 26 is a flowchart showing an example of the flow of information processing executed by the edge device 40 according to the first modified example of the tenth embodiment. Here, differences from the flow in Fig. 24 will be described. In the edge device 40, if the mode is not "remote operation" (state 632) (S1103a: No), the instruction control management unit 403 displays a notification on the screen of the operator terminal 20 (S1105). Thereafter, the flow in Fig. 26 ends.

[0136] Fig. 27 is a flowchart showing an example of the flow of information processing executed by the edge device 40 according to the first modified example of the tenth embodiment. Here, differences from the flow in Fig. 25 will be described. In the edge device 40, if the mode is not "remote assistance" (state 633) (S1103b: No), the instruction control management unit 403 displays a notification on the screen of the operator terminal 20 (S1105). Thereafter, the flow in Fig. 27 ends.

[0137] Fig. 28 is a diagram showing an example of a screen display on the operator terminal 20 according to the first modified example of the tenth embodiment. Fig. 28 shows an example of a display screen 640 that is displayed in the process of S1105 in Fig. 26 or 27 when mask control is being performed. As shown in Fig. 28, the display screen 640 displays an abnormality notification 641 such as "Abnormal communication detected" on the screen display of an image 621 for remote control.

[0138] Note that, assuming a use case in which driving instructions are frequently issued during remote control, for example, the abnormality notification 641 may be configured to notify only when an abnormality occurs once, but only when it occurs frequently within a short period of time. Here, the short period of time refers to a predetermined period that is determined in advance and stored in the internal memory of the edge device 40. Alternatively, the abnormality notification 641 may be configured to notify only when the cause of the abnormality is estimated to be an attack by a third party. This attack detection is executed, for example, by the edge device 40, but may also be detected outside the edge device 40, and the notification may be made based on the detection information from that outside. Alternatively, the abnormality notification 641 may not be made when the load on the operator is high, for example, by notifying after the operator has performed remote control.

[0139] In this way, in the remote control according to this modification, an abnormality may be notified when mask control is performed. With this configuration, the operator can easily recognize that an abnormality has occurred or what abnormality has occurred, thereby further improving safety in the event of each abnormality.

[0140] (Second modified example of the tenth embodiment) Fig. 29 is a diagram for explaining state transitions assumed in a remote control system 1 according to a second modified example of the tenth embodiment. Fig. 29 illustrates a state transition diagram 630 assuming a pattern in which autonomous driving starts immediately after remote control. The state transition diagram 630 in Fig. 29 is an example in which the state transition diagram 630 in Fig. 22 can be further transitioned from a "remote control" state 632 to an "autonomous driving" state 634 in response to a driving start instruction.

[0141] FIG. 30 is a diagram illustrating mask control in a remote control system 1 according to a second modification of the tenth embodiment. As shown in FIG. 30, the mask control for the transmission of control signals executed in the remote control according to this embodiment is similar to the mask control for the transmission of control signals described with reference to FIGS. 23 and 24, and therefore will not be described here. Meanwhile, as shown in FIG. 30, the mask control executed in the remote control according to this embodiment enables transmission of driving instruction signals indicating driving instructions, such as a driving start instruction or a driving stop instruction, from the server 30 to the vehicle 46 via the edge device 40 in the "remote operation" state 632 in addition to the "remote assistance" state 633. Meanwhile, the mask control enables transmission of time information only in the "remote assistance" state 633. Furthermore, the mask control masks the driving instruction signals in the edge device 40 and does not transmit them in states 631, 634 to 635 other than the "remote operation" and "remote assistance" states 632 to 633.

[0142] Fig. 31 is a flowchart showing an example of the flow of information processing executed by the edge device 40 according to the second modified example of the tenth embodiment. Here, differences from the flow in Fig. 25 will be described. In the edge device 40, after checking the remote control mode (S1102), if the mode is "remote operation" or "remote assistance" (states 632 to 633) (S1103c: Yes), the instruction control management unit 403 starts or stops the vehicle 46 from traveling by transmitting a travel instruction signal or an instruction (signal) corresponding to the travel instruction signal to the vehicle 46 (S1104b). On the other hand, if the mode is not "remote operation" or "remote assistance" (states 632 to 633) (S1103c: No), the flow in Fig. 31 ends.

[0143] As described above, in the remote control according to this modification, control signals received at times other than during remote operation and driving instruction requests received at times other than during remote operation assistance are ignored by the edge 4. Specifically, the edge device 40 accepts control signals and driving instruction signals during remote operation, and accepts driving instruction signals with added time information and periodically transmitted time information only during remote assistance, and executes masking control to mask these other signals. This configuration ensures safety even in a pattern in which autonomous driving begins immediately after remote operation.

[0144] (Eleventh embodiment) In the remote control according to the above-described embodiment, a remote assistance state is defined to enable accurate remote driving instructions, but when an operator provides remote assistance, for example, during remote monitoring, the operator must switch to the remote assistance state, which makes the operation cumbersome. Therefore, in this embodiment, a remote control system 1 that can automatically transition to the remote assistance state during remote monitoring will be described.

[0145] In the remote control system 1 according to this embodiment, the operator terminal 20 further functions as a quality signal transmitter. This quality signal transmitter transmits a time signal and video quality information to the server 30. Specifically, the quality signal transmitter assigns the transmission time of the latest vehicle information received by the vehicle information receiver 213 to the video quality information, and transmits the video quality information with the transmission time assigned to it to the server 30. The vehicle information monitor 209 of the operator terminal 20 also checks the delay information indicating the round-trip delay received as the vehicle information and the video quality information, and activates the driving instruction button 664a (see FIG. 36 ) when a specified threshold condition for delay and video quality are satisfied for a certain period of time. The threshold condition for the delay information, the threshold for the quality information, and the threshold for the duration thereof are each predetermined and stored, for example, in the internal memory of the operator terminal 20.

[0146] In the remote control system 1 according to this embodiment, the server 30 further functions as a quality signal transmission unit. This quality signal transmission unit transmits quality information from the operator terminal 20 to the edge device 40 that is paired with the server 30.

[0147] Furthermore, in the remote control system 1 according to this embodiment, the edge device 40 further functions as a quality information receiving unit. This quality information receiving unit receives quality information sent from the operator terminal 20 via the server 30. Furthermore, the instruction control management unit 403 of the edge device 40 performs processing to switch whether or not to send a driving instruction signal to the vehicle 46 during remote assistance, based on the received quality information.

[0148] Here, the flow of processing during remote monitoring among the operator terminal 20, server 30, and edge device 40 according to this embodiment will be described. Fig. 32 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system 1 according to the eleventh embodiment. Here, differences from the flow in Fig. 21A will be described.

[0149] In the operator terminal 20 that has received the vehicle information from the server 30 (S903), the vehicle information display unit 208 displays information about the vehicle 46 that is the target of remote operation assistance based on the received vehicle information (S905). Then, the quality signal transmission unit of the operator terminal 20 acquires "time A" assigned to the received vehicle information and assigns it to the quality information of the video, thereby generating video quality information including "time A" (S1201) and transmitting it to the server 30 (S1202). In the server 30 that has received this, the quality signal transmission unit transmits the received signal to the edge device 40 (S1203). In the edge device 40 that has received this, the quality information receiving unit extracts the quality information from the received signal (S1204).

[0150] During remote monitoring, the remote control system 1 repeatedly executes the processes of S901 to S905 and S1201 to S1204. The remote control system 1 further repeatedly executes the following processes of S1205 to 1211. The processes of S1205 to 1211 may be executed each time the processes of S901 to S905 and S1201 to S1204 are executed, or may be executed each time the processes of S901 to S905 and S1201 to S1204 are executed multiple times.

[0151] In the remote monitoring, the command control management unit 403 of the edge device 40 calculates a delay time, which is the difference between the current time and "time A," for the quality signal (S1205). Then, the vehicle information transmission unit 401 transmits delay information indicating the calculated delay time to the server 30 (S1206). Furthermore, the vehicle information transmission unit 301 in the server 30 that received this transmits the received delay information to the operator terminal 20 (S1207). Furthermore, the vehicle information monitoring unit 209 in the operator terminal 20 that received this verifies the round-trip delay indicated by the delay information received as vehicle information and the video quality of the camera video (S1208-S1209) and determines whether the specified delay and video quality are satisfied for a certain period of time. In other words, the vehicle information monitoring unit 209 determines whether the specified conditions for delay and video quality are satisfied for a predetermined duration. As an example, this determination determines whether the specified conditions, that the delay is less than a specified value and the video quality is equal to or greater than a specified value, are satisfied for a certain period of time. In this case, not meeting the specified conditions means that "the delay is greater than the specified value," "the video quality is less than the specified value," and / or "the delay is less than the specified value and the video quality is greater than the specified value but does not meet a certain time."

[0152] If the specified conditions are met, the vehicle information monitoring unit 209 of the operator terminal 20 enables the driving instruction button 664a (see FIG. 36) (S1210). On the other hand, if the specified conditions are not met, the vehicle information monitoring unit 209 of the operator terminal 20 disables the driving instruction button 664a (S1201).

[0153] As described above, in the remote control according to this embodiment, the validity / invalidity of the driving instruction button 664a is switched depending on whether the state in which the round-trip delay and video quality satisfy the specified values ​​is maintained for a certain period of time. Here, information processing by the operator terminal 20 according to this embodiment will be described in more detail. FIG. 33 is a flowchart showing an example of the flow of information processing executed by the operator terminal 20 according to the 11th embodiment. FIG. 34 is a diagram showing an example of the data configuration of delay information 651 according to the 11th embodiment. FIG. 35 is a diagram showing an example of the data configuration of quality information 652 according to the 11th embodiment.

[0154] First, the vehicle information monitoring unit 209 verifies the round-trip delay indicated by the delay information received as the vehicle information and the video quality of the camera video (S1208 to S1209). Specifically, the vehicle information monitoring unit 209 acquires the round-trip delay indicated by the delay information 651 received as the vehicle information (S1301). For example, as shown in FIG. 34, the delay information 651 includes a "vehicle ID" field for uniquely identifying the vehicle 46 being remotely monitored and a "round-trip delay time" field. The vehicle information monitoring unit 209 also acquires quality information 652 from the camera video received as the vehicle information (S1302). For example, as shown in FIG. 35, the quality information 652 includes a "vehicle ID" field for uniquely identifying the vehicle 46 being remotely monitored and at least one field related to video quality. The quality information 652 in FIG. 35 includes at least one field related to video quality: a "video resolution" field, a "frame rate" field, and a "bit rate" field.

[0155] Then, the vehicle information monitoring unit 209 determines whether the specified conditions are satisfied (S1303). If the specified conditions are satisfied (S1303: Yes), the vehicle information monitoring unit 209 enables the driving instruction button 664a (see FIG. 36) (S1210), and if the specified conditions are not satisfied (S1303: No), the vehicle information monitoring unit 209 disables the driving instruction button 664a (S1201).

[0156] FIG. 36 is a diagram illustrating an example of a screen display on the operator terminal 20 according to the eleventh embodiment. A display screen 660a in FIG. 36 illustrates an example of a screen display when specified conditions are met during remote monitoring. The display screen 660a includes a delay information display 662 and a quality information display 663, along with an image 661 for remote monitoring. The display screen 660a also includes a travel instruction button 664a for executing "remote assistance." The travel instruction button 664a is enabled on the display screen 660a in FIG. 36. On the other hand, a display screen 665a in FIG. 36 illustrates an example of a screen display when specified conditions are not met during remote monitoring. The display screen 665a illustrates a state in which the travel instruction button 664a is disabled on the display screen 660a.

[0157] As described above, in the remote control according to this embodiment, when the delay and video quality satisfy specified conditions for a certain period of time, the remote control automatically transitions to a state equivalent to remote assistance. Specifically, in the remote control according to this embodiment, after the operator selects a vehicle, if the operator is able to confirm video with low delay and sufficient quality for a certain period of time, driving instructions are enabled. More specifically, when the delay and video quality satisfy specified conditions for a certain period of time, the remote control automatically transitions to a state equivalent to remote assistance, in which the driving instruction button 664a is enabled on the front end 2. In other words, when the delay and video quality satisfy specified conditions for a certain period of time, remote assistance is permitted.

[0158] This configuration ensures safety in the event of an abnormality where an unintended control signal is sent during remote operation or remote assistance (control abnormality, vehicle information / control signal abnormality: H8), an abnormality where an unintended mode switching request is sent (mode switching abnormality: H20), or an abnormality where a state (mode) discrepancy occurs between the front end 2, back end 3, and edge 4 (mode switching abnormality, vehicle information / control signal abnormality: H21).In addition, there is no need to switch to remote assistance, and the operator only needs to issue a command to start driving, simplifying the operator's operations.

[0159] (First modified example of the eleventh embodiment) The remote control according to the eleventh embodiment described above is not limited to an automatic transition to a state equivalent to remote assistance, but may be configured to automatically transition to a state equivalent to remote operation.

[0160] FIG. 37 is a diagram illustrating an example of a screen display on the operator terminal 20 according to a first modified example of the eleventh embodiment. A display screen 660b in FIG. 37 illustrates an example of a screen display when a specified condition is satisfied during remote monitoring. The display screen 660b includes a delay information display 662 and a quality information display 663, as well as an image 661 for remote monitoring. The display screen 660b also includes a remote operation button 664b for executing "remote operation." In the display screen 660b in FIG. 37, the remote operation button 664b is enabled. On the other hand, a display screen 665b in FIG. 37 illustrates an example of a screen display when a specified condition is not satisfied during remote monitoring. In the display screen 665b, the remote operation button 664b is disabled on the display screen 660b.

[0161] In this way, in the remote control according to this modification, when the predetermined conditions for the time, delay, and video quality are satisfied, the remote control automatically transitions to a state equivalent to remote operation. Even with this configuration, the same effects as those of the above-described embodiment can be obtained.

[0162] (Second modified example of the eleventh embodiment) FIG. 38 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system 1 according to the second modified example of the eleventh embodiment. Here, differences from the flow shown in FIG. 32 will be described. In remote monitoring, the instruction control management unit 403 of the edge device 40 calculates the delay time, which is the difference between the current time and "time A," for a control signal or a driving instruction signal (S1205), and then verifies the round-trip delay calculated in the processing of S1205 (S1212). The instruction control management unit 403 also verifies quality information received from the operator terminal 20 via the server 30 (S1213). Then, the instruction control management unit 403 determines whether specified conditions, such as a specified delay and video quality being satisfied for a certain period of time, are satisfied.

[0163] If the specified conditions are met, the instruction control management unit 403 of the edge device 40 permits reception of the driving instruction signal (S1214). On the other hand, if the specified conditions are not met, the instruction control management unit 403 of the edge device 40 does not permit reception of the driving instruction signal (S1215).

[0164] Although FIG. 38 illustrates an example of a flow when switching between validity and invalidity of the driving instruction button 664a, it may be configured as a flow when switching between validity and invalidity of the remote control button 664b in a similar manner.

[0165] In this way, the instruction control management unit 403 in the edge device 40 switches whether or not to transmit a driving instruction signal to the vehicle 46 during remote monitoring based on the round-trip delay and video quality. Here, information processing by the edge device 40 according to this embodiment will be described in more detail. Figure 39 is a flowchart showing an example of the flow of information processing executed by the edge device 40 according to the second modified example of the eleventh embodiment.

[0166] First, the instruction control management unit 403 acquires the quality signal by calculating the delay time (S1401), and acquires the quality information received from the operator terminal 20 (S1402). Then, the instruction control management unit 403 determines whether a prescribed condition is satisfied (S1403). If the prescribed condition is satisfied (S1403: Yes), the vehicle information monitoring unit 209 permits reception of the driving instruction signal (S1214), and if the prescribed condition is not satisfied (S1403: No), it does not permit reception of the driving instruction signal (S1215).

[0167] In this way, the remote control according to this modification enables driving instructions to the edge 4 when the delay time and video quality satisfy specified values ​​for a certain period of time. This configuration further improves safety in the event of an abnormality in which an unintended control signal is sent during remote operation or remote assistance (control abnormality, vehicle information / control signal abnormality: H8), an abnormality in which an unintended mode switching request is sent (mode switching abnormality: H20), or an abnormality in which a state (mode) mismatch occurs between the front end 2, back end 3, and edge 4 (mode switching abnormality, vehicle information / control signal abnormality: H21).

[0168] (Twelfth embodiment) FIG. 40 is a flowchart showing an example of the flow of information processing executed by the edge device 40 according to the twelfth embodiment.

[0169] In the edge device 40, the control unit 406 receives a control signal transmitted from the operator terminal 20 via the server 30 (S1501). The control unit 406 also acquires, for example, a predetermined specified maximum amount of operation that is stored in an internal memory (S1502). As an example, the specified maximum amount of operation includes a specified value for "maximum vehicle speed" (e.g., 15 km / h). As an example, the specified maximum amount of operation includes a specified value for "maximum acceleration" (e.g., 0.3 G). The control unit 406 then determines whether the control value indicated by the received control signal exceeds the specified value for the specified maximum amount of operation (S1503).

[0170] If the control value indicated by the received control signal exceeds the specified value of the specified maximum operation amount (S1503: Yes), the control unit 406 corrects the control value indicated by the received control signal to the specified value of the specified maximum operation amount (S1504) and then executes control on the vehicle 46 (S1505). On the other hand, if the control value indicated by the received control signal does not exceed the specified value of the specified maximum operation amount (S1503: No), the control unit 406 executes control on the vehicle 46 with the control value indicated by the received control signal (S1505). Note that if the control value indicated by the received control signal exceeds the specified value of the specified maximum operation amount, an emergency stop may be performed instead of correcting the specified value.

[0171] In this way, in the remote control according to this embodiment, if the control value of the received control signal is greater than the upper limit, the control value is rounded down to the specified upper limit. This configuration ensures safety in the event of an abnormality where an unintended control signal is sent during remote operation or remote assistance (control abnormality, vehicle information / control signal abnormality: H8).

[0172] (Thirteenth embodiment) Fig. 41 is a flowchart showing an example of the flow of information processing executed by the edge device 40 according to the thirteenth embodiment. The flow in Fig. 41 is executed, for example, after the flow in Fig. 40 according to the twelfth embodiment.

[0173] In the edge device 40, the behavior monitoring unit 404 receives a control value (instruction content) for control of the vehicle 46 from the control unit 406 (S1601). The behavior monitoring unit 404 also receives a result value (execution result) of the control result for the vehicle 46 (S1602). The behavior monitoring unit 404 then determines whether the difference between the control value (instruction value) and the result value (actual measurement value) is equal to or greater than a predetermined threshold value stored in the internal memory of the edge device 40 (S1603). Here, the control value is, for example, a vehicle speed, and is an instructed vehicle speed instructed to the vehicle 46, and the result value is the actual vehicle speed of the vehicle 46 after the control is executed. For example, if the instructed vehicle speed is 15 km / h and the actual vehicle speed is 30 km / h, and there is a discrepancy between the control value and the result value, it is determined that an abnormality exists.

[0174] If the control result is not returned from the vehicle 46 after the control signal is transmitted, it is determined to be abnormal, just as when the difference between the control value and the result value is equal to or greater than the threshold value.

[0175] It is also possible to determine that an abnormality has occurred when the difference between the control value and the result value is equal to or greater than a threshold value for multiple times.

[0176] If the difference between the control value and the result value is equal to or greater than the threshold value (S1603: Yes), the behavior monitoring unit 404 executes an emergency stop of the vehicle 46 (S1604). After the emergency stop is executed, or if the difference between the control value and the result value is less than the threshold value (S1603: No), the flow in FIG. 41 ends.

[0177] The emergency stop signal for an emergency stop may be transmitted via a system separate from the system for transmitting the control signal. This configuration provides redundancy, further improving safety.

[0178] As described above, in the remote control according to this embodiment, the control signal sent to the vehicle 46 is compared with the control result returned from the vehicle 46, and an emergency stop is performed if they differ or if no control result is returned from the vehicle 46. This configuration ensures safety in the event of an abnormality (control abnormality, vehicle information / control signal abnormality: H8) in which an unintended control signal is sent during remote operation or remote assistance.

[0179] (Fourteenth embodiment) The following describes the flow of processing during remote operation assistance among the operator terminal 20, the server 30, and the edge device 40 according to this embodiment. Fig. 42 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system 1 according to the fourteenth embodiment.

[0180] First, the vehicle information input unit 402 in the edge device 40 acquires an image (camera video) from the camera 45 (S1701) and assigns count values ​​to frames of the acquired camera video (S1702). The vehicle information transmission unit 401 then transmits the camera video with the count values ​​assigned to the frames to the server 30 (S1703). At this time, the vehicle information input unit 402 increments the counter value by a specified amount. Note that the specified amount, which is the increment (or decrement) of the counter value per count, is assumed to be predetermined and stored, for example, in the internal memory of the edge device 40. The vehicle information transmission unit 301 in the server 30 that receives this transmits the received camera video to the operator terminal 20 (S1704).

[0181] In the remote operation assistance state, the vehicle information monitoring unit 209 in the operator terminal 20 compares the count value of the received video with the count value of the previously received video (S1705). If the count value has not been incremented from the count value of the previously received video, the emergency stop signal transmitting unit 205 transmits an emergency stop signal to the server 30 (S1706). In the server 30 that has received this signal, the emergency stop signal transmitting unit 306 transmits the emergency stop signal to the edge device 40 (S1707). In the edge device 40 that has received this signal, the emergency stop unit 408 performs an emergency stop of the vehicle 56 (S1708). Note that if the count value has not been incremented from the count value of the previously received video, the emergency stop is not necessarily performed, and the received video information may simply be discarded as invalid video.

[0182] On the other hand, if the count value has been incremented from the count value of the previously received video, the vehicle information monitoring unit 209 holds the count value of the video (S1709).

[0183] In this way, in the remote control according to this embodiment, if the video information is invalid data during remote operation assistance, an emergency stop is performed. Specifically, a counter is added to the video frame, and if a frame with the same counter is received, an emergency stop is performed. This configuration ensures safety in the event of an abnormality where the video freezes (video abnormality: H13).

[0184] (Fifteenth embodiment) 43 is a flowchart showing an example of the flow of information processing executed by the operator terminal 20 according to the fifteenth embodiment. Note that a similar determination process may be executed not only in the operator terminal 20 but also in the edge device 40.

[0185] The vehicle information receiving unit 213 receives camera images from the camera 45 (S1801). The vehicle information monitoring unit 209 receives mode information from the mode switching management unit 214 (S1802) and determines whether remote operation assistance is in progress (S1803).

[0186] If remote operation assistance is in progress (S1803: Yes), vehicle information monitoring unit 209 acquires, for example, required video quality information 670 (see FIG. 44) that is predetermined and stored in an internal memory (S1804), and determines the required quality of the received video based on required video quality information 670 (S1805). If the received video does not meet the required quality (S1805: Yes), vehicle information monitoring unit 209 and emergency stop signal transmitting unit 205 execute an emergency stop (S1806). After the emergency stop is executed, or if the received video meets the required quality (S1805: No), the flow in FIG. 43 ends.

[0187] If remote operation assistance is not in progress (S1803: No), the vehicle information monitoring unit 209 acquires required video quality information 670 (S1807) in the same manner as the processes of S1807 to S1805, and determines the required quality of the received video based on the required video quality information 670 (S1808). If the received video does not meet the required quality (S1808: Yes), the vehicle information monitoring unit 209 disables and inactivates the remote operation assistance switching buttons, such as the driving instruction button 664a and the remote operation button 664b (see FIG. 36) (S1809). On the other hand, if the received video meets the required quality (S1808: No), the vehicle information monitoring unit 209 enables and activates the remote operation assistance switching button (S1810). Thereafter, the flow of FIG. 43 ends.

[0188] FIG. 44 is a diagram illustrating an example of the data configuration of required video quality information 670 according to the fifteenth embodiment. As illustrated in FIG. 44, the required video quality information 670 stores information indicating the video quality required during remote assistance and during remote operation, for each of at least one item related to video quality. The required video quality information 670 in FIG. 44 includes, as at least one item related to video quality, an item for "required resolution," an item for "required bit rate," and an item for "required frame rate." As an example, the "required resolution" during remote assistance is higher than the "required resolution" during remote operation. As an example, the "required bit rate" during remote assistance is higher than the "required bit rate" during remote operation. As an example, the "required frame rate" during remote assistance is lower than the "required frame rate" during remote operation. Note that the video quality required during remote assistance and during remote operation illustrated in FIG. 44 is merely an example, and the required video quality during remote operation may be higher than the required video quality during remote assistance.

[0189] In this way, in the remote control according to this embodiment, an emergency stop is performed if the image is difficult to see during remote operation assistance. Specifically, an emergency stop is performed if an image of sufficient resolution specified by the required image quality information 670 cannot be received. This configuration ensures safety in the event of an abnormality (image abnormality: H14) in which the image quality (picture quality) deteriorates and the operator is unable to correctly recognize the surroundings from the image.

[0190] (16th embodiment) Fig. 45 is a diagram showing an example of the configuration of the controller 25 according to the sixteenth embodiment. As shown in Fig. 45, the controller 25 has a remote control release button 255 and a remote assistance release button 256 provided on the hub of a steering wheel 251 held by the operator, for example. The remote control release button 255 is an operation button for issuing an instruction to stop remote control. The remote assistance release button 256 is an operation button for issuing an instruction to stop remote assistance. Note that at least one of the remote control release button 255 and the remote assistance release button 256 may be realized by providing an operation button for receiving an operator operation, such as a click or a touch, on a screen displayed on the operator terminal 20.

[0191] 46 is a flowchart showing an example of the flow of information processing executed by the server 30 according to the sixteenth embodiment. When the remote operation cancel button 255 of the controller 25 is pressed, the input receiving unit 210 of the operator terminal 20 acquires an operation signal output from the controller 25 in response to the operator's operation. Furthermore, the control signal transmitting unit 204 transmits a control signal in response to the pressing of the remote operation cancel button 255 to the server 30 (S1901). In the server 30, which has received the control signal in response to the pressing of the remote operation cancel button 255, the control signal transmitting unit 305 stops transmitting the control signal to the edge device 40 (S1902). Thereafter, the flow in FIG. 46 ends.

[0192] 47 is a flowchart showing an example of the flow of information processing executed by the server 30 according to the sixteenth embodiment. When the remote assistance release button 256 of the controller 25 is pressed, the input receiving unit 210 of the operator terminal 20 acquires an operation signal output from the controller 25 in response to the operator's operation. In addition, the control signal transmitting unit 204 transmits a control signal in response to the pressing of the remote assistance release button 256 to the server 30 (S2001). In the server 30 that has received the control signal in response to the pressing of the remote assistance release button 256, the control signal transmitting unit 305 stops transmitting time information to the edge device 40 (S2002). In addition, the driving instruction signal transmitting unit 304 disables transmission of driving instruction signals to the edge device 40 (S2003). Thereafter, the flow of FIG. 47 ends.

[0193] In this way, in the remote control according to this embodiment, an operation button is provided on the controller 25 for the operator to stop the remote operation assistance when the image is poor, and the operator performs an operation to stop the remote operation assistance if the image is difficult to see during the remote operation assistance. This configuration ensures safety in the event of an abnormality (image abnormality: H15) in which the operator cannot correctly recognize the surroundings from the image due to the setting sun, backlighting, camera abnormality, etc.

[0194] (17th embodiment) FIG. 48 is a flowchart showing an example of the flow of information processing executed by the operator terminal 20 according to the seventeenth embodiment.

[0195] First, the operator uses the controller 25 connected to the operator terminal 20 to perform a selection operation to select a vehicle 46 that is the target of remote operation assistance. This selection operation may be performed by the operator's operation, such as clicking or touching an operation button, image, icon, or the like displayed on the screen of the operator terminal 20. The controller 25 outputs an operation signal corresponding to the operator's selection operation to the operator terminal 20. The input acceptance unit 210 in the operator terminal 20 then acquires the operation signal output from the controller 25 (S2101). The vehicle information receiving unit 213 then requests camera video for remote control of the selected vehicle 46 via the server 30 (S2102), and determines whether acquisition of the video has started in response to the video request (S2103).

[0196] When image acquisition has started in response to the image request (S2103: Yes), the mode switching management unit 214 acquires image transmission source information, such as camera information of the camera 45 and vehicle information of the vehicle 46 (S2104), and determines whether the image transmission source matches the target of remote operation assistance (S2105). As an example, when the image transmission source is the camera 45 that acquires camera image for remote control of the vehicle 46 that is the target of remote operation assistance, it is determined that the image transmission source matches the target of remote operation assistance. When the image transmission source matches the target of remote operation assistance (S2105: Yes), the mode switching management unit 214 enables and activates remote operation assistance switching buttons, such as the driving instruction button 664a and the remote operation button 664b (see FIG. 36) (S2106). After the switch button is activated, or if image acquisition has not started in response to the image request (S2103: No), or if the image sender and the target of remote operation assistance do not match (S2105: No), the flow in Figure 48 ends.

[0197] As described above, in the remote control according to this embodiment, remote operation assistance is permitted only when the operator terminal 20 can acquire the target video information. Specifically, a request is made to acquire a video stream related to the vehicle 46 with the same ID as the vehicle ID of the target for remote operation assistance, and if this does not exist or cannot be received, remote operation assistance is prohibited. This configuration ensures safety in the event of an abnormality where the vehicle video displayed on the front end 2 and the destination of the control signal are different (video linking: H16).

[0198] (18th embodiment) FIG. 49 is a flowchart showing an example of the flow of information processing executed by the operator terminal 20 according to the eighteenth embodiment.

[0199] When the input receiving unit 210 acquires an operation signal corresponding to an operator's operation instructing the startup of application software for remote operation assistance (S2201), the operator terminal application monitoring unit 207 checks the process being executed on the operator terminal 20 (S2202). The operator terminal application monitoring unit 207 also determines whether the application has already been started (S2203). If the application has already been started (S2203: Yes), the operator terminal application monitoring unit 207 stops the application instructed to be started this time (S2204), and if the application has not been started (S2203: No), the operator terminal application monitoring unit 207 starts the application instructed to be started this time (S2205).

[0200] In this way, in the remote control according to this embodiment, the applications for remote operation assistance that can be started on one operator terminal 20 are limited to the one application that was started first for remote operation assistance. This configuration ensures safety in the event of an abnormality (terminal restriction: H17) in which multiple remote operation applications are started and remote operation or remote assistance is performed on multiple vehicles 46 simultaneously.

[0201] (19th embodiment) FIG. 50 is a flowchart showing an example of the flow of information processing executed by the server 30 according to the nineteenth embodiment.

[0202] The operator terminal status monitoring unit 307 detects the start of signal transmission during remote operation assistance from at least one operator terminal 20 (S2301), and checks the edge device 40 to which the signal is to be sent from each operator terminal 20 (S2302). The operator terminal status monitoring unit 307 determines whether a signal from another operator terminal 20 has already been sent to the confirmed destination edge device 40 (S2303).

[0203] If a signal is already being transmitted from another operator terminal 20 (S2303: Yes), the operator terminal status monitoring unit 307 does not permit and rejects the signal transmission from the operator terminal 20 that has started transmission this time (S2304). On the other hand, if a signal is not being transmitted from another operator terminal 20 (S2303: No), the operator terminal status monitoring unit 307 permits the signal transmission from the operator terminal 20 that has started transmission this time (S2305). Thereafter, the flow in FIG. 50 ends.

[0204] In this way, in the remote control according to this embodiment, the operator terminals 20 that can provide remote operation assistance to one vehicle 46 are limited to the one operator terminal 20 that first started remote operation assistance. This configuration ensures safety in the event of an abnormality (terminal restriction: H18) in which the same vehicle 46 is remotely operated or remotely assisted by multiple terminals.

[0205] (Twentieth embodiment) FIG. 51 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system 1 according to the twentieth embodiment.

[0206] First, the input receiving unit 210 acquires an operation signal from the controller 25 in response to an operation by the operator instructing a switch to the remote operation assistance mode (S2401). The mode switching signal transmitting unit 202 transmits a remote operation assistance switching signal in response to the acquired operation signal to the server 30 (S2402). In the server 30 that has received this, the mode switching signal transmitting unit 303 transmits the received switching signal to the edge device 40 (S2403). In the edge device 40 that has received this, the instruction control managing unit 403 starts a standby period in which it waits for a remote operation assistance switching confirmation signal to be transmitted (S2404). It is assumed that the length of this standby period is, for example, predetermined and stored in the internal memory of the edge device 40.

[0207] If the instruction control management unit 403 times out without receiving a switching confirmation signal for remote operation assistance transmitted from the operator terminal 20 within a predetermined waiting period, the instruction control management unit 403 transmits a switching failure notification together with the reason for the timeout, etc. to the server 30 (S2405). Upon receiving this, the mode switching signal transmission unit 303 in the server 30 transmits the received switching failure notification to the operator terminal 20 (S2406). The operator terminal 20 also receives this notification.

[0208] In the operator terminal 20 that transmitted the switching signal for remote operation assistance, the mode switching signal transmitting unit 202 transmits a switching confirmation signal for remote operation assistance to the server 30 (S2407). In the server 30 that received this, the mode switching signal transmitting unit 303 transmits the received switching confirmation signal to the edge device 40 (S2408). In the edge device 40 that received this, the instruction control managing unit 403 compares the switching signal previously received in the processing of S2404 with the switching confirmation signal received this time (S2409).

[0209] If the matching between the switching signal and the switching confirmation signal fails, the instruction control management unit 403 transmits a switching failure notification together with the reason to the server 30 (S2410). Upon receiving this, the mode switching signal transmission unit 303 in the server 30 transmits the received switching failure notification to the operator terminal 20 (S2411). The operator terminal 20 also receives this notification.

[0210] On the other hand, if the matching between the switching signal and the switching confirmation signal is successful, the command control management unit 403 checks whether the speed of the target vehicle 46 is 0 km / h (S2412).

[0211] If the vehicle speed is greater than 0 km / h, the instruction control management unit 403 transmits a switching failure notification together with the reason to the server 30 (S2413). Upon receiving this, the mode switching signal transmission unit 303 in the server 30 transmits the received switching failure notification to the operator terminal 20 (S2414). The operator terminal 20 also receives this notification.

[0212] On the other hand, if the vehicle speed is 0 km / h, the command control management unit 403 switches the remote control mode for the target vehicle 46 to a mode corresponding to the operator operation acquired in S2401 (S2415).

[0213] When the operator terminal 20 receives a switching failure notification, the operator terminal 20 may display a notification on the screen to the effect that the switching has failed. At this time, the reason for receiving the notification may also be displayed on the screen.

[0214] Since the order of arrival of signals from the operator terminal 20 is not guaranteed, the instruction control management unit 403 checks the signal against the last received switching confirmation signal, and if successful, the check is deemed successful.

[0215] As described above, in the remote control according to this embodiment, a mode-switching command is issued multiple times via two or more different communication systems, and the mode is switched when all of the mode-switching commands are received. Here, issuing multiple mode-switching commands via different systems may mean transmitting the mode-switching command via different communication paths, e.g., with different hardware configurations, or transmitting the mode-switching command multiple times via a communication path with at least a portion of the command shared by both systems. Furthermore, mode switching is only accepted when the vehicle speed is 0 km / h. In other words, when the vehicle 46 is moving (e.g., when the speed is greater than 0 km / h), mode switching requests other than emergency stops are not accepted, i.e., remote operation assistance is not permitted. This configuration ensures safety in the event of an abnormality (mode switching abnormality: H20) in which an unintended mode-switching request is transmitted. While the present embodiment has been described with reference to the transmission of a mode-switching command, this is not limiting. In addition to or instead of the mode-switching command, the present invention may also be applied to the transmission of other signals for remote operation assistance, such as driving instruction signals and control signals.

[0216] (Modification of the 20th embodiment) FIG. 52 is a sequence diagram showing an example of the flow of information processing executed by each unit of the remote control system 1 according to a modification of the twentieth embodiment. Here, differences from the flow shown in FIG. 51 will be described. First, the input receiving unit 210 acquires an operation signal from the controller 25 in response to an operator's operation to instruct switching to the remote operation assistance mode (S2401). Then, the input receiving unit 210 checks whether the accelerator is being depressed, i.e., whether the operator is operating to accelerate, based on the operation signal from the controller 25 in response to the operator's accelerator operation (S2501). Then, when the accelerator is not being depressed, the processing of S2402 to S2415 is executed. Note that the processing of S2402 to S2415 may be configured to be executed only when the brake is being depressed without the accelerator being depressed.

[0217] In this way, in the remote control according to this modification, mode switching except for emergency stop is not accepted when the accelerator is being operated. This configuration ensures safety in the event of an abnormality in which an unintended mode switching request is sent (mode switching abnormality: H20).

[0218] (21st embodiment) FIG. 53 is a diagram showing an example of the configuration of the controller 25 according to the twenty-first embodiment. As shown in FIG. 53, the controller 25 has an unlock button 258 and a driving start button 259 provided on the hub of a steering wheel 251 held by the operator, for example. The unlock button 258 is an operation button for enabling the execution of remote assistance by pressing the driving start button 259. The driving start button 259 is an operation button for instructing the execution of remote assistance. Note that instead of or in addition to the driving start button 259, an operation button for instructing the execution of remote operation may be provided. Note that at least one of the unlock button 258 and the driving start button 259 may be realized by providing an operation button for obtaining an operator operation, such as a click or a touch, on a screen displayed on the operator terminal 20.

[0219] 54 is a flowchart showing an example of the flow of information processing executed by the operator terminal 20 according to the twenty-first embodiment. When the travel start button 259 of the controller 25 is pressed, the input receiving unit 210 of the operator terminal 20 acquires an operation signal output from the controller 25 in response to the operator's operation (S2601), and determines whether the travel start button 259 and the unlock button 258 are pressed together (S2602). If the travel start button 259 and the unlock button 258 are pressed together (S2602: Yes), the travel instruction signal transmitting unit 203 transmits a control signal corresponding to the pressing of the travel start button 259 to the server 30 (S2603). After the processing of S2603, or if the unlock button 258 is not pressed (S2602: No), the flow of FIG. 54 ends.

[0220] As described above, in the remote control according to this embodiment, the controller 25 is provided with an unlock button 258 for activating operation buttons that issue instructions for remote operation assistance when pressed. A driving instruction command is issued by performing double or more operations on these operation buttons, i.e., by operating a predetermined number of operation buttons out of two or more operation buttons. This operation may be realized by operating three or more operation buttons, such as by providing two or more unlock buttons 258 on the controller 25 or by assigning another operation button as the unlock button 258. This configuration ensures safety in the event of an abnormality in which an unintended mode switching request is sent (mode switching abnormality: H20).

[0221] (Twenty-second embodiment) The controller 25 according to the 22nd embodiment has a horn button (not shown) and a fixed phrase speech button (not shown) provided on the hub of a steering wheel 251 held by an operator, for example. The horn button is an operation button for instructing the operation (sounding) of a horn included in an audio output device mounted on the vehicle 46. The fixed phrase speech button is an operation button for instructing the audio output (speech) of a fixed phrase from a speaker included in an audio output device mounted on the vehicle 46.

[0222] Here, the standard phrase may be a word or sentence such as "please go ahead" or "I'm passing" that is spoken from a vehicle 46 being remotely controlled or remotely assisted to communicate with people in the vicinity.

[0223] The fixed phrase utterance button may be an operation button provided for each fixed phrase, or may be an operation button that, when pressed, triggers the display of a fixed phrase selection screen. At least one of the horn button and the fixed phrase utterance button may be realized by providing an operation button, such as a click or a touch, on the screen displayed on the operator terminal 20.

[0224] 55 and 56 are flowcharts showing an example of the flow of information processing executed by the operator terminal 20 according to the 22nd embodiment. Note that these flows may be executed in the edge device 40 that receives an operation signal from the operator terminal 20 via the server 30, for example.

[0225] FIG. 55 illustrates a flow when the horn button is pressed. When the horn button on the controller 25 is pressed, the input receiving unit 210 of the operator terminal 20 acquires an operation signal output from the controller 25 in response to the operator's operation (S2701). The control signal transmitting unit 204 generates a control signal for performing control in response to the pressing of the horn button, i.e., for sounding the horn of the vehicle 46, and transmits the generated control signal to the server 30. The control signal transmitting unit 305 in the server 30 that receives this control signal transmits the received control signal to the edge device 40. The control unit 406 in the edge device 40 that receives the control signal via the server 30 operates the horn of the vehicle 46 based on the control signal, and sounds the horn (S2702). The flow of FIG. 55 then ends.

[0226] FIG. 56 illustrates a flow when the fixed phrase utterance button is pressed. When the fixed phrase utterance button on the controller 25 is pressed, the input receiving unit 210 of the operator terminal 20 acquires an operation signal output from the controller 25 in response to the operator's operation (S2801). The control signal transmitting unit 204 generates a control signal for performing control in response to the pressing of the fixed phrase utterance button, i.e., for activating the speaker of the vehicle 46 to utter a fixed phrase, and transmits the generated control signal to the server 30. The control signal transmitting unit 305 in the server 30 that receives this control signal transmits the received control signal to the edge device 40. The control unit 406 in the edge device 40 that receives the control signal via the server 30 operates the speaker of the vehicle 46 based on the control signal, causing the speaker to utter the fixed phrase (S2802). The flow of FIG. 56 then ends.

[0227] In this way, in the remote control system 1 according to this embodiment, audio can be output in response to both the horn button and the fixed phrase speech button, i.e., audio output is multiplexed by sounding the horn and speaking from the speaker. This multiplexing may be achieved by three or more audio output means. This configuration ensures safety in the event of an abnormality (notification multiplexing: H19) in which the device that issues a warning to people around the vehicle fails, such as when the vehicle is operated without a means of communicating with people around the vehicle during remote operation or remote assistance.

[0228] The above-described embodiments and modifications can be combined in any manner.

[0229] Note that some or all of the functions of each device (controller 25, server 30, edge device 40, and vehicle 46) of the remote control system 1 according to the present disclosure described above may be realized by other devices of the remote control system 1.

[0230] In the above-described embodiment, the determination of "whether it is A or not" may be realized by determining only that it is A, or by determining only that it is not A, or by determining both of these.

[0231] In the above embodiment, "any of A" means "at least one of A."

[0232] In addition, the programs executed by each device of the remote control system 1 according to the above-described embodiment may be provided by being recorded in an installable or executable file format on a computer-readable recording medium (Computer Program Product) such as a CD-ROM, FD, CD-R, or DVD.

[0233] The programs executed by the devices in the remote control system 1 according to the above-described embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The programs executed by the devices in the remote control system 1 according to the above-described embodiment may be provided or distributed via a network such as the Internet.

[0234] Furthermore, the programs executed by the devices of the remote control system 1 according to the above-described embodiment may be provided by being pre-installed in a ROM or the like.

[0235] According to at least one of the embodiments described above, safety can be ensured even if a malfunction occurs due to a hardware problem.

[0236] Although the embodiments of the present disclosure have been described above, the above-described embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and modifications thereof are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0237] (Addendum) The above description of the embodiments discloses the following techniques. (1) An information processing method executed by an information processing system that supports remote control by at least one operator of a plurality of mobile objects, each configured to move autonomously and perform a predetermined task, and includes: at least one front end that acquires operations of the at least one operator; a plurality of edges including the plurality of mobile objects; and a back end that is communicatively connected to the at least one front end and each of the plurality of edges, In response to the operator's operation acquired by the front end, a first transmission mode in which an image of a target moving object among the plurality of moving objects is transmitted from the corresponding edge to the front end to remotely monitor the target moving object; a second transmission mode in which a single driving instruction signal is transmitted to the target moving body to remotely instruct the target moving body to be controlled; a third transmission mode for remotely controlling the target moving object by transmitting a continuous control signal to the target moving object; In the second transmission mode and the third transmission mode, The front end periodically transmits a notification signal different from the driving instruction signal and the control signal to the back end; The back end monitors the notification signal from the front end, and stops the target moving body when a delay occurs in the period of the notification signal. Information processing methods. (2) and stopping the target moving object by the front end in the second transmission mode and the third transmission mode when an image of the target moving object from the edge does not meet a predetermined quality. The information processing method according to (1) above. (3) The front end executes a screen display including an image of the target moving object from the edge, and transmits quality information indicating the quality of the image at the front end to the edge; When the quality information from the front end satisfies a predetermined specified quality, an operation button for instructing a transition to the second transmission mode is enabled on the screen display by the edge. The information processing method according to (1) above. (4) and allowing execution of the second transmission mode by the edge when the quality information from the front end satisfies the specified quality for a predetermined duration. The information processing method described in (3) above. (5) the front end includes a controller operated by the operator; the controller includes two or more operation buttons for instructing the target moving object to stop; the front end stops the target moving object when any of the two or more operation buttons is pressed; The information processing method according to (1) above. (6) the front end includes a controller operated by the operator; the controller includes two or more operation buttons for instructing execution of a specific transmission mode, which is the second transmission mode or the third transmission mode; transmitting a signal requesting execution of the specific transmission mode to the edge when a predetermined number of the two or more operation buttons are pressed by the front end; The information processing method according to (1) above. [Explanation of symbols]

[0238] 1. Remote control system 2. Frontend 20 Operator terminal 201 Screen transition management department 202 Mode switching signal transmitter 203 Driving instruction signal transmitter 204 Control signal transmitter 205 Emergency stop signal transmitter 206 Operator terminal status transmission unit 207 Operator terminal application monitoring unit 208 Vehicle information display unit 209 Vehicle Information Monitoring Department 210 Input reception unit 211 Input device monitoring unit 212 Emergency stop input section 213 Vehicle information receiving unit 214 Mode Switching Management Unit 25 Controller 251 Steering Wheel 252a First emergency stop button 252b Second emergency stop button 255 Remote control release button 256 Remote Assistance Cancel Button 258 Unlock button 259 Start button 3 Backend 30 Servers (information processing devices) 301 Vehicle Information Transmission Unit 302 Operator terminal connection monitoring unit 303 Mode switching signal transmission section 304 Driving instruction signal transmission unit 305 Control signal transmission unit 306 Emergency stop signal transmission unit 307 Operator terminal status monitoring unit 308 Server Status Sending Unit 4. Edge 40 Edge Devices 401 Vehicle information transmission unit 402 Vehicle information input section 403 Command and Control Management Department 404 Behavior Monitoring Department 405 Driving Instruction Unit 406 Control Unit 407 Signal Monitoring Department 408 Emergency stop section 409 Server Status Monitoring Department 410 Edge State Transmitter 45 Camera 46 Vehicles (moving objects) 601 Intersection 602 Vehicle route 603 Stop line 604 Running Vehicles 605 Running Track 606 Street Parking 610 display screen 611 images 613 Map 615 Help button 616 Settings button 617 Logout button 618a~618d Transition buttons 620 display screen 621 images 622 Delay Time 623 Alert 630 State Transition Diagram 631~635 Status 640 display screen 641 Abnormal notification 651 Delay Information 652 Quality information 660a,660b display screen 661 images 662 Delay Information Display 663 Quality information display 664a Driving command button 664b Remote control button 665a,665b Display screen 670 Required Video Quality Information 8. Information processing equipment 81 processors 82 ROM 83 RAM 84 Equipment I / F section

Claims

1. An information processing method executed by an information processing system that supports remote control by at least one operator of a plurality of mobile objects, each configured to move autonomously and perform a predetermined task, and includes: at least one front end that acquires operations of the at least one operator; a plurality of edges including the plurality of mobile objects; and a back end that is communicatively connected to the at least one front end and each of the plurality of edges, In response to the operator's operation acquired by the front end, a first transmission mode in which an image of a target moving object among the plurality of moving objects is transmitted from the corresponding edge to the front end to remotely monitor the target moving object; a second transmission mode in which a single driving instruction signal is transmitted to the target moving object to remotely instruct the target moving object to be controlled; a third transmission mode for remotely controlling the target moving object by transmitting continuous control signals to the target moving object; In the second transmission mode and the third transmission mode, The front end periodically transmits a notification signal different from the driving instruction signal and the control signal to the back end; The back end monitors the notification signal from the front end, and stops the target moving body when a delay occurs in the period of the notification signal. Information processing methods.

2. In the second transmission mode and the third transmission mode, the front end executes two predetermined internal processes each outputting an execution signal, acquires the two execution signals, and transmits the notification signal to the back end when a difference between acquisition times of the two execution signals is equal to or less than a predetermined threshold. The information processing method according to claim 1 .

3. the front end includes a controller operated by the operator; The front end monitors a transmission cycle of a control signal periodically transmitted from the controller, and stops transmission of the notification signal to the back end when the transmission cycle is delayed. The information processing method according to claim 1 .

4. In the second transmission mode and the third transmission mode, The back end periodically transmits a notification signal different from the driving instruction signal and the control signal to the edge, The edge monitors the notification signal from the back end, and stops the target moving body when a delay occurs in the period of the notification signal. The information processing method according to claim 1 .

5. In the second transmission mode and the third transmission mode, the backend executes two predetermined internal processes each outputting an execution signal, acquires the two execution signals, and stops sending the notification signal to the edge when a difference between acquisition times of the two execution signals exceeds a predetermined threshold. The information processing method according to claim 4.

6. In the second transmission mode and the third transmission mode, a notification signal different from the driving instruction signal and the control signal is periodically transmitted to the target moving body by the edge; The target moving body monitors the notification signal from the edge, and stops the moving body when a delay occurs in the period of the notification signal. The information processing method according to claim 1 .

7. In the second transmission mode and the third transmission mode, two predetermined internal processes each outputting an execution signal are executed by the edge, the two execution signals are acquired, and when a difference between the acquisition times of the two execution signals exceeds a predetermined threshold, transmission of the notification signal to the target moving body is stopped. The information processing method according to claim 4.

8. In a transmission mode other than the third transmission mode, the control signal is masked by the edge. The information processing method according to claim 1 .

9. In a transmission mode other than the second transmission mode and the third transmission mode, the driving instruction signal is masked by the edge. The information processing method according to claim 1 .

10. In a transmission mode other than the second transmission mode, the driving instruction signal is masked by the edge. The information processing method according to claim 1 .

11. The edge does not permit execution of the second transmission mode or the third transmission mode when the target moving body is moving. The information processing method according to claim 1 .

12. In the second transmission mode and the third transmission mode, the edge is used to compare an instruction indicated by the control signal or the travel instruction signal transmitted to the target moving body with an execution result of control from the target moving body in accordance with the instruction, and the target moving body is stopped if there is a discrepancy between the instruction and the execution result. The information processing method according to claim 1 .

13. In the second transmission mode and the third transmission mode, The edge adds time information to an image of the target moving object and transmits the image to the front end via the back end; the front end stops the target moving object when a reception time of an image related to the target moving object from the edge is delayed from the time information by a predetermined threshold or more; The information processing method according to claim 1 .

14. The front end executes a screen display including an image of the target moving object from the edge, and transmits quality information indicating the quality of the image at the front end to the edge; When a round-trip delay related to the transmission of the quality information between the edge and the front end satisfies a predetermined threshold condition, an operation button for instructing a transition to the second transmission mode is enabled on the screen display by the edge. The information processing method according to claim 1 .

15. the edge permits execution of the second transmission mode when the round trip delay satisfies the specified threshold condition for a predetermined duration. The information processing method according to claim 14.

16. the edge restricts movement of the target moving object unless the round trip delay satisfies the specified threshold condition for a predetermined duration; The information processing method according to claim 14.

17. the back end restricts execution of the second transmission mode and the third transmission mode for one of the target mobile units to any one of the at least one front end that starts the transmission first; The information processing method according to claim 1 .

18. limiting execution of the second transmission mode and the third transmission mode in one of the front ends to any one edge that starts the transmission first among the plurality of edges; The information processing method according to claim 1 .

19. The front end acquires an operation of the operator instructing execution of one of the transmission modes, and transmits a signal corresponding to the acquired operation of the operator to the edge via two or more communication systems; When a signal corresponding to the operation of the operator is received in all of the two or more communication systems by the edge, a transmission mode based on the signal is executed. The information processing method according to claim 1 .

20. The front end transmits the driving instruction signal or the control signal to the edge via the two or more communication systems; When the edge receives a signal from the front end in all of the two or more communication systems, the edge controls the target moving object based on the signal.

20. The information processing method according to claim 19.

21. An information processing device that realizes one of the at least one frontends in an information processing system including at least one frontend that acquires operations of at least one operator and supports remote control by the at least one operator of a plurality of mobile objects, each of which is configured to move autonomously and execute a predetermined task, a plurality of edges including the plurality of mobile objects, and a backend that is communicatively connected to the at least one frontend and each of the plurality of edges, Acquire the operator's operation; In response to the acquired operator's operation, a first transmission mode in which an image of a target moving object among the plurality of moving objects is transmitted from the corresponding edge to the front end to remotely monitor the target moving object; a second transmission mode in which a single driving instruction signal is transmitted to the target moving object to remotely instruct the target moving object to be controlled; a third transmission mode for remotely controlling the target moving object by transmitting continuous control signals to the target moving object; In the second transmission mode and the third transmission mode, a notification signal different from the driving instruction signal and the control signal, the notification signal being monitored at the back end, and the target moving body being stopped if a delay occurs in the period of the notification signal; Information processing device.

22. In an information processing system including at least one front end that acquires operations of at least one operator and supports remote control of a plurality of mobile objects, each of which is configured to move autonomously and execute a predetermined task, a plurality of edges including the plurality of mobile objects, and a back end that is communicatively connected to the at least one front end and each of the plurality of edges, a computer that realizes any one of the at least one front end is provided, Acquire the operator's operation; In response to the acquired operator's operation, a first transmission mode in which an image of a target moving object among the plurality of moving objects is transmitted from the corresponding edge to the front end to remotely monitor the target moving object; a second transmission mode in which a single driving instruction signal is transmitted to the target moving object to remotely instruct the target moving object to be controlled; a third transmission mode for remotely controlling the target moving object by transmitting continuous control signals to the target moving object; In the second transmission mode and the third transmission mode, a notification signal different from the driving instruction signal and the control signal is periodically transmitted to the back end, the notification signal being monitored at the back end, and the target moving body being stopped if a delay occurs in the period of the notification signal; program.

Images