Information processing method, terminal device, and program

Abstract

This information processing method executed by a computer involves: acquiring location information indicating the current location of a mobile body and route information indicating the movement route of the mobile body (S11); acquiring road information (S12); predicting, on the basis of the location information, the route information, and the road information, at least one among a site and a timing at which an operator capable of remotely operating the mobile body starts a remote operation for the mobile body (S13); determining reliability for the prediction of the at least one among the site and the timing (S14); and determining, on the basis of the determined reliability, a display mode of state information indicating the state of the mobile body displayed to the operator prior to starting the remote operation (S16).

Description

Information processing method, terminal device, and program

[0001] The present disclosure relates to an information processing method, a terminal device, and a program.

[0002] Patent Literature 1 discloses a technology for a vehicle capable of driving control in both autonomous driving mode and remote driving mode, which predicts remote driving sections on a driving route where vehicle driving control in remote driving mode is required based on route information related to autonomous driving, thereby enabling smooth switching from autonomous driving mode to remote driving mode.

[0003] Japanese Patent Application Laid-Open No. 2021-22319

[0004] However, the technology of Patent Document 1 does not take into account the possibility that predictions of remote driving sections may be incorrect, so for example, if information indicating the status of the vehicle is displayed to the operator remotely operating the vehicle before the vehicle enters the remote driving section, unnecessary burden may be placed on the operator.

[0005] Therefore, the present disclosure provides an information processing method, a terminal device, and a program that can reduce unnecessary burdens on an operator who can remotely operate a moving object.

[0006] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, which acquires location information indicating the current location of a mobile body and route information indicating the path of movement of the mobile body, acquires road information, predicts at least one of the location and timing at which an operator capable of remotely controlling the mobile body will begin remotely controlling the mobile body based on the location information, the route information, and the road information, determines a reliability of the prediction of at least one of the location and the timing, and determines a display mode of status information indicating the status of the mobile body to be displayed to the operator before the remote control begins based on the determined reliability.

[0007] A terminal device according to one aspect of the present disclosure includes an acquisition unit that acquires information indicating a display mode for displaying the status information of the mobile body determined by the above-described information processing method, and a control unit that displays the status information in the acquired display mode on a display device of an operator in charge of the mobile body.

[0008] A program according to one aspect of the present disclosure is a program for causing a computer to execute the above-described information processing method.

[0009] According to one aspect of the present disclosure, it is possible to realize an information processing method and the like that can reduce unnecessary burdens on an operator who can remotely operate a moving object.

[0010] FIG. 1 is a diagram showing a schematic configuration of an information processing system according to an embodiment. FIG. 2 is a block diagram showing a functional configuration of a server device according to an embodiment. FIG. 3 is a block diagram showing a functional configuration of an in-vehicle device according to an embodiment. FIG. 4 is a block diagram showing a functional configuration of an operator terminal device according to an embodiment. FIG. 5 is a flowchart showing an operation of the information processing system according to an embodiment. FIG. 6 is a diagram showing an example of status information displayed on a display device according to an embodiment. FIG. 7 is a diagram showing another example of status information displayed on a display device according to an embodiment. FIG. 8 is a flowchart showing an operation of an information processing system according to a first modification of the embodiment. FIG. 9 is a flowchart showing an operation of an information processing system according to a second modification of the embodiment. FIG. 10 is a diagram showing an example of status information displayed on a display device according to the second modification of the embodiment. FIG. 11 is a flowchart showing an example of an operation for selecting a vehicle to be displayed on a split display screen according to the second modification of the embodiment. FIG. 12 is a flowchart showing another example of an operation for selecting a vehicle to be displayed on a split display screen according to the second modification of the embodiment. FIG. 13 is a diagram for explaining the timing of switching display screens according to a third modification of the embodiment. FIG. 14A is a diagram showing an example of status information displayed on a display device according to a fourth modification of the embodiment. Fig. 14B is a diagram showing an example of a display in which one or a preset number of pieces of status information shown in Fig. 14A are left and display of the other remote intervention timings is omitted. Fig. 15 is a diagram showing a first example of status information displayed on a display device according to a fifth modification of the embodiment. Fig. 16 is a diagram showing a second example of status information displayed on a display device according to a fifth modification of the embodiment. Fig. 17 is a diagram showing a third example of status information displayed on a display device according to the fifth modification of the embodiment.

[0011] (Background to the Invention of the Present Disclosure) Before describing the embodiments of the present disclosure, the background to the invention of the present disclosure will be described.

[0012] In a situation where remote intervention is required for a vehicle, it is being considered to present the vehicle's status to the operator in advance on a screen (for example, an individual screen, which will be described later). This allows the operator to grasp the vehicle's status in advance, enabling them to quickly begin responding when remote intervention is required.

[0013] Remote intervention means that an operator remotely controls the driving of an autonomously traveling vehicle. An individual screen is a screen that allows the operator to grasp the vehicle's status (e.g., the situation around the vehicle) using, for example, video and audio from an onboard camera, microphone, etc., and is a screen with higher information detail. In other words, an individual screen is a screen that places a greater burden and time on the operator when grasping the vehicle's status.

[0014] The level of information detail does not refer to the data size of the information, but rather to the complexity of the information as perceived by the operator. The more complex the information, the more detailed the vehicle's situation can be grasped, but this requires more effort and time. For example, even if the data size is the same when viewing a vehicle on a map screen and when viewing an image from a camera mounted on the vehicle, the image from the camera is more complex and has a higher level of information detail as perceived by the operator.

[0015] In this case, a prediction is made in advance as to whether remote intervention is necessary for the vehicle, and if it is predicted that remote intervention is necessary, the status of the vehicle is displayed on the screen. However, it is possible that a prediction error occurs in predicting whether remote intervention is necessary. A prediction error is when the prediction is incorrect, and for example, it means that a prediction that remote intervention is necessary is erroneously made when in fact remote intervention is not necessary, that is, a false detection that remote intervention is necessary. The necessity of remote intervention means that remote operation by an operator is necessary.

[0016] When the status of a vehicle that is erroneously detected as requiring remote intervention is displayed on the screen, the operator has to check the status of the vehicle on a separate screen, which does not require remote intervention, placing an unnecessary burden on the operator. Furthermore, whether or not the detection is erroneous is difficult to determine in advance, since it is not clear until the vehicle actually arrives at the location where remote intervention is predicted. The burden here refers to the burden on the operator caused by remote monitoring before remote intervention.

[0017] In Patent Document 1, the possibility of false detection is not taken into consideration, which may result in unnecessary burden on the operator.

[0018] Therefore, the inventors of the present application have conducted extensive research into information processing methods etc. that can reduce the unnecessary burden that can be placed on an operator who can remotely operate a mobile object due to erroneous detection, and have devised the information processing methods etc. described below.

[0019] An information processing method according to a first aspect of the present disclosure is an information processing method executed by a computer, which acquires location information indicating the current location of a mobile body and route information indicating the path of movement of the mobile body, acquires road information, predicts at least one of the location and timing at which an operator capable of remotely controlling the mobile body will begin remotely controlling the mobile body based on the location information, the route information, and the road information, determines a reliability of the prediction of at least one of the location and the timing, and determines a display mode of status information indicating the status of the mobile body to be displayed to the operator before the remote control begins based on the determined reliability.

[0020] This allows the situation information to be displayed in a display mode that corresponds to the reliability of the remote operation prediction. For example, in the case of a prediction with low reliability, a display mode that reduces the burden on the operator is determined, thereby reducing unnecessary burden on the operator when the prediction is incorrect. Therefore, according to the information processing method, unnecessary burden on the operator who can remotely control the mobile object can be reduced.

[0021] Furthermore, for example, an information processing method according to a second aspect may be the information processing method according to the first aspect, in which the reliability is determined based on accuracy of the prediction regarding the remote operation.

[0022] This makes it possible to reduce unnecessary burdens on operators who can remotely operate mobile objects by using reliability according to the accuracy of the prediction.

[0023] Furthermore, for example, the information processing method according to the third aspect may be the information processing method according to the second aspect, in which the reliability is determined based on performance information including predictions of past remote control for the same reason as the reason for predicting that remote control is necessary for the moving body and the performance of the remote control in response to the predictions.

[0024] This makes it possible to reduce unnecessary burdens on operators who can remotely control mobile objects by using reliability according to performance information in which the reason for predicting the need for remote control is the same.

[0025] Also, for example, an information processing method according to a fourth aspect may be an information processing method according to the third aspect, in which the performance information includes information regarding the accuracy rate of the prediction, and the reliability is determined based on the accuracy rate.

[0026] In this way, by using a reliability corresponding to the accuracy rate of the prediction, it is possible to reduce unnecessary burden on an operator who can remotely operate a mobile object.

[0027] Furthermore, for example, an information processing method according to a fifth aspect may be the information processing method according to the fourth aspect, wherein there are a plurality of moving bodies, and the plurality of moving bodies are grouped into one or more groups based on at least one of the type of moving body, environmental information related to the movement of the moving bodies, and time information related to the time at which the moving bodies move, and the accuracy rate for calculating the reliability of the moving body may be calculated based on the performance information of one or more moving bodies of the group to which the moving body belongs.

[0028] This allows the hit rate to be calculated based on the performance information of one or more moving objects of the group to which the moving object belongs, so that the hit rate can be calculated according to the moving object.

[0029] Also, for example, the information processing method according to the sixth aspect may be the information processing method according to the first aspect, in which the reliability is determined based on the probability of occurrence of the reason for predicting that remote control of the moving body is required.

[0030] In this way, by using a reliability according to the probability of occurrence of the reason, it is possible to reduce unnecessary burden on an operator who can remotely operate a mobile object.

[0031] Furthermore, for example, an information processing method according to a seventh aspect may be the information processing method according to the sixth aspect, wherein there are a plurality of moving bodies, and the plurality of moving bodies are grouped into one or more groups based on at least one of the type of moving body, environmental information related to the movement of the moving bodies, and time information related to the time at which the moving bodies move, and the occurrence probability for calculating the reliability of the moving body is calculated based on performance information of one or more moving bodies of the group to which the moving body belongs, and the performance information may include information related to the occurrence probability.

[0032] This allows the occurrence probability to be calculated based on performance information of one or more moving bodies of the group to which the moving body belongs, making it possible to calculate the occurrence probability according to the moving body.

[0033] Furthermore, for example, an information processing method according to the eighth aspect may be an information processing method according to any one of the first to seventh aspects, in which it is determined whether the reliability exceeds a threshold, and if the reliability exceeds the threshold, the display mode may be determined to have a higher level of information detail than when the reliability is equal to or less than the threshold.

[0034] As a result, when the reliability is below the threshold, a display mode with a low level of information detail is selected, so even if a prediction with a reliability below the threshold is incorrect, unnecessary burden on the operator who can remotely operate the mobile object can be further reduced.

[0035] Also, for example, an information processing method according to a ninth aspect may be the information processing method according to the eighth aspect, in which, if the reliability exceeds the threshold, an image of the surroundings of the moving body acquired from the moving body is displayed, and, if the reliability is equal to or less than the threshold, information about the moving body is displayed in map information.

[0036] This allows the operator to check the vehicle's condition in detail using images when the prediction reliability exceeds a threshold, and allows the operator to check the vehicle's condition in detail using map information with less burden when the prediction reliability is below the threshold.

[0037] Furthermore, for example, an information processing method according to a tenth aspect is the information processing method according to the ninth aspect, and when the reliability exceeds the threshold, the image may be displayed for a certain period of time.

[0038] This reduces the burden on the operator caused by frequent changes in the display.

[0039] Also, for example, an information processing method according to an eleventh aspect may be an information processing method according to the ninth aspect, in which it is determined whether or not to display the image based on historical information of a determination of whether or not the reliability exceeds the threshold value.

[0040] This reduces the burden on the operator caused by frequent changes in the display.

[0041] Furthermore, for example, an information processing method according to a twelfth aspect may be an information processing method according to any one of the eighth to eleventh aspects, in which the reliability for each of a plurality of moving bodies is determined, and if there are two or more moving bodies that exceed the threshold, a display mode of status information indicating the status of one of the two or more moving bodies that is identified based on the impact of a delay in the start of the remote control may be determined.

[0042] As a result, when there are two or more moving objects that exceed the threshold value, it is possible to determine one moving object for which the display mode is to be determined, based on the effect of a delay in the start of remote control.

[0043] Furthermore, for example, an information processing method according to a thirteenth aspect may be an information processing method according to any one of the first to twelfth aspects, in which the reliability of each of a plurality of moving bodies is determined, and status information of the predetermined number of moving bodies is displayed in a predetermined number of divided display areas into which the display screen of a display device is divided, based on the reliability of each of the plurality of moving bodies.

[0044] This makes it possible to display status information for a predetermined number of moving bodies to the operator when there are multiple predictions whose reliability exceeds the threshold.

[0045] Also, for example, an information processing method according to a fourteenth aspect may be an information processing method according to the thirteenth aspect, in which, of the predetermined number of split display areas, a split display area in which status information of a new moving object is displayed may be highlighted.

[0046] This allows the operator to easily know which moving object status information has been newly added to the split display screen, thereby reducing the burden on the operator when using the split display screen.

[0047] Also, for example, the information processing method according to the fifteenth aspect may be the information processing method according to the thirteenth or fourteenth aspect, in which the predetermined number is determined by the content of the remote operation required of the operator.

[0048] This allows the predetermined number to be determined according to the content of the remote operation, so that it is possible to prevent the burden on the operator from increasing due to, for example, the predetermined number being set excessively.

[0049] Furthermore, for example, an information processing method according to a sixteenth aspect may be an information processing method according to any one of the first to fourteenth aspects, in which the reliability of each of a plurality of moving bodies is determined, and if the reliability of each of the plurality of moving bodies is equal to or less than a threshold value, information for changing the threshold value is presented to the operator.

[0050] This allows the threshold to be changed. For example, by setting the threshold higher, the proportion of images with a high level of information detail displayed decreases, thereby reducing the burden on the operator when checking the situation information.

[0051] Furthermore, for example, an information processing method according to a seventeenth aspect is an information processing method according to any one of the first to fifteenth aspects, in which the road information includes section information indicating at least one of a section where autonomous driving is prohibited and a section where autonomous driving cannot be continued, and at least one of the location and the timing may be predicted based on the section information.

[0052] This allows predictions regarding remote operations to be made using the section information, resulting in more accurate predictions.

[0053] In addition, a terminal device according to an eighteenth aspect of the present disclosure includes an acquisition unit that acquires information indicating a display mode for displaying the status information of the moving body determined by an information processing method according to any one of the first to seventeenth aspects, and a control unit that displays the status information in the acquired display mode on a display device of an operator in charge of the moving body.

[0054] This allows the terminal device to display the status information to the operator in a display format that corresponds to the reliability of the prediction of the remote operation. For example, if the prediction has a low reliability, the status information can be displayed in a display format that reduces the burden on the operator. Therefore, the terminal device can reduce unnecessary burden on the operator who can remotely operate the mobile object.

[0055] A program according to a nineteenth aspect of the present disclosure is a program for causing a computer to execute the information processing method according to any one of the first to seventeenth aspects.

[0056] This provides the same effect as the above-described information processing method.

[0057] These general or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of the system, method, integrated circuit, computer program, or recording medium. The program may be pre-stored in the recording medium, or may be supplied to the recording medium via a wide area communication network including the Internet.

[0058] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0059] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not described in independent claims are described as optional components.

[0060] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and redundant explanations are omitted or simplified.

[0061] Furthermore, in this specification, numerical values ​​and numerical ranges are not expressions that express only the strict meaning, but are expressions that mean that they also include a substantially equivalent range, for example, a difference of about several percent (or about 10%).

[0062] (Embodiment) Hereinafter, an information processing method according to the present embodiment will be described with reference to FIGS.

[0063] 1. Configuration of Information Processing System First, the configuration of an information processing system according to this embodiment will be described with reference to Figures 1 to 4. Figure 1 is a diagram showing a schematic configuration of an information processing system 1 according to this embodiment.

[0064] As shown in Fig. 1, the information processing system 1 is an information processing system for remotely monitoring and remotely operating a plurality of vehicles (vehicles M1, ..., Mm) that can be operated by a plurality of operators (operators O1, ..., On). The information processing system 1 includes a server device 10 and operator terminal devices D1, ..., Dn. The information processing system 1 may also include vehicles M1, ..., Mm and a service system 50. Note that remote operation means that the operator O1, etc. controls the acceleration / deceleration and steering angle of the vehicle M1, etc., via a network N as necessary.

[0065] The number of operator terminal devices D1, ..., Dn and vehicles M1, ..., Mm included in the information processing system 1 is not particularly limited as long as they are one or more. Hereinafter, the operator terminal devices D1, ..., Dn will also be referred to as "operator terminal device D1, etc.", and the vehicles M1, ..., Mm will also be referred to as "vehicles M1, etc." Operators O1, ..., On are assigned to the operator terminal devices D1, etc., to remotely monitor and remotely operate the vehicles M1, etc., using the operator terminal devices. Hereinafter, the operators O1, ..., On will also be referred to as "operators O1, etc." Each of the operators O1, ..., On may be responsible for remotely operating different vehicles, for example.

[0066] The server device 10 is a processing device that executes various processes for remotely monitoring and remotely operating the vehicle M1, etc. The server device 10 is characterized in that it changes the method of notifying the operator O1, etc. of the status of the vehicle M1, etc., based on the prediction accuracy when it predicts that remote operation of the vehicle M1, etc. is necessary. The notification method is, for example, the display method (display mode), which will be described in detail later. Furthermore, the prediction accuracy can be obtained based on the reason for predicting that remote operation is necessary.

[0067] The reason for predicting the need for remote control is set in advance and may be acquired from, for example, road information. Examples of reasons for predicting the need for remote control include, but are not limited to, traffic congestion, a section where autonomous driving is not possible, an accident, or the presence of an obstacle such as a fallen object.

[0068] The server device 10 is installed in a location where it can be communicatively connected to the network N, and its physical location may be anywhere. As an example, if the server device 10 is installed inside or near the space where the operator terminal device D1, etc. are installed, there is an advantage that communication with the operator terminal device D1, etc. can be established in a short time.

[0069] FIG. 2 is a block diagram showing the functional configuration of the server device 10 according to the present embodiment.

[0070] 2, the server device 10 includes a communication unit 11, a vehicle information acquisition unit 12, an operator information acquisition unit 13, a remote intervention timing prediction unit 14, a prediction reliability determination unit 15, a storage unit 16, and a screen switching determination unit 17. The server device 10 is realized by, for example, a computer.

[0071] The communication unit 11 communicates with each of the vehicles M1, etc., each of the operator terminal devices D1, etc., and the service system 50 via the network N. The communication unit 11 is configured to include a communication circuit (communication module).

[0072] The vehicle information acquisition unit 12 acquires vehicle information from each of the vehicles M1, etc. via the communication unit 11. The vehicle information includes sensing data from one or more sensors equipped in the vehicle, current position information of the vehicle, image data of the surroundings of the vehicle, etc. The vehicle information may also include route information indicating the route of travel of the vehicle during autonomous travel, audio information indicating audio collected by a microphone mounted on the vehicle, etc. The travel route is an example of a movement route.

[0073] Furthermore, the vehicle information acquisition unit 12 acquires road information from a server device or the like that manages road information. The road information includes section information that indicates at least one of sections where autonomous driving is prohibited and sections where autonomous driving cannot be continued. The road information includes, for example, information about traffic congestion, information about events on the road such as construction, accidents, and festivals, and information about sections where the vehicle M1 or the like cannot continue driving.

[0074] The operator information acquisition unit 13 acquires operator information about each operator O1, etc. from each operator terminal device D1, etc. via the communication unit 11. The operator information includes information indicating the status of the operator. The operator information may include at least one of screen information presented to the operator O1, the operator O1's skill in remote intervention, and the operator O1's fatigue level. The screen information may include information indicating at least one of the vehicle presented on an individual screen (e.g., see FIG. 6 ), the remote intervention timing, the prediction reliability, and whether the operator O1 is currently remotely operating the vehicle.

[0075] The remote intervention timing prediction unit 14 predicts the timing of remote intervention by the operator O1, etc. for each of the vehicles M1, etc. The remote intervention timing prediction unit 14 predicts the timing of remote intervention for the vehicle M1, for example, based on the position information of the vehicle M1, the route information of the vehicle M1, and road information. Note that the remote intervention timing prediction unit 14 may also predict that there is no timing for remote intervention, i.e., that remote intervention is not necessary.

[0076] Furthermore, the remote intervention timing prediction unit 14 may predict a remote intervention point of an operator O1 or the like for each of the vehicles M1 or the like, instead of or in addition to the remote intervention timing. The remote intervention point is a point where remote operation by the operator O1 or the like starts, and is a point on the travel route of the vehicle that has not yet been driven. The remote intervention point may be indicated by, for example, latitude, longitude, and altitude.

[0077] In this way, the remote intervention timing prediction unit 14 may predict, based on the position information, route information, and road information, at least one of the point and timing at which an operator O1, etc. capable of remotely operating a vehicle M1, etc., will start remotely operating the vehicle M1, etc. Furthermore, if the road information includes section information, the remote intervention timing prediction unit 14 may predict, based on the position information, route information, and section information, at least one of the point and timing at which remote operation will start.

[0078] The prediction reliability determination unit 15 determines the reliability (prediction reliability) of the prediction by the remote intervention timing prediction unit 14, i.e., the prediction of at least one of the remote intervention timing and the remote intervention point, based on the reason for predicting the need for remote operation of the vehicle M1, etc. The reliability means the degree of possibility that the remote intervention predicted by the remote intervention timing prediction unit 14 will actually be performed, i.e., the prediction accuracy of the remote intervention prediction by the remote intervention timing prediction unit 14.

[0079] The prediction reliability determiner 15 may determine the reliability based on, for example, past predictions of remote intervention timing and performance information indicating whether or not a remote intervention was actually performed within a predetermined period including the predicted remote intervention timing. Also, the prediction reliability determiner 15 may determine the reliability based on, for example, past predictions of remote intervention location and performance information indicating whether or not a remote intervention was actually performed within a predetermined range including the predicted remote intervention location.

[0080] In this way, the prediction reliability determination unit 15 determines the reliability based on the past remote control predictions for the same reason as the current remote control necessity prediction and the performance information including the performance of the remote control for the prediction. Note that the performance information exists for each reason for predicting the need for remote control of the vehicle M1, etc.

[0081] The storage unit 16 stores various information used for various processes of the server device 10. The storage unit 16 stores map information, performance information related to the predictions made by the remote intervention timing prediction unit 14, and the like. The performance information may include, for example, information indicating the accuracy rate of the predictions made by the remote intervention timing prediction unit 14, or may include the probability (certainty) of the occurrence of the reason. The occurrence probability may be the probability that the reason occurs among multiple reasons, or the probability that the reason occurs within a predetermined period of time. The storage unit 16 may also store route information for each vehicle M1, etc.

[0082] The storage unit 16 is realized by, for example, a semiconductor memory or the like, but is not limited to this.

[0083] The screen switching determination unit 17 determines a display mode of status information of the vehicle M1, etc., to be displayed to the operator O1, etc. before the start of remote operation, based on the reliability determined by the prediction reliability determination unit 15. For example, when the operator O1 is in charge of the vehicle M1, the screen switching determination unit 17 determines a display mode of a display showing the status of the vehicle M1 to the operator O1, based on the reliability of the vehicle M1.

[0084] Furthermore, the screen switching determination unit 17 determines whether the vehicle currently displayed on the individual screen for the operator O1, which is included in the operator information, is different from the vehicle predicted as the next remote intervention timing, and if the two vehicles are different, determines whether to switch the vehicle displayed on the individual screen. For example, if the two vehicles are different, the screen switching determination unit 17 may determine to switch the vehicle displayed on the individual screen.

[0085] Furthermore, even if the vehicle currently displayed to the operator O1 on the individual screen and the vehicle predicted as the next remote intervention timing are the same vehicle, if the reliability of the prediction of at least one of the remote intervention timing and the remote intervention point falls below a predetermined value, the screen switching determination unit 17 may determine whether to switch from the current display mode to another display mode. For example, if the reliability of the prediction for one vehicle changes over time from a state higher than a predetermined value to a predetermined value or below, the screen switching determination unit 17 may determine to switch the display mode of the one vehicle from the individual screen to the overall management screen.

[0086] Furthermore, if the operator O1 is already remotely operating another vehicle, the screen switching determination unit 17 may determine not to switch the display mode, or may increase the threshold value of the prediction reliability of at least one of the remote intervention timing and the remote intervention location, which are used to determine whether to switch to the individual screen. Thus, the screen switching determination unit 17 may determine not to switch to the individual screen even if the prediction reliability is equal to or greater than a threshold value, or may dynamically change the threshold value, depending on the situation of the operator O1. The threshold value may be determined for at least one of the time when the remote intervention prediction was made, the location of the vehicle when the remote intervention prediction was made, the content of the remote intervention, and information about the vehicle. The information about the vehicle includes at least one of the vehicle model, vehicle performance, and vehicle size.

[0087] Furthermore, the screen switching determination unit 17 may determine which vehicle should be prioritized for operation: the vehicle currently presented to the operator O1, or the vehicle predicted as the next vehicle for remote intervention, based on safety risks, service effects, etc. For example, the screen switching determination unit 17 may determine the vehicle to be switched to an individual screen based on safety risks, service effects, etc. in addition to the reliability of the prediction.

[0088] For example, the screen switching determination unit 17 may prioritize remote operation response to a vehicle that is stuck on a road with heavy traffic over a vehicle that is stuck on a road with light traffic. This prioritizes displaying an image of a vehicle that is stuck on a road with heavy traffic, which is expected to have the effect of reducing safety risks. Furthermore, the screen switching determination unit 17 may prioritize remote operation response when a vehicle that is heading to a user waiting at a destination, such as a food delivery vehicle, becomes stuck over a vehicle that is patrolling to receive orders. This prioritizes displaying an image of a vehicle that is heading to a user waiting at a destination, such as a food delivery vehicle, which is expected to have the effect of improving service effectiveness.

[0089] The vehicle M1, etc. is a vehicle with or without a human on board, and may be, for example, a self-driving vehicle or an autonomous mobile robot that performs a predetermined service such as delivery. The vehicle M1, etc. is a vehicle that can be remotely monitored and remotely operated by an operator O1, etc. The vehicle M1, etc. is present on a road, a parking lot, a sidewalk, etc. The vehicle M1, etc. is an example of a moving body.

[0090] The vehicle M1 etc. is equipped with one or more cameras that capture images of the surroundings of the vehicle, as well as multiple sensors that detect the vehicle's speed, acceleration, jerk (jerk), steering angle, remaining fuel, operation status of turn signals, operation status of an anti-lock braking system (ABS) or automatic emergency braking (AEB), the presence or absence, number and type of surrounding objects, distance to surrounding objects, speed difference, acceleration difference, etc. The vehicle M1 etc. transmits vehicle information including image data captured by the one or more cameras and sensing data acquired by the multiple sensors, such as the vehicle's speed, the presence or absence of surrounding objects, to the server device 10 via a communication interface and the network N.

[0091] Each of the vehicles M1 and the like is equipped with an in-vehicle device 40. The in-vehicle device 40 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the functional configuration of the in-vehicle device 40 according to this embodiment.

[0092] As shown in Figure 3, the in-vehicle device 40 includes a communication unit 41, a destination acquisition unit 42, a detection unit 43, a self-position estimation unit 44, a judgment unit 45, a remote control information acquisition unit 46, a vehicle control unit 47, and a vehicle information generation unit 48.

[0093] The communication unit 41 communicates with the server device 10, the operator terminal device D1, etc. via the network N. The communication unit 41 includes a communication circuit (communication module).

[0094] The destination acquisition unit 42 acquires destination information indicating a destination of the vehicle's travel via the communication unit 41. If the vehicle is a vehicle that provides a predetermined service, the destination acquisition unit 42 may acquire the destination information from the service system 50. Furthermore, if a passenger is on board the vehicle, the destination acquisition unit 42 may acquire the destination information based on an operation on an input device by the passenger. The input device may be, for example, a button, a touch panel, or the like, but may also be configured to accept operations by voice or gesture.

[0095] The detection unit 43 is a sensor that detects information related to the vehicle. The detection unit 43 includes a sensor that detects information for estimating the current position of the vehicle, and may include, for example, a GPS (Global Positioning System) sensor that receives GPS signals output from GPS satellites. The detection unit 43 may also include multiple sensors described above.

[0096] The self-position estimation unit 44 estimates the current position of the vehicle based on the detection result of the detection unit 43. The self-position estimation unit 44 may estimate the current position of the vehicle based on a GPS signal received by the detection unit 43, for example.

[0097] The current location may be estimated using any known technique other than using GPS signals.

[0098] The determination unit 45 determines travel information for autonomous travel based on the current position and destination of the vehicle. The travel information includes a travel route, a travel schedule, and the like.

[0099] The remote operation information acquisition unit 46 acquires remote operation information from the operator terminal device D1 and the like via the communication unit 41.

[0100] The vehicle control unit 47 controls the traveling of the vehicle. When the vehicle is traveling autonomously, the vehicle control unit 47 controls the traveling of the vehicle based on the traveling information determined by the determination unit 45. When the vehicle is traveling under remote control, the vehicle control unit 47 controls the traveling of the vehicle based on the remote control information acquired by the remote control information acquisition unit 46.

[0101] The vehicle information generation unit 48 generates vehicle information based on the detection result of the detection unit 43. The vehicle information generation unit 48 may generate vehicle information that includes, for example, at least the current position of the vehicle and image data of the surrounding area. The generated vehicle information is transmitted to the server device 10 via the communication unit 41.

[0102] 1 again, the operator terminal device D1 etc. is a terminal device used by the operator O1 etc., and executes processing to present (e.g., display) information from the server device 10 to the operator O1 etc. and output information (e.g., remote control information) based on input from the operator O1 etc. to the vehicle M1 etc. The operator terminal device D1 etc. is an example of a terminal device.

[0103] 4 is a block diagram showing the functional configuration of the operator terminal device D1 according to this embodiment. Note that the configurations of the operator terminal devices D1 and the like may be the same, and the following describes the operator terminal device D1 used by the operator O1.

[0104] 4, the operator terminal device D1 includes a communication unit 31, a display generation unit 32, a terminal control unit 33, and an operator information generation unit 34. The operator terminal device D1 is connected to a first display device 21 and a second display device 22. The number of first display devices 21 and second display devices 22 connected to the operator terminal device D1 is not particularly limited, and may be one or more.

[0105] The first display device 21 displays, for example, status information of a vehicle predicted to require remote intervention, such as vehicle M1. In this embodiment, the first display device 21 displays an image of the surroundings of the vehicle predicted to require remote intervention, captured by the vehicle. The first display device 21 can also be considered a display device for displaying individual screens, which will be described later. This allows the operator O1 to know the detailed status of the vehicle predicted to require remote intervention before starting the remote intervention, simply by looking at the screen of the first display device 21.

[0106] The second display device 22 displays, for example, an image showing a simplified view of the status of one or more vehicles, such as vehicle M1, that the operator is responsible for. In this embodiment, the second display device 22 displays information about each of the one or more vehicles that the operator is responsible for on a map, allowing the operator to view the information from a bird's-eye view. The second display device 22 can also be considered a display device for displaying an overall management screen (see, for example, FIG. 7 ), which will be described later. This allows the operator O1 to know the status of each of the one or more vehicles that he or she is responsible for, simply by looking at the image on the second display device 22. The overall management screen is, for example, a map screen on which the vehicle's location, driving route, etc. are superimposed, and provides less detailed information than the individual screens. In other words, the overall management screen is a screen that requires less burden and time than the individual screens when the operator O1 grasps the status of the vehicles.

[0107] The first display device 21 and the second display device 22 are realized by, for example, a liquid crystal display device, an organic EL (Electro Luminescence) display device, or the like, but are not limited to these.

[0108] The second display device 22 of each of the operator terminal devices D1, etc. may display the same image. For example, the second display device 22 of each of the operator terminal devices D1, etc. may display an image that simply shows all the statuses of the vehicle M1, etc. included in the information processing system 1. Furthermore, two or more operators O1, etc. may share one second display device 22.

[0109] The first display device 21 may display, on an individual screen, an image of the surroundings of a vehicle whose prediction reliability exceeds a threshold, or may display, on an individual screen, an image of the surroundings of only vehicles whose prediction reliability exceeds a threshold and whose remote intervention timing is within a predetermined period from the current time (for example, imminent). Status information of vehicles whose prediction reliability exceeds a threshold and whose remote intervention timing is more than a predetermined period from the current time may be displayed on the overall management screen of the second display device 22. In this way, the first display device 21 may display, on an individual screen, an image of the surroundings of the vehicle only when the prediction reliability exceeds a threshold and other predetermined conditions are satisfied. Note that "imminent" may be, for example, several minutes or several tens of minutes.

[0110] The first display device 21 and the second display device 22 may be realized by different display devices (displays) or by a single display device (display). When realized by a single display device, the first display device 21 and the second display device 22 may be realized by dividing the same display into a first display section and a second display section. In this case, the first display section and the second display section are realized by different display areas of a single display.

[0111] The communication unit 31 communicates with the server device 10 and the vehicle M1, etc., via the network N. The communication unit 31 acquires, for example, information indicating a display mode for displaying status information of the vehicle M1, etc., determined by the server device 10. The communication unit 31 is configured to include a communication circuit (communication module). The communication unit 31 is an example of an acquisition unit.

[0112] The display generation unit 32 generates images to be displayed on the first display device 21 and the second display device 22 based on the display mode determined by the server device 10 .

[0113] The terminal control unit 33 controls each component of the operator terminal device D1. The terminal control unit 33 also controls the display mode of status information of the vehicle M1, etc., displayed on the first display device 21 and the second display device 22. When the operator O1 is in charge of the vehicle M1, the terminal control unit 33 causes the display device connected to the operator terminal device D1 to display the status information of the vehicle M1 in the display mode acquired via the communication unit 31. The terminal control unit 33 outputs the image data generated by the display generation unit 32 to the first display device 21 and the second display device 22.

[0114] The terminal control unit 33 may also be connected to a user interface (not shown) that accepts input from the operator O1 when remotely operating the vehicle M1, and may acquire input from the operator O1 from the user interface. The user interface is a device that accepts input from the operator O1 using a button, a stick, a touch panel, or the like, but may also be a device that accepts input by voice, gesture, or the like. The user interface may also be a device dedicated to the mobile object, such as a steering wheel, brake, or accelerator. The terminal control unit 33 is an example of a control unit.

[0115] The operator information generation unit 34 generates operator information based on at least one of the screen information presented to the operator O1 by the first display device 21 and the second display device 22 and the input of the operator O1 acquired via the user interface on the screen, and transmits the generated operator information to the server device 10, etc. via the communication unit 31. The operator information includes remote operation information, information indicating the status of the operator, etc.

[0116] The operator terminal device D1 may be configured to include a first display device 21 and a second display device 22 .

[0117] The service system 50 is an information processing system that executes processes related to a predetermined service executed by the vehicle M1, etc. The service system 50 may be configured to include, for example, a server device managed by a business that provides the service. The service system 50 may, for example, transmit destination information to the vehicle that executes the service.

[0118] 2. Operation of Information Processing System Next, the operation of the information processing system 1 configured as described above will be described with reference to FIGS. 5 to 7. FIG. 5 is a flowchart showing the operation (information processing method) of the information processing system 1 according to this embodiment. FIG. 5 shows the operation executed by the server device 10. In FIG. 5, an example will be described in which the remote intervention timing prediction unit 14 predicts the remote intervention timing out of the remote intervention point and the remote intervention timing.

[0119] 5, the vehicle information acquisition unit 12 of the server device 10 acquires vehicle information including the current position and the driving route of each vehicle M1 etc. from the vehicle M1 etc. via the communication unit 11 (S11). The vehicle information acquisition unit 12 periodically acquires the vehicle information from each vehicle M1 etc., for example.

[0120] Next, the vehicle information acquisition unit 12 acquires current road information from a server device or the like that manages road information (S12). The vehicle information acquisition unit 12 may acquire, for example, road information for an area including the driving route of each vehicle M1, etc. The vehicle information acquisition unit 12 may acquire the road information periodically, or may acquire the road information at the same time as acquiring the vehicle information, for example.

[0121] Next, the remote intervention timing prediction unit 14 predicts the timing of remote intervention for each vehicle (e.g., vehicle M1) based on location information indicating the current location of each vehicle (e.g., vehicle M1), route information for each vehicle (e.g., vehicle M1), and road information (S13). Based on the location information, route information, and road information for vehicle M1, the remote intervention timing prediction unit 14 determines whether remote operation of vehicle M1 is required, and predicts the timing of remote intervention for vehicle M1 for which remote operation is determined to be required. For example, the remote intervention timing prediction unit 14 predicts the time when vehicle M1 will enter the section indicated by the section information included in the road information, based on the current location and driving route of vehicle M1. The remote intervention timing is a time in the future relative to the time when the prediction is made by the remote intervention timing prediction unit 14.

[0122] The remote intervention timing is, for example, the predicted time when the vehicle M1 will first pass any position within the set section. The remote intervention timing may also be, for example, the time when the vehicle M1 will first enter the set section. The route information of the vehicle M1 is information about the travel route of the vehicle M1. The remote intervention timing prediction unit 14 may predict the time when the vehicle M1 will pass the position of the section based on the distance between the current position and the position of the section that the vehicle M1 is predicted to pass first on the travel route from the current position, and on speed information of the vehicle M1.

[0123] The section includes, for example, at least one of a section in the road information where autonomous driving is prohibited, a section where autonomous driving cannot continue, a congested section, a section where an accident has occurred, a section where there is an obstacle such as a fallen object, and a section where an event on the road such as a festival is occurring, etc. The remote intervention point is a position within such a section.

[0124] The multiple sections may be set by dividing a group of routes into multiple sections using a predetermined point such as an intersection as a dividing point. A single route from one intersection to the next may be divided into two sections, each section being separated by a traveling direction. Furthermore, assuming remote intervention at an intersection, the intersection may be divided into multiple sections according to the route of entry. In this way, a specific area such as an intersection may be divided into multiple sections. Furthermore, if a section is longer than a certain distance, a unit distance may be defined, and the section may be divided into separate sections for each unit distance. For example, each of the multiple sections may be a section equal to or shorter than a predetermined unit distance.

[0125] Next, the prediction reliability determination unit 15 determines the reliability by calculating the reliability of the prediction of the remote intervention timing for each of the vehicles M1, etc., made by the remote intervention timing prediction unit 14 (S14). The prediction reliability determination unit 15 determines the reliability, for example, based on performance information. The prediction reliability determination unit 15 may calculate the accuracy rate of the prediction made by the remote intervention timing prediction unit 14 for each reason for predicting that remote operation is required for the vehicles M1, etc., which is stored in the storage unit 16, and calculate a value based on the accuracy rate as the reliability. The prediction reliability determination unit 15 may calculate the reliability so that the higher the accuracy rate of the prediction, the higher the reliability value.

[0126] The accuracy rate may be calculated from the ratio of "correct" cases where remote intervention was actually performed at the remote intervention timing (or a predetermined period including the remote intervention timing) predicted by the remote intervention timing prediction unit 14 in the past to "error" cases where remote intervention was not performed at the remote intervention timing (or the predetermined period including the remote intervention timing). For example, if the remote intervention timing prediction unit 14 has predicted the remote intervention timing for a certain reason 10 times in the past, and 8 of the 10 predictions were "correct" and the remaining 2 were "incorrect," the prediction reliability determiner 15 may determine the accuracy rate for the certain reason to be 80%. Note that the accuracy and error of the prediction of the remote intervention timing are examples of performance information.

[0127] The accuracy rate may be calculated from the ratio of a "correct answer" when a remote intervention was actually performed at the remote intervention point (or a predetermined range including the remote intervention point) predicted by the remote intervention timing prediction unit 14 in the past to an "incorrect answer" when no remote intervention was performed at the remote intervention point (or the predetermined range including the remote intervention point). For example, if the remote intervention timing prediction unit 14 has predicted the remote intervention point for a certain reason 10 times in the past, and 7 of the 10 predictions were "correct" and the remaining 3 were "incorrect," the prediction reliability determiner 15 may determine the accuracy rate for the certain reason to be 70%.

[0128] The prediction reliability determination unit 15 may group multiple vehicles M1, etc. based on at least one of the type of vehicle M1, environmental information related to the traveling of the vehicle M1, and time information related to the traveling time of the vehicle M1, and calculate a hit rate for calculating the reliability of the vehicle M1 based on performance information of one or more vehicles in the group to which the vehicle M1 belongs. The environmental information includes the area in which the vehicle M1 is traveling, the weather in the area in which the vehicle M1 is traveling, etc. The time information includes the time period and day of the week in which the vehicle M1 is traveling, etc.

[0129] It should be noted that there may be multiple predictions for one vehicle.

[0130] The prediction reliability determination unit 15 may determine the reliability based on the probability of occurrence of a certain reason instead of or in addition to the hit rate. For example, the prediction reliability determination unit 15 may determine a higher reliability value as the probability of occurrence of a certain reason increases. For example, if a remote intervention occurs at a certain point at time t0 due to a temporary event (e.g., a parking lot on the route) and a vehicle passes through the certain point at time t1 (time t1 > t0), remote intervention for the vehicle passing at time t1 becomes unnecessary once the temporary event that occurred at time t0 is resolved (e.g., the parking lot disappears), so the probability of occurrence of remote intervention can be considered to decrease over time from time t0. The occurrence probability that decreases over time may be estimated using a statistical method of performance information and used as the occurrence probability of a certain reason. Such an occurrence probability may become smaller as the time elapses from time t0.

[0131] In addition, the prediction reliability determination unit 15 may group multiple vehicles M1, etc. based on at least one of the type of vehicle M1, environmental information regarding the driving of vehicle M1, and time information regarding the time the vehicle M1 is driving, and calculate the occurrence probability for calculating the reliability of vehicle M1 based on the performance information of one or more vehicles in the group to which vehicle M1 belongs.

[0132] Next, the screen switching determination unit 17 determines whether or not there is one or more predictions with a reliability exceeding a threshold value among the reliability of each of the vehicles M1, etc. (S15). The threshold value is set in advance and stored in the storage unit 16.

[0133] Next, when the screen switching determination unit 17 determines that there is one or more predictions with reliability exceeding the threshold (Yes in S15), it determines to switch the display related to the vehicle from the overall management screen to an individual screen (S16). It can also be said that the screen switching determination unit 17 determines the display mode of information related to the vehicle to be the individual screen. Furthermore, when there are two or more predictions with reliability exceeding the threshold, the screen switching determination unit 17 may identify the vehicle with the closest remote intervention timing, i.e., the vehicle predicted to require remote intervention first, and determine to switch the display related to the identified vehicle from the overall management screen to an individual screen.

[0134] Furthermore, if the display of the vehicle that is the subject of the individual screen of the first display device 21 is already being performed, the screen switching determination unit 17 determines whether the newly identified vehicle matches the vehicle that is the subject of the individual screen, and if it determines that they do not match, switches to the individual screen of the newly identified vehicle. Note that the vehicle that is the subject of the individual screen may be, for example, a vehicle that is already being displayed on the individual screen of the first display device 21.

[0135] Here, the individual screen will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of status information displayed on the display device according to this embodiment. Fig. 6 shows an example of an individual screen displayed on the first display device 21. For convenience, the background portion of the individual screen is indicated by dots in Fig. 6. Fig. 6 describes a case where an operator O1 is in charge of remotely operating the vehicle M1.

[0136] As shown in FIG. 6 , for a prediction whose reliability is determined to exceed a threshold, an image captured by a camera mounted on the vehicle M1 corresponding to the prediction is displayed on the first display device 21 for confirmation by the operator O1. In the example of FIG. 6 , the individual screen is divided into a forward display area 21F displaying an image captured in front of the vehicle M1, a rear display area 21B displaying an image captured behind the vehicle M1, a right display area 21R displaying an image captured to the right of the vehicle M1, and a left display area 21L displaying an image captured to the left of the vehicle M1. In this manner, the individual screen displays images captured by the vehicle M1. Furthermore, information indicating reliability may be displayed on the individual screen as information indicating the reason why the vehicle M1 is displayed on the individual screen. In the example of FIG. 6 , "High Reliability" is displayed as the information indicating reliability.

[0137] This allows the operator O1 to check the status of vehicles for which the timing of remote intervention has been predicted and whose reliability exceeds a threshold, i.e., vehicles that are likely to actually require remote intervention, in a display format with higher information detail.

[0138] Note that the individual screen is not limited to displaying images captured in four directions, i.e., the front, rear, left, and right directions, of the vehicle M1. For example, the individual screen may be controlled so that the greater the reliability value, the more directions from which images are displayed.

[0139] 5 again, the screen switching determination unit 17 outputs control information to the operator terminal device D1 to allow the operator O1 to check the status of the vehicle M1 on the first display device 21 (S17). That is, the screen switching determination unit 17 switches the display mode of the display showing the status of the vehicle M1 that the operator O1 is in charge of, based on the determined reliability.

[0140] The screen switching determination unit 17 may prevent the operator O1 from being burdened with unnecessary tasks due to frequent changes in the vehicles displayed on the individual screens. For example, the screen switching determination unit 17 may display information (e.g., an image) for a certain period of time to confirm the vehicle determined to be switched. In other words, the screen switching determination unit 17 may prohibit the vehicle displayed on the individual screen from being switched to another vehicle within a certain period of time. The screen switching determination unit 17 may also determine whether to display an image based on history information on the determination of whether the reliability exceeds a threshold. For example, the screen switching determination unit 17 may execute steps S11 to S15 shown in FIG. 5 multiple times and determine that a vehicle for which a predetermined number of consecutive determinations in step S15 are "Yes" or a vehicle for which a determination in step S15 is made most frequently among the multiple determinations is a vehicle to be switched to the individual screen.

[0141] Furthermore, for example, in step S15, when there are two or more vehicles that have a reliability exceeding a threshold and will reach the timing for remote intervention within a predetermined time, the screen switching determination unit 17 may determine one of these vehicles to switch to an individual screen, taking into consideration the impact of a delay in remote intervention.It can also be said that the screen switching determination unit 17 determines the display mode of the status information of one vehicle identified based on the impact of a delay in starting remote operation among the two or more vehicles that have a reliability exceeding a threshold and will reach the timing for remote intervention within a predetermined time.

[0142] For example, the screen switching determination unit 17 may determine to switch to an individual screen a vehicle that has the greatest impact on safety risk among two or more vehicles. The safety risk may be a possibility that the vehicle will be involved in an accident due to a delay in remote intervention. For example, a case in which the vehicle is entering an intersection may be considered.

[0143] Furthermore, for example, the screen switching determination unit 17 may determine to switch to an individual screen the vehicle that has the greatest impact on the cost of recovery among two or more vehicles. The highest cost of recovery means that if remote intervention is delayed, the operator O1 will need to put in the most effort to resolve the problem. By having the operator O1 check the status of such a vehicle on the individual screen in advance, remote operation can be performed smoothly. For example, if the vehicle is approaching an obstacle, having the operator O1 check the status of the vehicle on the individual screen early on allows the vehicle to change lanes early to avoid the obstacle.

[0144] Furthermore, for example, the screen switching determination unit 17 may determine to switch the individual screen for the vehicle that has the greatest impact on losses. The vehicle that has the greatest loss means a vehicle that will have a longer delay in service such as delivery, or a vehicle that will have a greater opportunity loss due to vehicle delays. The vehicle that has the greatest loss can also be said to be a vehicle that has the least time to spare. This can prevent economic losses from occurring.

[0145] Furthermore, if the screen switching determination unit 17 determines that there is no prediction having a reliability exceeding the threshold (No in S15), the processing ends. In this case, the screen switching determination unit 17 may determine not to display on an individual screen in order to prevent unnecessary increase in the burden on the operator O1. For example, if the determination is No in step S15, the first display device 21 may not display an image for checking the status of the vehicle. Furthermore, the screen switching determination unit 17 may highlight the vehicle M1 corresponding to a prediction having a reliability equal to or lower than the threshold on the overall management screen.

[0146] The overall management screen will now be described with reference to FIG. 7 . FIG. 7 is a diagram showing another example of status information displayed on the display device according to the present embodiment. FIG. 7 shows an example of the overall management screen displayed on the second display device 22. Note that FIG. 7 illustrates a case in which operator O1 is assigned responsibility for four vehicles with vehicle IDs X11 to X14. Also, FIG. 7 displays a mark B11 indicating the location of a vehicle with vehicle ID "X11," a mark B12 indicating the location of a vehicle with vehicle ID "X12," a mark B13 indicating the location of a vehicle with vehicle ID "X13," and a mark B14 indicating the location of a vehicle with vehicle ID "X14." Furthermore, FIG. 7 also displays the travel route of each vehicle with a dashed line. Also, in FIG. 7 , the reliability threshold setting is described as "Reliability: X." For example, "High reliability X11 approximately 10 minutes later" indicates that the vehicle ID of the vehicle predicted to be the first to reach the timing for remote intervention (approximately 10 minutes later) among predictions with "High reliability" or higher is "X11." Also, "Medium reliability X13 approximately 5 minutes later" indicates that the vehicle ID of the vehicle predicted to be the first to reach the timing for remote intervention (approximately 5 minutes later) among predictions with "Medium reliability" or higher is "X13." The same applies to "Low reliability X12 30 seconds later."

[0147] As shown in Fig. 7, the second display device 22 displays status information for all four vehicles managed by the operator O1 on one screen. The second display device 22 displays an image as an overall management screen for checking the current positions, travel routes, travel conditions, etc. of the four vehicles from a bird's-eye view. Travel conditions include "traveling," "waiting for route setting," etc. "Waiting for route setting" means that the vehicle is stopped and a destination, etc. has not been set.

[0148] This allows the operator O1 to check the status of vehicles for which the timing of remote intervention has been predicted and the reliability is below a threshold, i.e., vehicles for which the possibility of actual remote intervention is low, in a display mode with a lower level of information detail. This reduces the burden and time on the operator O1 to check the status of vehicles for which the reliability is below a threshold, thereby reducing the unnecessary burden on the operator O1 in the event of a false detection compared to when the information is displayed on a separate screen.

[0149] The second display device 22 may highlight at least one of the vehicles scheduled for remote intervention relative to the other vehicles. In the example of FIG. 7 , the vehicle with vehicle ID “X12” that is predicted to be the first to be remotely intervened, with a prediction of “low reliability” or higher, among the vehicles scheduled for remote intervention, is the vehicle with vehicle ID “X12,” and this vehicle is highlighted with a thick line. Note that a vehicle with vehicle ID “X11” with “high reliability” or a vehicle with vehicle ID “X13” with “medium reliability” may be highlighted relative to the other vehicles. Vehicle ID “X11” is a vehicle predicted to be remotely intervened in 10 minutes, vehicle ID “X12” is a vehicle predicted to be remotely intervened in 30 seconds, and vehicle ID “X13” is a vehicle predicted to be remotely intervened in 5 minutes. Note that which vehicle is highlighted may be determined, for example, by input from the operator O1.

[0150] Furthermore, information indicating reliability may be displayed on the overall management screen as information indicating the reason why the vehicle is displayed on the overall management screen.

[0151] In this way, the server device 10 determines the display mode when the reliability exceeds the threshold so that the information is more detailed than when the reliability is equal to or less than the threshold. For example, when the reliability exceeds the threshold, the server device 10 causes the first display device 21 to display an image of the surroundings of the vehicle M1 acquired from the vehicle M1, and when the reliability is equal to or less than the threshold, the server device 10 causes the second display device 22 to display an image in which information about the vehicle M1 is superimposed on map information.

[0152] Note that the first display device 21 may display the individual screen of a vehicle with a vehicle ID of "X12" (a vehicle with "low reliability"), for example, when the first display device 21 is not displaying an individual screen or when there is sufficient time until the next remote intervention timing. Because remote intervention may occur even in a vehicle with "low reliability," even in such a scenario, it is expected that the operator O1 can smoothly perform remote intervention on the vehicle with "low reliability" and resume service. Furthermore, for example, if the operator O1 determines that there is sufficient time until the next remote intervention timing for a vehicle with "high or medium reliability," and then checks the individual screen of the vehicle with "low reliability," the operator O1 checks the individual screen after understanding that there is sufficient time until the next remote intervention timing for the vehicle with "high or medium reliability," thereby preventing a loss of system reliability.

[0153] On the other hand, monitoring the individual screens of vehicles with "low reliability" still increases the burden on the operator O1. In contrast, the decision of whether to monitor the individual screens of vehicles with "low reliability" may be left to the operator O1, and an incentive may be provided, such as paying the operator O1 a separate reward if the operator O1 smoothly intervenes remotely and successfully resumes service. For example, when the display generation unit 32 receives information from the operator O1 via the communication unit 31 indicating that the operator O1 will monitor the individual screens of vehicles with "low reliability," the display generation unit 32 may cause the first display device 21 to display the individual screens of the vehicles with "low reliability."

[0154] Note that the display modes with a high level of information detail and the display modes with a low level of information detail are not limited to the display modes shown in Figures 6 and 7. A display mode with a low level of information detail may display images with a lower level of information detail than a display mode with a high level of information detail, for example, a display mode that displays fewer images than a display mode with a high level of information detail. For example, a display mode with a high level of information detail may display four images (front, rear, left, and right), while a display mode with a low level of information detail may display three or fewer images among the front, rear, left, and right. Furthermore, for example, a display mode with a high level of information detail may display the position, driving route, and status of the vehicle on a map, while a display mode with a low level of information detail may display only the vehicle position on the map.

[0155] (Variations of the Embodiments) Variations of the embodiments will be described below with reference to FIGS. 8 to 17. The following description will focus on differences from the embodiment, and descriptions of content that is the same as or similar to the embodiment will be omitted or simplified. The configuration of the information processing system of each variation may be the same as the information processing system 1 of the embodiment, and descriptions thereof will be omitted. Furthermore, the following description will use the reference numerals of the information processing system 1 of the embodiment.

[0156] (First Modification of the Embodiment) Hereinafter, an information processing system according to this modification will be described with reference to FIG.

[0157] FIG. 8 is a flowchart showing the operation (information processing method) of the information processing system 1 according to this modified example.

[0158] 8 , when the screen switching determination unit 17 determines that there is no prediction having a reliability exceeding the threshold (No in S15), it further determines whether a threshold change instruction has been received (S18a). The threshold change instruction is an instruction to change the threshold used in the determination in step S15. If the result in step S15 is No, the screen switching determination unit 17 may display information on one of the first display device 21 and the second display device 22 to prompt the operator O1 to select whether to change (e.g., lower) the reliability threshold, and may acquire the selection of the operator O1 received by the input device as a threshold change instruction.

[0159] If the screen switching judgment unit 17 determines that a threshold change instruction has been received (Yes in S18a), it updates the threshold stored in the memory unit 16 (S18b), and if it determines that a threshold change instruction has not been received (No in S18a), it terminates the processing.

[0160] The screen switching determination unit 17 may change the threshold value according to the situation of the operator O1. For example, the screen switching determination unit 17 may cause one of the first display device 21 and the second display device 22 to display a message that allows the operator O1 to actively change the threshold value at any timing. The screen switching determination unit 17 may change the threshold value according to an input from the operator O1.

[0161] (Second Modification of the Embodiment) Hereinafter, an information processing system according to this modification will be described with reference to FIGS.

[0162] FIG. 9 is a flowchart showing the operation (information processing method) of the information processing system 1 according to this modified example.

[0163] As shown in FIG. 9 , when the screen switching determination unit 17 determines that there is one or more predictions with reliability exceeding the threshold (Yes in S15), it determines to switch the display related to the vehicle from the overall management screen to a split display screen (see FIG. 10 described later) (S19). The split display screen is a screen for simultaneously displaying a predetermined number of vehicles with reliability exceeding the threshold on the same screen, and is divided into a predetermined number of split display areas. For example, one split display area displays a display for checking the status of one vehicle. In other words, the split display screen allows an operator O1 or the like to simultaneously check the status of a predetermined number of vehicles in detail. Furthermore, because the reliability of the predetermined number of vehicles is higher than the threshold, checking the status is unlikely to be a waste of time.

[0164] When the screen switching determination unit 17 determines that there are two or more predictions with reliability exceeding a threshold, the screen switching determination unit 17 may select a candidate vehicle for which remote intervention timing will occur within a predetermined period (for example, the first one). Then, if the number of selected vehicles is equal to or less than the number that can be displayed in a split screen, the screen switching determination unit 17 may cause the operator terminal device D1 to generate a split display screen for those vehicles. For example, the screen switching determination unit 17 may determine the screen layout of multiple vehicles so that they are arranged in the order of the remote intervention timing.

[0165] Furthermore, when the first display device 21 is currently displaying the status of one or more vehicles on an individual screen or a split display screen and it is newly determined that there is one prediction with a reliability exceeding a threshold, the screen switching determination unit 17 may display all of those vehicles on the split display screen. In this case, the screen switching determination unit 17 may highlight the area displaying the status of the vehicle newly added to the split display screen more than the areas displaying the status of the other vehicles.

[0166] Furthermore, when the screen switching determination unit 17 determines that there is one prediction with a reliability exceeding the threshold, it may display the single prediction on a split display screen (i.e., a screen with one division) for displaying only that single prediction. The split display screen with one division may be the same as an individual screen.

[0167] In addition, if the screen switching judgment unit 17 determines that there is no prediction with a reliability exceeding the threshold (No in S15), that is, if the reliability of each vehicle M1, etc. is below the threshold, it may present information to the operator O1, etc. to change the threshold.

[0168] Here, the split display screen will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of status information displayed on the display device according to this embodiment. Fig. 10 shows an example of the split display screen displayed on the first display device 21. Note that Fig. 10 describes a case where the statuses of four vehicles out of a plurality of vehicles M1 etc. that are in charge of an operator O1 are simultaneously displayed on the same screen.

[0169] As shown in Fig. 10 , for four predictions whose reliability is determined to exceed the threshold, images captured by cameras mounted on the four vehicles corresponding to the predictions are displayed on the first display device 21 of the operator O1. In the example of Fig. 10 , the split display screen is divided into four areas separated by dashed dotted lines. In this case, "4" is an example of the predetermined number (number of divisions).

[0170] The first divided display area (upper left area) is divided into a forward display area 21F1 that displays images captured in front of the first vehicle, a rear display area 21B1 that displays images captured behind the first vehicle, a right display area 21R1 that displays images captured to the right of the first vehicle, and a left display area 21L1 that displays images captured to the left of the first vehicle.

[0171] Similarly, the second divided display area (lower left area), the third divided display area (upper right area), and the fourth divided display area (lower right area) are divided into forward display areas 21F2 to 21F4, rear display areas 21B2 to 21B4, right display areas 21R2 to 21R4, and left display areas 21L2 to 21L4, which display images captured in each direction of the second to fourth vehicles.

[0172] In this way, on the split display screen, images captured by a plurality of vehicles are displayed side by side at the same time in order to check the conditions of the plurality of vehicles.

[0173] The number of divisions in the split display screen (i.e., the number of vehicles that can be displayed) may be determined based on the type of remote operation required of the operator. The number of divisions may be changed, for example, based on the difficulty of remote monitoring, remote operation, etc. The number of divisions may be determined based on a table in which the type of remote operation and the number of divisions are associated. For example, the number of divisions may be reduced as the difficulty of remote monitoring, remote operation, etc. increases, and increased as the difficulty decreases. The number of divisions corresponds to the above-mentioned predetermined number.

[0174] Next, an operation for selecting a vehicle to be displayed on the split display screen will be described with reference to Figs. 11 and 12. Fig. 11 is a flowchart showing an example of an operation (information processing method) for selecting a vehicle to be displayed on the split display screen according to this modified example. Fig. 11 describes an example in which a plurality of vehicles whose remote intervention timing is close are selected as vehicles to be displayed on the split display screen. Note that the operation shown in Fig. 11 may be executed, for example, after a Yes determination is made in step S15.

[0175] 11 , the screen switching determination unit 17 sorts the vehicles in descending order of the timing of remote intervention (S21), and selects the first vehicle in the sorting as a candidate for switching (S22). The selected vehicle may be excluded from the sorting. In other words, the sorting may be updated by deleting the vehicle selected as a candidate for switching.

[0176] Next, the screen switching determination unit 17 sets the next remote intervention timing time of the vehicle selected as a candidate as a reference value (S23), and determines whether the remote intervention timing of the first vehicle in the sorting is different from the reference value by a certain amount or more (S24). The first vehicle in step S24 means the first vehicle in the sorting from which the vehicle selected in step S22 has been removed.

[0177] Next, if the screen switching determination unit 17 determines that the remote intervention timing of the first vehicle in the sorting is different from the reference value by a certain amount or more (Yes in S24), it determines whether the number of candidates exceeds a certain number (S25), and if it determines that the number of candidates exceeds the certain number (Yes in S25), it ends the processing. As a result, a certain number of vehicles with close remote intervention timings are selected as vehicles to be displayed on the split display screen.

[0178] Furthermore, if the screen switching judgment unit 17 determines that the remote intervention timing of the first vehicle in the sorting does not differ from the reference value by a certain amount (No in S24), or if it determines that the number of candidates does not exceed a certain number (No in S25), it returns to step S21 and continues processing.

[0179] This allows vehicles with an upcoming remote intervention timing to be selected as vehicles to be displayed on the split display screen. For example, it is possible to prevent vehicles with high reliability but different remote intervention timing from being displayed on the split display screen, so that the operator O1 or the like does not have to check vehicles that do not need to be checked in the near future, and it is possible to prevent unnecessary burden on the operator O1 or the like.

[0180] The process of step S25 does not have to be executed. This allows all vehicles whose remote intervention timing is approaching to be selected as candidates for vehicles to be displayed on the split display screen.

[0181] Fig. 12 is a flowchart showing another example of the operation (information processing method) of selecting vehicles to be displayed on the split display screen according to this modified example. Fig. 12 illustrates an example of selecting vehicles by limiting the number of candidates, such as the number of vehicles that can be displayed on the split display. Note that the operation shown in Fig. 12 may be executed after determining "Yes" in step S15, for example. Note that steps S31 and S32 are the same processes as steps S21 and S22 shown in Fig. 11, and therefore will not be described here.

[0182] 12 , when the screen switching determination unit 17 selects the first vehicle in the sorting as a candidate for switching (S32), it determines whether the number of candidates has reached a certain number (S33). If it determines that the number of candidates has reached the certain number (Yes in S33), it terminates the processing. If the screen switching determination unit 17 determines that the number of candidates has not reached the certain number (No in S33), it sets the time of the next remote intervention timing for the vehicle selected as a candidate as a reference value (S34) and determines whether the remote intervention timing of the first vehicle in the sorting differs from the reference value by a certain amount or more (S35). If the screen switching determination unit 17 determines that the difference is a certain amount or more (Yes in S35), it terminates the processing. If it determines that the difference is not a certain amount or more (No in S35), it returns to step S32 and continues the processing.

[0183] This allows a limited number of vehicles to be selected for display on the split display screen. For example, only the statuses of a limited (determined) number of vehicles depending on the skill, fatigue level, etc. of the operator O1, etc. are displayed on the split display screen, so that excessive burden on the operator O1, etc. can be prevented by displaying more vehicles than the number appropriate for the operator O1, etc. The skill, fatigue level, etc. of the operator O1, etc. may be included in the operator information.

[0184] When the time comes to perform remote intervention, the first display device 21 may be switched from the split display screen to a screen (e.g., an individual screen) for checking the status of only the vehicle for which remote intervention is to be performed.

[0185] (Third Modification of the Embodiment) Hereinafter, an information processing system according to this modification will be described with reference to FIG.

[0186] Fig. 13 is a diagram for explaining the timing of switching the display screen according to this modification. Fig. 13 shows the timing at which the display screen of the operator O1 actually switches when, for example, the screen switching determination unit 17 determines to switch to an individual screen in step S16 of Fig. 8. Position p11 indicates the current vehicle position, position p13 indicates the predicted position of the remote intervention point, and position p12 indicates the screen switching position.

[0187] As shown in FIG. 13 , the terminal control unit 33 according to this modification controls screen switching (e.g., switching to an individual screen) of the first display device 21 being viewed by the operator O1 when the vehicle reaches a location (position p12) a predetermined distance before the location (position p13) where the remote intervention request is predicted. The predetermined distance corresponds to, but is not limited to, a distance corresponding to the start point of advance preparation for remote intervention by the operator O1. The predetermined distance is set in advance. Furthermore, the predetermined distance may be a common value for multiple operators O1, or may be a distance that differs from one another. The predetermined distance may be set, for example, based on the time required for the operator O1 to grasp the vehicle's situation in advance.

[0188] For example, if the screen switching of the first display device 21 is too fast, the operator O1 will have to watch the screen for a long time, which increases the burden of monitoring on the operator O1. Also, if the screen switching is too slow, the operator O1 may not be able to fully grasp the vehicle's condition in advance, which may hinder smooth remote control.

[0189] 13, the screen display is switched during the period required for advance preparation for remote intervention. This reduces the time that the operator O1 needs to gaze at the screen of the first display device 21, thereby reducing the burden on the operator of monitoring O1 and enabling smoother remote control.

[0190] In addition, the terminal control unit 33 may control the screen switching of the first display device 21 that the operator O1 is viewing when the time reaches a predetermined time before the predicted time of arrival at the predicted remote intervention point.

[0191] (Fourth Modification of the Embodiment) Hereinafter, an information processing system according to this modification will be described with reference to FIGS. 14A and 14B.

[0192] In the embodiment shown in FIG. 7, an example in which the reliability thresholds are "high", "medium", and "low" has been described, but in this modification, other examples of the reliability thresholds will be described.

[0193] Fig. 14A is a diagram showing an example of status information displayed on a display device (here, the second display device 22) according to this modification. Fig. 14A shows an example of an overall management screen displayed on the second display device 22. Figs. 14A and 14B illustrate a case in which operator O1 is assigned to be in charge of four vehicles with vehicle IDs X11 to X14, as in the case of Fig. 7 .

[0194] 14A and 14B, the setting of the reliability threshold is described as "reliability: numerical value." For example, "reliability: 1.0 X11 approximately 10 minutes later" indicates that the vehicle ID of the vehicle predicted to be the first to reach the timing for remote intervention (approximately 10 minutes later) among predictions with a "reliability: 1.0" or higher is "X11." Also, "reliability: 0.9 X11 approximately 10 minutes later" indicates that the vehicle ID of the vehicle predicted to be the first to reach the timing for remote intervention (approximately 10 minutes later) among predictions with a "reliability: 0.9" or higher is "X11." The same applies to "reliability: 0.8 X11 7 minutes later," etc.

[0195] As shown in Fig. 14A, the reliability threshold may be indicated by a numerical value. Fig. 14A shows an example in which the threshold has 10 levels (e.g., step size of 0.1) with a maximum value of 1.0 and a minimum value of 0.0, but the maximum and minimum values ​​of the threshold, the number of threshold levels, and the step size are not limited to this. Also, Fig. 14A shows a case in which some thresholds (reliability: 0.5 or less) cannot be displayed due to the large number of threshold options.

[0196] FIG. 14B is a diagram showing an example of a display in which one or a preset number of items of status information shown in FIG. 14A are left and the display of the other remote intervention timings is omitted.

[0197] In FIG. 14A , "Reliability: 1.0 X11, approximately 10 minutes later" and "Reliability: 0.9 X11, approximately 10 minutes later" show an example in which the remote intervention timing remains the same even when the threshold is changed. In this case, as shown in FIG. 14B , the display of the remote intervention timing may be omitted except for one or a preset number of remote intervention timings. In the example of FIG. 14B , the display of "Reliability: 0.9 X11, approximately 10 minutes later" (the display with the lower reliability) is omitted.

[0198] If there are many options for the threshold, the display may be controlled so that other thresholds can be selected by scrolling. The display control is executed by the terminal control unit 33.

[0199] (Fifth Modification of the Embodiment) Hereinafter, an information processing system according to this modification will be described with reference to FIGS.

[0200] In this modification, another example of the screen displayed on the first display device 21 will be described.

[0201] 15 to 17 are diagrams showing examples of status information displayed on a display device (here, the first display device 21) according to this modified example. 15 and 16 show display examples in which the number of vehicles that the operator O1 needs to check is three. The display on the first display device 21 is executed by the terminal control unit 33.

[0202] Since the prediction result of the next vehicle to require remote intervention may differ depending on the reliability threshold, as shown in Figure 15, the first display device 21 may display a split screen consisting of individual screens of vehicles predicted to require remote intervention next for each threshold. In the example of Figure 15, the first display device 21 displays a split screen (three split screens) when the earliest timing for remote intervention among predictions with a reliability threshold of "low" or higher is 30 seconds after vehicle X12, the earliest timing for remote intervention among predictions with a reliability threshold of "medium" or higher is 5 minutes after vehicle X13, and the earliest timing for remote intervention among predictions with a reliability threshold of "high" or higher is 10 minutes after vehicle X11. Note that the number of split screens is not limited to three and may be two or more.

[0203] Furthermore, if the number of vehicles that one operator O1 needs to check is not enormous (for example, if the number is less than a threshold number set for that operator O1), the first display device 21 may always display a split screen consisting of individual screens for all vehicles X11 to X13, as shown in Figure 16.

[0204] Furthermore, the first display device 21 may further display the timing of remote intervention for each reliability threshold (the hatched square frame in FIG. 16 ) in a Gantt chart. In the Gantt chart, the vertical axis indicates time (remote intervention timing) and the horizontal axis indicates the reliability threshold. Note that the vertical axis may not be displayed at equal intervals of one second, but may instead be displayed with the time width exponentially decaying with elapsed time. This allows the operator O1 to observe the state (remote intervention status) in the more distant future.

[0205] Furthermore, by displaying the Gantt chart, the operator O1 can intuitively grasp which vehicle will be next at the timing for remote intervention according to the threshold value.

[0206] Note that the first display device 21 may move the rectangular frame in a direction in which the time decreases (upward in the example of FIG. 16 ) as time passes. For example, the rectangular frame may be moved upward in accordance with a predetermined time interval (a predetermined rhythm).

[0207] Furthermore, the first display device 21 may duplicate the same map as the number of threshold variations (three: "High," "Medium," and "Low" in the example of FIG. 17 ), and display the vehicle icons color-coded according to the duration until the predicted remote intervention timing as an overall management screen. The map indicates the locations of the vehicles. Note that in FIG. 17 , different colors are indicated by the shape of the frames. In FIG. 17 , red icons of vehicles whose predicted remote intervention timing is within 30 seconds are indicated by oval frames, orange icons of vehicles whose predicted remote intervention timing is between 30 seconds and 5 minutes are indicated by square frames, green icons of vehicles whose predicted remote intervention timing is between 5 minutes and 10 minutes are indicated by diamond frames, and gray icons of vehicles whose predicted remote intervention timing is other than those (e.g., longer than 10 minutes) are indicated by trapezoidal frames. Note that the color coding based on the remote intervention timing and the time range of the remote intervention timing for which different colors are used are merely examples and are not limited to those shown in FIG. 17 .

[0208] As shown in Figure 17, the first display device 21 may display the same map for each of the reliability levels: "high," "medium," and "low," and may display different vehicle icons depending on the time until the remote intervention timing according to the reliability level.

[0209] 17, operator O1 can first check the map for "high reliability" and see that there is only one vehicle (vehicle X11) that will require remote intervention within 10 minutes. As a result, operator O1 can determine that there is sufficient time and consider the timing of remote intervention for lower reliability levels. In this way, displaying the screen shown in FIG. 17 allows operator O1 to perform remote intervention effectively.

[0210] (Variation 6 of the embodiment) In this variation, an example will be described in which an expected value is calculated using the reliability of the prediction described above and the impact on at least one of vehicle safety and service efficiency, and the calculated expected value is used as the reliability to be compared with a threshold. The impact indicates the degree of impact on safety and efficiency when the reliability prediction is incorrect, and is assigned a value (priority) of 1 to N (N: natural number) depending on the vehicle situation (e.g., the state of the vehicle's surroundings), for example. For example, an impact may be set in advance for each vehicle depending on the vehicle situation.

[0211] The priority of safety may be determined based on actual performance from a first perspective, such as whether an accident or incident actually occurs due to an operational delay under the circumstances. The priority of efficiency may be determined based on actual performance from a second perspective, such as how much loss an operational delay in the circumstances actually contributes to, for example, the increase in the time it takes for a food delivery service to arrive after a customer places an order. If prioritization cannot be performed on the same level due to differences in perspectives, a multi-level priority system may be designed by assigning a priority to each perspective and then assigning priorities between perspectives. The priority between perspectives is a degree indicating which of multiple perspectives, including the first and second perspectives, is given priority.

[0212] The prediction reliability determination unit 15 defines the reliability of the prediction as a probability between 0 and 1, assigns priorities of safety and efficiency according to the vehicle situation as influences, and calculates an expected value based on the probability and the influences. The expected value may be calculated, for example, by calculating (e.g., multiplying) the probability and the influences. The prediction reliability determination unit 15 may then determine the calculated expected value as the reliability to be compared with a threshold value.

[0213] The prediction reliability determination unit 15 may also predict which of a plurality of situations corresponds to safety and efficiency. In this case, the prediction reliability determination unit 15 may also calculate the impact itself using an expected value.

[0214] (Other embodiments) The present disclosure has been described above based on the embodiments and various modified examples (hereinafter also referred to as embodiments, etc.), but the present disclosure is not limited to the above-mentioned embodiments, etc.

[0215] For example, in the above-described embodiments, the prediction reliability determination unit 15 may determine the reliability of the prediction of whether remote intervention is necessary based on the reason for predicting that remote operation of the vehicle M1, etc. is necessary.

[0216] Furthermore, in the above embodiments, a vehicle is used as an example of a moving body, but the moving body may be any object that can switch between autonomous driving and remotely controlled driving, and may be, for example, an air vehicle (e.g., a drone).

[0217] The threshold for the reliability in the above-described embodiments may be determined as appropriate. For example, if the threshold is set to a value representing the case where remote intervention is actually required only once out of four predicted remote interventions, the threshold may be set to 25%. The threshold may also be automatically updated when performance information (e.g., accuracy rate) is updated. The threshold may be set to a single value common to all reasons for the prediction that remote operation will be required, or may be set for each reason.

[0218] In the above embodiment, if there are multiple predictions that exceed the threshold, the prediction with the highest priority may be identified based on a predetermined priority order, and an image of the vehicle corresponding to the identified prediction may be displayed on a separate screen. The priority order may be set, for example, according to the reason why remote control is predicted to be necessary.

[0219] Furthermore, on each of the individual screens, the overall management screen, and the split display screens in the above-described embodiments, an explanation of the predicted remote intervention and the contents to be confirmed may be notified to the operator by text or voice.

[0220] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0221] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Some of the steps may be executed simultaneously (in parallel) with other steps, or some of the steps may not be executed.

[0222] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0223] Furthermore, the server device according to the above-described embodiments may be realized as a single device or may be realized by multiple devices. When the server device is realized by multiple devices, the components of the server device may be distributed among the multiple devices in any manner. When the server device is realized by multiple devices, the communication method between the multiple devices is not particularly limited and may be wireless communication or wired communication. Furthermore, wireless communication and wired communication may be combined between the devices. Furthermore, at least one function of the server device may be realized by an operator terminal device or an in-vehicle device.

[0224] Furthermore, each component described in the above embodiments may be implemented as software or, typically, as an LSI, which is an integrated circuit. These components may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. Here, the term "LSI" is used, but depending on the level of integration, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit (a general-purpose circuit that executes a dedicated program) or a general-purpose processor. A field programmable gate array (FPGA), which can be programmed after LSI fabrication, or a reconfigurable processor, which can reconfigure the connections or settings of circuit cells within an LSI, may also be used. Furthermore, if an integrated circuit implementation technology that replaces LSI emerges due to advances in semiconductor technology or a derivative technology, that technology may naturally be used to integrate the components.

[0225] A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple processing units on a single chip. Specifically, it is a computer system that includes a microprocessor, ROM (Read Only Memory), RAM (Random Access Memory), etc. Computer programs are stored in the ROM. The system LSI achieves its functions when the microprocessor operates in accordance with the computer program.

[0226] Furthermore, one aspect of the present disclosure may be a computer program that causes a computer to execute each of the characteristic steps included in the information processing method shown in any of Figures 5, 8, 9, 11, and 12.

[0227] Furthermore, for example, the program may be a program to be executed by a computer. Another aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the program may be executed by the processor, thereby causing the device to perform each of the above processes.

[0228] The present disclosure is applicable to an information processing method for an operator to remotely control a mobile object.

[0229] REFERENCE SIGNS LIST 1 Information processing system 10 Server device 11, 41 Communication unit 12 Vehicle information acquisition unit 13 Operator information acquisition unit 14 Remote intervention timing prediction unit 15 Prediction reliability determination unit 16 Storage unit 17 Screen switching determination unit 21 First display device 21B, 21B1, 21B2, 21B3, 21B4 Rear display area 21F, 21F1, 21F2, 21F3, 21F4 Front display area 21L, 21L1, 21L2, 21L3, 21L4 Left display area 21R, 21R1, 21R2, 21R3, 21R4 Right display area 22 Second display device 31 Communication unit (acquisition unit) 32 Display generation unit 33 Terminal control unit (control unit) 34 Operator information generation unit 40 In-vehicle device 42 Destination acquisition unit 43 Detection unit 44 Self-position estimation unit 45 Determination unit 46 Remote operation information acquisition unit 47 Vehicle control unit 48 Vehicle information generation unit 50 Service system D1, Dn Operator terminal device (terminal device) M1, Mm Vehicle (mobile body) N Network O1, On Operator p11, p12, p13 Position

Claims

1. An information processing method executed by a computer, comprising: acquiring location information indicating the current location of a moving object, and route information indicating the path of movement of the moving object; acquiring road information; predicting at least one of a point and a timing at which an operator capable of remotely controlling the moving object will begin remote control of the moving object based on the location information, the route information, and the road information; determining a reliability of the prediction of at least one of the point and the timing; and determining a display manner of status information indicating the status of the moving object to be displayed to the operator before the start of the remote control based on the determined reliability.

2. The information processing method according to claim 1, wherein the reliability is determined based on accuracy of the prediction regarding the remote operation.

3. The information processing method according to claim 2, wherein the reliability is determined based on performance information including predictions of past remote control operations for the same reasons as those for which the need for remote control of the moving object was predicted, and the performance of the remote control operations in response to those predictions.

4. The information processing method according to claim 3, wherein the performance information includes information relating to a hit rate of the prediction, and the reliability is determined based on the hit rate.

5. The information processing method according to claim 4, wherein there are a plurality of the moving bodies, the plurality of moving bodies are grouped into one or more groups based on at least one of the type of the moving body, environmental information relating to the movement of the moving bodies, and time information relating to the time when the moving bodies move, and the accuracy rate for calculating the reliability of the moving body is calculated based on the performance information of one or more moving bodies of the group to which the moving body belongs.

6. The information processing method according to claim 1, further comprising determining the reliability based on the occurrence probability of the reason for predicting the need for remote control of the mobile object.

7. The information processing method according to claim 6, wherein there are a plurality of the moving bodies, the plurality of moving bodies are grouped into one or more groups based on at least one of the type of the moving body, environmental information relating to the movement of the moving bodies, and time information relating to the time when the moving bodies move, and the occurrence probability for calculating the reliability of the moving body is calculated based on performance information of one or more moving bodies of the group to which the moving body belongs, and the performance information includes information relating to the occurrence probability.

8. An information processing method according to any one of claims 1 to 7, comprising determining whether the reliability exceeds a threshold, and if the reliability exceeds the threshold, determining a display mode having a higher level of information detail than if the reliability is equal to or lower than the threshold.

9. The information processing method according to claim 8, further comprising the steps of: displaying an image of the surroundings of the moving body acquired from the moving body when the reliability exceeds the threshold; and displaying information of the moving body on map information when the reliability is equal to or less than the threshold.

10. The information processing method according to claim 9, further comprising displaying the image for a certain period of time if the reliability exceeds the threshold.

11. The information processing method according to claim 9, further comprising the step of determining whether or not to display the image based on history information of the determination as to whether or not the reliability exceeds the threshold value.

12. The information processing method according to claim 8, further comprising determining the reliability of each of the plurality of moving bodies for that moving body, and, if there are two or more moving bodies that exceed the threshold, determining a display manner of status information indicating the status of one of the two or more moving bodies that is identified based on the impact of a delay in the start of the remote control.

13. An information processing method according to any one of claims 1 to 7, further comprising: determining the reliability of each of a plurality of said moving bodies; and displaying status information of the predetermined number of said moving bodies in a predetermined number of divided display areas into which a display screen of a display device is divided, based on the reliability of each of the plurality of said moving bodies.

14. The information processing method according to claim 13, further comprising highlighting a split display area in which status information of a new moving object is displayed, out of the predetermined number of split display areas.

15. The information processing method according to claim 13, wherein the predetermined number is determined based on the content of the remote operation required of the operator.

16. An information processing method according to any one of claims 1 to 7, further comprising: determining the reliability of each of a plurality of said moving bodies; and, if the reliability of each of the plurality of said moving bodies is equal to or lower than a threshold, presenting information for changing the threshold to the operator.

17. An information processing method according to any one of claims 1 to 7, wherein the road information includes section information indicating at least one of a section where autonomous driving is prohibited and a section where autonomous driving cannot be continued, and at least one of the location and the timing is predicted based on the section information.

18. A terminal device comprising: an acquisition unit that acquires information indicating a display mode for displaying the status information of the moving object determined by the information processing method described in any one of claims 1 to 7; and a control unit that displays the status information in the acquired display mode on a display device of an operator in charge of the moving object.

19. A program for causing a computer to execute the information processing method according to any one of claims 1 to 7.

Images