Information processing method and information processing apparatus

Abstract

Provided is an information processing method and an information processing device. The information processing device includes: an acquisition unit that obtains sensor data from a sensor of a vehicle; a detection processor that performs object detection processing for each vehicle using the sensor data; a decision unit that decides a form of a prompt using a result of the object detection processing; and a prompt controller that controls a prompt by a prompt device in accordance with the decided form, wherein, in the object detection processing, an object outside a vehicle in motion and a position of the object are detected, the decision unit calculates a degree of danger outside the vehicle for each of the vehicles using the detected object and the position of the object, a priority of the vehicle is decided in accordance with the degree of danger outside the vehicle, the form of the prompt to an operator is decided in accordance with the decided priority, and the form is a form in which a vehicle for which the calculated degree of danger is higher is given priority to be monitored.

Description

[0001] This application is a divisional application of the patent application with the Chinese patent application number 201910638562.1, the title of "Information processing method and information processing device", which was filed on 2019 / 07 / 16. TECHNICAL FIELD

[0002] The present disclosure relates to an information processing method and an information processing device. BACKGROUND

[0003] In recent years, technologies for making vehicles travel in an autonomous driving manner have appeared, and research and development have become more active. In the future, along with the development of autonomous driving technologies, it is estimated that unmanned autonomous vehicles will become widespread. In the operation of unmanned autonomous vehicles, it is required from the viewpoint of safety that an operator at a remote location monitor the vehicles. And there have been some proposals for technologies related to vehicle monitoring.

[0004] For example, in patent literature 1, a technology is disclosed in which, in the case where a vehicle enters a remote operation target area, a remote operation request is made to a control center, and the control center decides a remote operation device that remotely operates the vehicle that made the remote operation request.

[0005] In addition, in patent literature 2, a technology is disclosed in which, according to the number of passengers who use a station, the departure time of a vehicle is modified.

[0006] (Prior art documents)

[0007] (Patent literature)

[0008] Patent literature 1: Japanese Patent Application Publication No. 2017-147626

[0009] Patent literature 2: Japanese Patent Application Publication No. 2015-182623

[0010] However, in the prior art disclosed in patent literature 1, an operator simultaneously takes a plurality of vehicles as remote operation targets in the case where the plurality of vehicles enter a remote operation target area. However, it is difficult for one operator to remotely operate a plurality of vehicles at the same time.

[0011] In addition, the prior art disclosed in patent literature 2 is a technology for an operator who rides in a single vehicle, and it is not within the scope of consideration that one operator simultaneously manages a plurality of vehicles. SUMMARY

[0012] Therefore, the present disclosure provides an information processing method and an information processing device that, by monitoring or controlling a plurality of vehicles, can both ensure the safety of the plurality of vehicles and reduce the burden on an operator who performs the monitoring or control.

[0013] One aspect of the present disclosure relates to an information processing method in which a sensor data obtained from a vehicle is obtained by an information processing apparatus, the sensor data being data obtained from a sensor of the vehicle; an object detection process is executed for each vehicle by the information processing apparatus using each of the sensor data; a form of a presentation for an operator to monitor or control the vehicle is decided by the information processing apparatus using a result of the object detection process; and the presentation by a presentation apparatus is controlled in the decided form; wherein, in each of the object detection processes for each of the vehicles, an object outside a vehicle in travel and a position of the object are detected, in the decision of the form of the presentation, the information processing apparatus calculates a degree of danger outside the vehicle for each of the vehicles using the detected object and the position of the object, the information processing apparatus decides a priority of the vehicle to preferentially present to the operator according to the degree of danger outside the vehicle calculated for each vehicle, and the form of the presentation to the operator is decided according to the decided priority, the form being a form in which a vehicle with a higher calculated degree of danger among the vehicles is presented as a vehicle to be preferentially monitored.

[0014] Further, one aspect of the present disclosure relates to an information processing apparatus including an acquisition unit that obtains sensor data from a vehicle, the sensor data being data obtained from a sensor of the vehicle; an detection processor that executes an object detection process for each vehicle using each of the sensor data; a decision unit that decides a form of a presentation for an operator to monitor or control the vehicle using a result of the object detection process; and a presentation controller that controls the presentation by a presentation apparatus in the decided form, wherein, in each of the object detection processes for each of the vehicles, an object outside a vehicle in travel and a position of the object are detected, the decision unit calculates a degree of danger outside the vehicle for each of the vehicles using the detected object and the position of the object, the decision unit decides a priority of the vehicle to preferentially present to the operator according to the degree of danger outside the vehicle calculated for each vehicle, and the decision unit decides the form of the presentation to the operator according to the decided priority, the form being a form in which a vehicle with a higher calculated degree of danger among the vehicles is presented as a vehicle to be preferentially monitored.

[0015] Further, one aspect of the present disclosure relates to an information processing method of obtaining sensing data from a plurality of vehicles, the sensing data being data obtained from sensors each of the plurality of vehicles has, performing each of object detection processing using each of the sensing data, deciding a form of presentation for monitoring or controlling the plurality of vehicles using each result of the object detection processing, and controlling the presentation to a presentation device in the decided form.

[0016] In addition, these general or specific aspects can be implemented by a system, a method, an integrated circuit, a computer program, or a recording medium such as a CD-ROM storing the computer program, or any combination of the system, the method, the integrated circuit, the computer program, and the recording medium.

[0017] By the information processing method and the like provided by the present disclosure, the plurality of vehicles are monitored or controlled, so that the safety of the plurality of vehicles can be ensured and the burden on an operator who performs the monitoring or the control can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a block diagram illustrating an information processing system according to Embodiment 1.

[0019] Figure 2A is a diagram illustrating priority information according to Embodiment 1.

[0020] Figure 2B is a diagram illustrating a database according to Embodiment 1.

[0021] Figure 2C is a diagram illustrating a database according to Embodiment 1.

[0022] Figure 3 is a sequence diagram illustrating an operation of an information processing system according to Embodiment 1.

[0023] Figure 4A is a diagram illustrating a presentation of a terminal device according to Embodiment 1.

[0024] Figure 4B is another diagram illustrating a presentation of a terminal device according to Embodiment 1.

[0025] Figure 4C is another diagram illustrating a presentation of a terminal device according to Embodiment 1.

[0026] Figure 4D is another diagram illustrating a presentation of a terminal device according to Embodiment 1.

[0027] Figure 4E is another diagram illustrating a presentation of a terminal device according to Embodiment 1.

[0028] Figure 5 is a flowchart illustrating a process of deciding priority in the action of the information processing apparatus according to Embodiment 1.

[0029] Figure 6 is a flowchart illustrating a process of calculating the alighting end time using a database in the action of the information processing apparatus according to Embodiment 1.

[0030] Figure 7 is a block diagram illustrating the information processing system according to Embodiment 2.

[0031] Figure 8A is a diagram illustrating a database according to Embodiment 2.

[0032] Figure 8B is another diagram illustrating a database according to Embodiment 2.

[0033] Figure 8C is another diagram illustrating a database according to Embodiment 2.

[0034] Figure 9 is a sequence diagram illustrating the action of the information processing system according to Embodiment 2.

[0035] Figure 10A is a diagram illustrating a prompt of the terminal apparatus according to Embodiment 2.

[0036] Figure 10B is another diagram illustrating a prompt of the terminal apparatus according to Embodiment 2.

[0037] Figure 10C is another diagram illustrating a prompt of the terminal apparatus according to Embodiment 2.

[0038] Figure 10D is another diagram illustrating a prompt of the terminal apparatus according to Embodiment 2.

[0039] Figure 10E is another diagram illustrating a prompt of the terminal apparatus according to Embodiment 2.

[0040] Figure 10F is another diagram illustrating a prompt of the terminal apparatus according to Embodiment 2.

[0041] Figure 11 is a flowchart illustrating a process of deciding priority in the action of the information processing apparatus according to Embodiment 2.

[0042] Symbol explanation

[0043] 1, 101 information processing system

[0044] 3, 130 vehicle

[0045] 5, 150 information processing device

[0046] 7, 170 terminal device (prompt device)

[0047] 33 third obtaining section (sensor)

[0048] 51, 151 detection processing section

[0049] 52, 152 decision section

[0050] 54, 154 communication section (obtaining section)

[0051] 55, 155 prompt control section

[0052] 137 seventh obtaining section (sensor) DETAILED DESCRIPTION

[0053] One aspect of the present disclosure relates to an information processing method that obtains sensor data from a plurality of vehicles, respectively, the sensor data being data obtained from sensors possessed by the plurality of vehicles, respectively, performs each of object detection processing using each of the sensor data, respectively, decides a form of a prompt for monitoring or controlling the plurality of vehicles using a result of each of the object detection processing, and controls the prompt of a prompt device in the decided form.

[0054] Thus, the prompt for monitoring or controlling the plurality of vehicles (hereinafter collectively referred to as monitoring) is performed in the form decided using the result obtained from each of the object detection processing, so that an operator monitoring the plurality of vehicles can easily grasp a vehicle that should be monitored. That is, it is possible to reduce the risk of the operator overlooking a vehicle that should be monitored and to reduce the number of vehicles simultaneously monitored by the operator. Thus, by monitoring the plurality of vehicles, it is possible to ensure the safety of the plurality of vehicles and to reduce the burden on the operator performing the monitoring.

[0055] Further, an information processing device according to one aspect of the present disclosure includes: an obtaining section that obtains sensor data from a plurality of vehicles, respectively, the sensor data being data obtained from sensors possessed by the plurality of vehicles, respectively; a detection processing section that performs each of object detection processing using each of the sensor data, respectively; a decision section that decides a form of a prompt for monitoring or controlling the plurality of vehicles using a result of each of the object detection processing; and a prompt control section that controls the prompt of a prompt device in the decided form.

[0056] Further, an information processing system according to one aspect of the present disclosure includes an information processing device and the prompt device. Further, an information processing system according to one aspect of the present disclosure includes an information processing device and the prompt device.

[0057] These information processing apparatus and information processing system also have the same effects as described above.

[0058] Further, one aspect of the present disclosure relates to an information processing method in which, according to each result of the object detection processing, a plurality of priorities respectively corresponding to the plurality of vehicles is determined, and the form of the prompt is determined in accordance with the plurality of priorities.

[0059] Thus, the prompt device performs a prompt for monitoring or controlling each vehicle in accordance with the priority of each vehicle, so an operator who monitors the plurality of vehicles can grasp a vehicle that should be monitored preferentially among the plurality of vehicles. Therefore, the operator can monitor a vehicle having a higher priority than other vehicles, thereby ensuring safety of the vehicles and reducing the burden on the operator.

[0060] Further, in the information processing method according to one aspect of the present disclosure, each of the states of the plurality of vehicles is obtained from each of the plurality of vehicles, and the plurality of priorities is determined according to each result of the object detection processing and each of the states of the plurality of vehicles.

[0061] Thus, the priority of each vehicle is determined more appropriately using not only each result of the object detection processing but also the state of the plurality of vehicles. Therefore, an operator who monitors the plurality of vehicles can more easily grasp a vehicle that should be monitored preferentially among the plurality of vehicles. Therefore, the operator can more easily monitor a vehicle having a higher priority than other vehicles, thereby ensuring safety of the vehicles and further reducing the burden on the operator.

[0062] Further, in the information processing method according to one aspect of the present disclosure, in the object detection processing, at least one of an on-vehicle passenger and an off-vehicle passenger for a vehicle in a stop is detected, and the form of the prompt is determined in accordance with the detected at least one of the on-vehicle passenger and the off-vehicle passenger.

[0063] In the past, it has been difficult to determine the timing of the departure control in response to the end of the boarding of the vehicle or the end of the alighting from the vehicle. For example, it is difficult to determine whether a person located in the vicinity of the vehicle is a boarding reservation based on the sensor data. If an erroneous determination occurs, it can cause a danger to the passenger. On the other hand, it is possible to consider that the timing of the departure is controlled by an operator who performs the monitoring, but the burden on the operator is increased because the boarding and alighting of the vehicle are often monitored. In this regard, according to the present configuration, the form of the prompt for the monitoring is controlled in accordance with the passenger, so that the operator can easily grasp the vehicle in a condition that should be monitored. Therefore, at the time of the boarding and alighting, the safety of the plurality of vehicles can be ensured, and the burden on the operator can be ensured. In addition, the present configuration is effective for the case where the departure is automatically controlled. As described above, the departure is a condition in which a danger can occur, so the operator is required to monitor preferentially.

[0064] Further, in the information processing method according to one aspect of the present disclosure, in the object detection processing, an attribute or a number of at least one of the boarding passenger and the alighting passenger is detected, and the form of the prompt is determined in accordance with the detected attribute or number.

[0065] In this way, the prompt is performed in accordance with the form of the element related to the priority of the monitoring at the time of the boarding and alighting, so that the operator can more correctly grasp the vehicle that should be monitored.

[0066] Further, in the information processing method according to one aspect of the present disclosure, a time until the end of at least one of the boarding of the boarding passenger to the vehicle in the stop and the alighting of the alighting passenger from the vehicle in the stop is calculated using the attribute or the number, and the form of the prompt is determined in accordance with the calculated time.

[0067] In this way, the prompt is performed in accordance with the form of the boarding and alighting time, so that the operator can be prompted about the vehicle in accordance with the timing that should be monitored. Therefore, the vehicle that should be monitored can be prompted more in real time. For example, the vehicle that should be monitored can be prompted in a manner in which the vehicle with a shorter calculated time is preferentially monitored to the vehicle with a longer calculated time.

[0068] Further, in the information processing method according to one aspect of the present disclosure, in the object detection processing, an object outside the vehicle in the running and a position of the object are detected, and the form of the prompt is determined in accordance with the detected object and the position of the object.

[0069] There has been a technology that controls the behavior of a vehicle in accordance with the state of an object outside the vehicle. However, there are various states of the object, and it is difficult to control all of them. On the other hand, an operator who performs monitoring can be considered to confirm the outside of the vehicle, but it is often burdensome to monitor the outside of a plurality of vehicles. For this reason, with the present configuration, the form of the prompt for monitoring is controlled in accordance with the object outside the vehicle, so that the operator can easily grasp the vehicle in a state that should be monitored. Thus, in the case where there is an object outside the vehicle, both the safety of a plurality of vehicles and the burden on the operator can be reduced.

[0070] Further, in the information processing method according to one aspect of the present disclosure, at least one of a degree of difficulty of automatic driving and a degree of danger outside the plurality of vehicles is calculated for each of the plurality of vehicles using the object and the position of the object, and the form of the prompt is determined in accordance with at least one of the calculated degree of difficulty and the degree of danger.

[0071] Thus, the prompt is performed in accordance with the form of at least one of the degree of difficulty of automatic driving and the degree of danger outside the vehicle, so that the operator can more correctly grasp the vehicle that should be monitored. For example, a vehicle in which at least one of the degree of difficulty of automatic driving or the degree of danger outside the vehicle is relatively high is prompted as a vehicle that should be monitored with priority.

[0072] Further, in the information processing method according to one aspect of the present disclosure, the prompt includes displaying an image that reflects at least one of an upper door and a lower door of each of the plurality of vehicles.

[0073] Thus, the operator can easily confirm the safety when getting on and off the vehicle.

[0074] Further, in the information processing method according to one aspect of the present disclosure, the prompt includes displaying an image that reflects the outside of each of the plurality of vehicles.

[0075] Thus, the operator can easily confirm the safety of the outside of the vehicle.

[0076] Further, in the information processing method according to one aspect of the present disclosure, the form of the prompt is determined in such a manner that the higher the priority in the plurality of priorities, the higher the visual confirmability of the prompt.

[0077] Thus, the higher the priority of the vehicle, the higher the visual confirmability, so that the operator can quickly confirm the information of the vehicle that should be monitored.

[0078] Further, in the information processing method according to one aspect of the present disclosure, the form of the prompt is determined in such a manner that the higher the priority in the plurality of priorities, the more the amount of information of the prompt.

[0079] Thus, the higher the priority, the more information is prompted, so the operator can grasp the information of the vehicle that should be monitored in detail.

[0080] Further, the lower the priority, the less information is prompted, so the frame rate of the vehicle prompted to the operator can be reduced, or the image quality can be reduced. Thus, the network load between the device implementing the information processing method, the vehicle, and the prompting device can be reduced.

[0081] Further, in the information processing method according to one aspect of the present disclosure, the form of the prompting is determined in such a manner that the higher the priority among the plurality of priorities, the more the prompting is emphasized.

[0082] Thus, the higher the priority, the more the prompting is emphasized, so the operator can quickly determine the vehicle that should be monitored.

[0083] Further, in the information processing method according to one aspect of the present disclosure, the form of the prompting is determined in such a manner that when the priority among the plurality of priorities is higher than a threshold value, the prompting is started.

[0084] Thus, only the vehicle whose priority is equal to or higher than the threshold value is prompted, so the operator can focus on the vehicle that should be monitored.

[0085] Further, in the information processing method according to one aspect of the present disclosure, the form of the prompting is determined in such a manner that the order of the prompting corresponds to the order of the height of the priority among the plurality of priorities.

[0086] Thus, the vehicle is prompted in the order of the height of the priority, so the operator can quickly determine the vehicle that should be monitored. Further, the vehicle that should be monitored next can be confirmed in advance, so the efficiency of the monitoring operation can be improved.

[0087] Further, in the information processing method according to one aspect of the present disclosure, the priority is determined further based on at least one of the position of the vehicle, the time corresponding to the position of the vehicle, the history of the required time from the boarding passenger and the alighting passenger to the end of boarding and alighting, and the climate information corresponding to the position of the vehicle.

[0088] By the above, each priority can be more appropriately set for a plurality of vehicles. Thus, the vehicle that should be monitored can be more appropriately prompted by the prompting device.

[0089] In addition, these general or specific aspects can be implemented by a system, a method, an integrated circuit, a computer program, or a recording medium such as a CD-ROM, and can be implemented by any combination of the system, the method, the integrated circuit, the computer program, and the recording medium.

[0090] Embodiments of the present application will be described below with reference to the drawings. The embodiments described below each represent a specific example of the present disclosure. Numerical values, shapes, materials, components, positions of components, connection modes, steps, orders of steps, and the like shown in the following embodiments are merely examples, and the gist of the present disclosure is not limited to them. Furthermore, among components of the following embodiments, components not recited in the most general concept are described as arbitrary components.

[0091] Furthermore, each drawing is a schematic view, and is not a strict illustration. Furthermore, in each drawing, substantially the same components are given the same symbol, and overlapping descriptions are omitted or simplified.

[0092] An information processing method, an information processing apparatus, and an information processing system relating to an embodiment of the present disclosure will be described below.

[0093] (Embodiment 1)

[0094] [Configuration]

[0095] Figure 1 is a block diagram illustrating an information processing system 1 relating to Embodiment 1.

[0096] The information processing system 1 has a vehicle 3, an information processing apparatus 5, and a terminal apparatus 7.

[0097] The vehicle 3 is an automated vehicle capable of not only traveling by remote operation of a remote operator of the vehicle 3, but also automatically traveling without remote operation. The vehicle 3 is, for example, a bus, a taxi, a train, or the like. In the present embodiment, a case where the vehicle 3 is assumed to be a bus, and a case where the vehicle 3 is assumed to be parked at a parking place (a bus stop) are described, and the following embodiment is described. Hereinafter, a person who has boarded or is about to board the vehicle 3 is referred to as a boarding passenger, and a person who has alighted or is about to alight from the vehicle 3 is referred to as an alighting passenger. Furthermore, when both the boarding passenger and the alighting passenger are indicated, it is referred to as a passenger.

[0098] The vehicle 3 has a first acquisition unit 31, a second acquisition unit 32, a third acquisition unit 33, a fourth acquisition unit 34, and a communication unit 35.

[0099] The first acquisition unit 31 is an apparatus that acquires information indicating a boarding / alighting start time, and is, for example, a table. Specifically, the boarding / alighting start time is a time indicating that a passenger starts boarding / alighting the vehicle 3 when the vehicle 3 is parked at a parking place. The first acquisition unit 31 transmits information indicating the boarding / alighting start time to the information processing apparatus 5. Note that the boarding / alighting start time can also be a time when the vehicle 3 is parked at a parking place by automated driving.

[0100] The second obtaining section 32 obtains vehicle position information (for example, GPS (Global Positioning System) information) indicating the current position of the vehicle 3. The second obtaining section 32 transmits the obtained vehicle position information to the information processing device 5 and the terminal device 7. The current position of the vehicle 3 is an example of the position of the vehicle 3.

[0101] The third obtaining section 33 is a device that obtains a passenger image, for example, a camera or the like, which is an image that captures a passenger who gets on and off the vehicle 3. Specifically, the third obtaining section 33 captures a person who is waiting at a parking place of the vehicle 3 or the like (for example, a passenger who gets on) or a person who stands up inside the vehicle (for example, a passenger who gets off), and generates a passenger image. The third obtaining section 33 transmits the passenger image to the information processing device 5. The third obtaining section 33 is an example of a sensor. Furthermore, the passenger image is an example of sensor data.

[0102] The fourth obtaining section 34 is a device that obtains a vehicle interior and exterior image, for example, a camera or the like, which is an image indicating the inside of the vehicle and the outside of the vehicle. The vehicle interior and exterior image includes information of a pedestrian, a vehicle, a bicycle, or the like outside the vehicle, that is, outside the vehicle 3, information of a passenger inside the vehicle, and information of a passenger near a door. The fourth obtaining section 34 transmits the vehicle interior and exterior image to the terminal device 7. In addition, the fourth obtaining section 34 can be used in common with the third obtaining section 33, and the passenger image can be the vehicle interior and exterior image.

[0103] The communication section 35 is a communication interface that transmits information indicating the boarding and alighting start time, vehicle position information, and a passenger image to the information processing device 5, and transmits vehicle position information and information indicating a vehicle interior and exterior image to the terminal device 7. Furthermore, the communication section 35 accepts a control command for remote operation in a case where the vehicle 3 is remotely operated by a remote operator. The communication section 35 outputs the control command to an unillustrated actuator, a display, or the like that controls the vehicle 3.

[0104] Here, there are a plurality of vehicles 3, and each vehicle 3 has an identification of a vehicle ID. The communication section 35 transmits information indicating the boarding and alighting start time, vehicle position information, and a passenger image, each of which is assigned with a vehicle ID, to the information processing device 5, or transmits a vehicle interior and exterior image, which is assigned with a vehicle ID, to the terminal device 7.

[0105] The information processing device 5 is a device that can remotely control a plurality of vehicles 3 that are automatically driven according to an instruction of an operator who is located at a remote place. The information processing device 5 has a detection processing section 51, a decision section 52, a storage control section 53, a communication section 54, and a prompt control section 55.

[0106] The detection processing unit 51 performs object detection processing for each vehicle. Object detection processing involves detecting at least one of the passengers boarding or disembarking a stopped vehicle 3, or detecting the attributes or number of at least one of the passengers boarding or disembarking. Specifically, the detection processing unit 51 obtains a passenger image from the third acquisition unit 33 of the vehicle 3, detects the number of passengers and the attributes of each passenger based on the passenger image, and generates passenger information for each vehicle 3. Passenger attributes include information such as adult, child, and elderly. The detection processing unit 51 outputs the passenger information to the determination unit 52.

[0107] The determination unit 52 uses each result of the object detection process to determine the form of presentation for monitoring or controlling the plurality of vehicles 3. Details of the presentation form will be described later.

[0108] Furthermore, upon receiving the individual passenger information from the detection processing unit 51 and obtaining the boarding / getting-off start time and vehicle position information from the first obtaining unit 31 and the second obtaining unit 32 of the vehicle 3, the determining unit 52 obtains past boarding / getting-off data from the database stored in the storage control unit 53. Specifically, the determining unit 52 requests the storage control unit 53 for past boarding / getting-off data similar to the individual passenger information, boarding / getting-off start time, and vehicle position information, and obtains the similar past boarding / getting-off data from the storage control unit 53.

[0109] The decision unit 52 obtains the past boarding and alighting data from the storage control unit 53 and determines the priority level as the priority monitoring index for each vehicle 3. Figure 2A As shown, the determination unit 52 transmits each priority information in which each calculated priority is associated with the vehicle ID obtained from each vehicle 3 to the terminal device 7 . Figure 2A This is a diagram illustrating priority information according to the first embodiment.

[0110] The storage control unit 53 stores past boarding and alighting data compiled into a database. This boarding and alighting data includes information such as boarding and alighting start time, parking location, passenger attributes, the number of passengers for each attribute, the time required for all passengers to complete boarding and alighting, and boarding and alighting times by passenger attribute. Upon request from the decision unit 52, the storage control unit 53 searches for past boarding and alighting data with similar passenger information and transmits such data to the decision unit 52. Alternatively, the storage control unit 53 may store a learned model based on past boarding and alighting data, instead of the past boarding and alighting data compiled into the database. In this case, the storage control unit 53 transmits the learned model to the decision unit 52 upon request from the decision unit 52.

[0111] The boarding and alighting data is provided by Figure 2B as well as Figure 2Cshown. Figure 2B and Figure 2C is a diagram illustrating the boarding / alighting data involved in Embodiment 1. As shown in Figure 2B , the boarding / alighting data is constituted by "time" when the vehicle 3 stops at a parking place, "place" where the vehicle stops, "attribute of passenger and number of passengers", and "required time" until the boarding / alighting of all the passengers is completed. Figure 2C As shown in , the boarding / alighting data further has "boarding / alighting time" of each of the passengers in the attribute of the passengers.

[0112] As shown in Figure 1 , the communication section 54 is a communication interface that receives information indicating the boarding / alighting start time, vehicle position information, and passenger image, or transmits priority information to the terminal device 7. The communication section 54 is an example of the obtaining section.

[0113] The prompting control section 55 controls the form of the prompt of the terminal device 7 in accordance with the form of the prompt decided by the deciding section 52. In other words, the prompting control section 55 outputs a command to control the prompt of the terminal device 7 in a manner that the terminal device 7 prompts in accordance with the decided priority.

[0114] The terminal device 7 is an operation terminal for an operator to monitor the running state of the vehicle 3, or to remotely operate the vehicle 3, and is, for example, a personal computer, a tablet terminal, or the like. The terminal device 7 has a prompting section 71, an operation section 72, a timing section 73, and a communication section 74.

[0115] The prompting section 71 is a display that prompts the in-vehicle and out-vehicle image, the vehicle position information, and the like for each vehicle 3. Specifically, the prompting section 71 prompts the image of the inside of the vehicle, the image of the door of the vehicle 3, the image of the outside (the surroundings of the vehicle 3) of the vehicle 3, the priority of the vehicle 3, the boarding / alighting completion time, the vehicle ID, the parking place, and the like in relation to the vehicle 3 for each vehicle 3.

[0116] The prompting section 71 prompts the information in relation to the vehicle 3 in various forms. For example, the prompting section 71 prompts the information of the vehicle 3 with the highest priority with a large image, or prompts the information of only the vehicle 3 with a priority above a threshold value from among a plurality of vehicles 3, or prompts the information of the vehicle 3 with a high priority in detail, or prompts the information in relation to the vehicle 3 in a manner that the information is arranged in order of the priority height when the images of the vehicles 3 are prompted in the same size, or prompts the information in a manner that the image of the vehicle 3 with the highest priority is emphasized by attaching a color thereto.

[0117] The operation section 72 is an input section for accepting remote operation by the operator to the vehicle 3. The operator, confirming that all passengers have finished boarding and alighting (boarding and alighting completion confirmation), operates the operation section 72 when causing the vehicle 3 to depart. The operation section 72 is, for example, a touch panel, a keyboard, or the like. The operation section 72 transmits a control command for remote operation to the vehicle 3 via the communication section 74.

[0118] The time measuring section 73 measures a time point at which the operator operates the operation section 72 to transmit a control command for causing the vehicle 3 to depart, and is, for example, a watch.

[0119] The communication section 74 is a communication interface that transmits a control command to the vehicle 3.

[0120] In addition, in the present embodiment, the time measuring section 73 of the terminal device 7 is not an essential component, and the second obtaining section 32 of the vehicle 3 is not an essential component. Therefore, either the time measuring section 73 or the second obtaining section 32 can not be provided.

[0121] [Operation]

[0122] Next, the operation of the information processing method, the information processing device 5, and the information processing system 1 of the present embodiment will be described. Specifically, the operation of the information processing system 1 when the vehicle 3 is parked at a bus stop and until the vehicle 3 departs after a certain period of time will be described.

[0123] Figure 3 is a sequence diagram showing the operation of the information processing system 1 according to Embodiment 1.

[0124] As shown in Figure 3 , the vehicle 3 obtains information indicating a boarding and alighting start time, and transmits the obtained information to the information processing device 5 (step S101). In addition, the vehicle 3 obtains vehicle position information indicating the current position of the vehicle 3, and transmits the obtained information to the information processing device 5 and the terminal device 7 (step S102). The decision section 52 of the information processing device 5 receives the information indicating the boarding and alighting start time and the vehicle position information via the communication section 54.

[0125] In addition, the vehicle 3 obtains a passenger image indicating an image of a person who boards and alights the vehicle 3, and transmits the obtained passenger image to the information processing device 5 (step S103). The detection processing section 51 of the information processing device 5 receives the passenger image via the communication section 54.

[0126] On the other hand, the vehicle 3 obtains a vehicle interior and exterior image, and transmits the vehicle interior and exterior image to the terminal device 7 (step S104).

[0127] The detection processing section 51 generates passenger information indicating the number of passengers detected, attributes of the passengers, and the like, based on the passenger image in step S103 (step S105). The detection processing section 51 outputs the passenger information to the decision section 52 (step S106).

[0128] The decision section 52 requests the storage control section 53 for past boarding / alighting data similar to the obtained passenger information, boarding / alighting start time, and vehicle position information (step S107).

[0129] The storage control section 53, upon request from the decision section 52, searches for past boarding / alighting data similar to the passenger information, boarding / alighting start time, and vehicle position information, and outputs the similar past boarding / alighting data to the decision section 52 (step S108). The boarding / alighting data is constituted by, for example, the following information: time at which the vehicle 3 stopped at the stop location, stop location, attribute of the passengers, number of passengers for each attribute, time required for all passengers to start boarding / alighting until completion, boarding / alighting time for each attribute of the passengers, and the like. In addition, the storage control section 53, upon request from the decision section 52, can transmit the learned model to the decision section 52, instead of outputting the similar past boarding / alighting data to the decision section 52 upon request from the decision section 52. Figure 2B Figure 2C The decision section 52, from the storage control section 53, obtains past boarding / alighting data, and sets a priority of monitoring for each vehicle 3 (step S109). The decision section 52 decides the priority of each vehicle 3 based on the passenger information, boarding / alighting start time, and vehicle position information. The priority will be described later. The decision section 52 transmits priority information in which the set priority is associated with the vehicle ID, to the terminal device 7 (step S110). In addition, the decision section 52 can obtain the learned model from the storage control section 53, and set a priority of monitoring for each vehicle 3, instead of obtaining past boarding / alighting data from the storage control section 53. In this case, the decision section 52 decides the priority of each vehicle 3 using the obtained learned model.

[0130] Here, the decision section 52 decides, for the information about the vehicle 3 prompted by the terminal device 7, a prompt form in which the higher the priority, the higher the visual confirmation of the prompt, or the higher the priority, the more the amount of information is increased, or the higher the priority, the more the prompt is emphasized. Further, the decision section 52 decides the prompt form of the information about the vehicle 3 to be arranged in order of the height of the priority. The prompt control section 55 controls the prompt of the terminal device 7 in the decided form, in other words, causes the terminal device 7 to start prompting the information about the vehicle 3 when the priority is higher than a threshold value.

[0131] Here, the decision section 52 decides, for the information about the vehicle 3 prompted by the terminal device 7, a prompt form in which the higher the priority, the higher the visual confirmation of the prompt, or the higher the priority, the more the amount of information is increased, or the higher the priority, the more the prompt is emphasized. Further, the decision section 52 decides the prompt form of the information about the vehicle 3 to be arranged in order of the height of the priority. The prompt control section 55 controls the prompt of the terminal device 7 in the decided form, in other words, causes the terminal device 7 to start prompting the information about the vehicle 3 when the priority is higher than a threshold value.

[0132] ​Next, the terminal device 7 receives the in-vehicle and out-vehicle images obtained in step S104 and the priority information obtained in step S110. The terminal device 7 performs a presentation corresponding to the priority (step S111). The terminal device 7 performs the presentation in various modes according to the priority. In Figures 4A-4E , an example of the mode of presentation by the presentation section 71 is shown.

[0133] Figure 4A is a view illustrating the presentation by the terminal device 7 according to Embodiment 1. As shown in Figure 4A , the presentation section 71 presents the vehicle 3 having the highest priority with a large-area image (in a mode with high visual confirmation). Specifically, among the plurality of vehicles 3, only the vehicle 3 having a prescribed priority (threshold) or more is presented, and the vehicle 3 having the highest priority is presented with a large-area image. The information about the vehicle 3 having the first priority shows the time required for all the people to get on and off the vehicle 3, an image about the vehicle 3, a vehicle ID, and a parking place. The image about the vehicle 3 is an image of the inside of the vehicle, an image of the door of the vehicle 3, an image of the outside of the vehicle, and the like. Further, for example, the vehicle 3 having the second priority or more, only simple information about the vehicle 3 is presented, for example, only one image about the vehicle 3 is presented, for example, the frame rate of the image is reduced, or the quality of the image is reduced. Further, the vehicle 3 having a priority less than the prescribed priority, which is the vehicle 3 having the sixth priority or more in this embodiment, the presentation section 71 does not present information about the vehicle 3.

[0134] Figure 4B is another view illustrating the presentation by the terminal device 7 according to Embodiment 1. As shown in Figure 4B , the presentation section 71 presents only information about the vehicle 3 having a prescribed priority or more among the plurality of vehicles 3. In Figure 4B , only information about the vehicle 3 having a time close to the time when getting on and off is completed is presented preferentially. In this embodiment, information about two vehicles 3 is presented, and information about the other vehicle 3 having the third priority or more is not presented.

[0135] Figure 4C is another view illustrating the presentation by the terminal device 7 according to Embodiment 1. Figure 4C As shown in Figure 4C , the presentation section 71 presents information about the vehicle 3 having the highest priority with a large-area image, and presents only the vehicle 3 having a prescribed priority or more among the plurality of vehicles 3. In , an image about the vehicle 3 is presented.

[0136] Figure 4D is another view illustrating the presentation by the terminal device 7 according to Embodiment 1. As shown in Figure 4DAs shown, the prompter 71 prompts the information on the vehicle 3 in a size that the operator can visually confirm the information on the vehicle 3 through the prompter 71. In Figure 4D the information on the vehicle 3 prompted in the above is arranged in order of priority level.

[0137] Figure 4E is another diagram illustrating the prompt of the terminal device 7 according to Embodiment 1. As shown, the prompter 71 prompts the information on the vehicle 3 with the highest priority level with emphasis by attaching a color thereto. Figure 4E

[0138] In addition, Figures 4A-4E the prompt is only an example and is not limited to these. For example, the prompt can include displaying an image that maps an image of at least one of the boarding door and the alighting door of each of the plurality of vehicles 3. Further, the prompt includes displaying an image that maps an image of the outside of each of the plurality of vehicles 3.

[0139] Returning to the explanation of the sequence diagram of Figure 3

[0140] When the all passengers finish boarding and alighting the vehicle 3, the terminal device 7 accepts a departure instruction from the operator (step S112). For example, the operator confirms that the boarding and alighting is finished via the terminal device 7. When the operator confirms that the boarding and alighting is finished, the operator inputs a departure instruction to the terminal device 7. The terminal device 7 accepts the input departure instruction. In addition, the confirmation that the boarding and alighting is finished can be performed by the terminal device 7 confirming whether or not there is a person near the door by image recognition.

[0141] When the terminal device 7 accepts the departure instruction from the operator, the terminal device 7 transmits a control command for causing the vehicle 3 to depart to the vehicle 3 (step S113).

[0142] When the vehicle 3 receives the control command of step S113, the vehicle 3 departs (step S114).

[0143] Further, when the terminal device 7 accepts the departure instruction at step S112, the timer 73 measures a boarding and alighting finish time corresponding to the acceptance of the departure instruction and transmits information indicating the boarding and alighting finish time to the information processing device 5 (step S115).

[0144] Next, the determination section 52 stores the information indicating the boarding and alighting start time received at step S101 in the storage control section 53 (step S116), thereby updating the database of the storage control section 53 (step S118).

[0145] ​​Further, the decision section 52 stores the passenger information obtained in step S106 in the storage control section 53 (step S117), thereby updating the database of the storage control section 53 (step S118).

[0146] Further, the decision section 52 stores the passenger information obtained in step S106 in the storage control section 53 (step S117), thereby updating the database of the storage control section 53 (step S118).

[0147] Thus, the processing of the information processing system 1 is ended. Further, in this information processing system 1, the database update of the storage control section 53 and the update of the learned model in steps S115 to S118 are not necessary processing, and such processing can not be performed.

[0148] Next, the operation in the information processing apparatus 5 will be described with reference to Figure 5 Here, the flow is performed when the vehicle 3 is parked at a prescribed position. Figure 5 is a flowchart illustrating the operation of the information processing apparatus 5 related to Embodiment 1.

[0149] First, as shown in Figure 5 , the detection processing section 51 of the information processing apparatus 5 receives the passenger image from the vehicle 3 (step S161). The detection processing section 51 detects the number of passengers, the attributes of the passengers, and the like from the passenger image, and generates passenger information. The detection processing section 51 outputs the passenger information to the decision section 52.

[0150] Further, the decision section 52 of the information processing apparatus 5 receives the information indicating the boarding / alighting start time and the vehicle position information from the vehicle 3 (step S162).

[0151] The decision section 52 determines whether there are passengers with respect to the vehicle 3 on the basis of the passenger information (step S163). Specifically, the decision section 52 determines whether the number of passengers included in the passenger information is one or more.

[0152] In a case where there are passengers in the passenger information (YES in step S163), the decision section 52 calculates the boarding / alighting end time. Specifically, the decision section 52 calculates the time required from the start of boarding / alighting by all the passengers to the end on the basis of the passenger information and past boarding / alighting data.

[0153] The decision section 52 adds the calculated required time to the boarding / alighting start time obtained in step S162, and calculates the boarding / alighting end time for each vehicle 3 (step S164).

[0154] Next, the decision section 52 sets a higher priority to the vehicle 3 that is closer to the boarding / alighting end time from the current time among the plurality of vehicles 3. The decision section 52 sets the priority of each vehicle 3 each time the vehicle 3 stops at the station, and updates the priority of each vehicle 3 in order of the boarding / alighting end time (step S165). The current time is an example of the time.

[0155] On the other hand, in the passenger information, in the case where there is no passenger (NO in step S163), the decision section 52 sets the vehicle 3 for which it is judged that there is no passenger as the vehicle 3 having the highest priority (step S167). In addition, in the case where there are a plurality of vehicles 3 having no passenger, the decision section 52 sets all of the vehicles 3 as the vehicle 3 having the highest priority. The decision section 52 decides the priority of the other vehicles 3 respectively after calculating the boarding / alighting end time as in step S164, and updates the priority of each vehicle 3 (step S168).

[0156] Next, the decision section 52 transmits information indicating the priority of each vehicle 3 to the terminal device 7 via the communication section 54 (step S166). Then, the process of the information processing device 5 ends. In addition, the same process is performed for the next prescribed position at which the vehicle 3 stops.

[0157] The process of calculating the boarding / alighting end time of step S164 will be described in detail. Figure 6 Figure 5 The process of calculating the boarding / alighting end time of step S164 will be described in detail. Figure 6 is a flowchart illustrating the process of calculating the boarding / alighting end time using a database in the operation of the information processing device 5 according to Embodiment 1. In addition, instead of the database, a learned model can be used to calculate the boarding / alighting end time.

[0158] First, as shown in Figure 6 , the decision section 52 refers to the past boarding / alighting data stored in the storage control section 53 (step S171). Specifically, the decision section 52 requests the past boarding / alighting data similar to the passenger information, the boarding / alighting start time, and the vehicle position information from the storage control section 53, and obtains the past boarding / alighting data corresponding to the request from the storage control section 53.

[0159] ​The decision section 52 determines whether there is past boarding and alighting data similar to the passenger information, the boarding and alighting start time, and the vehicle position information (step S172). For example, a similar case refers to a case where at least one of the passenger information, the boarding and alighting start time, and the vehicle position information coincides with past boarding and alighting data. In the case of the boarding and alighting start time, for example, a case where the boarding and alighting start time is within 1 hour or the like from the boarding and alighting start time, or the like is considered to be similar.

[0160] In the case where there is past boarding and alighting data similar to the passenger information (YES in step S172), the decision section 52 adopts the required time from the start to the end of boarding and alighting by all people (past required time) as the required time from the start to the end of boarding and alighting by all people at present (step S173).

[0161] The decision section 52 adds the boarding start time to the required time adopted in step S173 to calculate the alighting end time (step S174). Then, this flow is ended.

[0162] On the other hand, in the case where there is no past boarding and alighting data similar to the passenger information (NO in step S172), the decision section 52 extracts the attribute of the passenger and the boarding and alighting time corresponding to the attribute of the passenger from the past boarding and alighting data stored in the storage control section 53 (step S175).

[0163] The decision section 52 calculates the required time from the start to the end of boarding and alighting by all people on the basis of step S175 (step S176). The decision section 52 proceeds to step S174 to calculate the alighting end time. Then, this flow is ended.

[0164] As described above, according to the information processing method, the information processing apparatus 5, and the information processing system 1, when the operator monitors a plurality of vehicles 3 in which the stop is in progress, the time required from the start of the boarding and alighting of all passengers to the end of the boarding and alighting can be calculated for each vehicle 3 according to the respective numbers and respective attributes of the boarding passengers and the alighting passengers. The information processing apparatus 5 sets a higher priority to the vehicle 3 for which the time until the end of the boarding and alighting is shorter according to the start time of the boarding and alighting and the required time. The terminal apparatus 7 prompts the vehicle 3 for which the priority is high with a large-area image, or only prompts the vehicle 3 for which the priority is above a threshold value, or prompts the relevant information of the vehicle 3 arranged in order of the priority height, or prompts the information related to the vehicle 3 in detail, or prompts in a manner in which the image of the vehicle 3 for which the priority is the highest is emphasized by attaching a color. Therefore, the operator only needs to monitor the plurality of vehicles 3 prompted on the terminal apparatus 7, in other words, can easily grasp the vehicle that should be monitored. As a result, the number of vehicles 3 that the operator needs to monitor at the same time is reduced, so the risk that the operator overlooks the plurality of vehicles 3 that should be monitored can be reduced. As a result, by monitoring the plurality of vehicles 3, both the safety of the plurality of vehicles 3 and the burden of the operator who monitors the plurality of vehicles 3 can be reduced.

[0165] (Embodiment 2)

[0166] [Configuration]

[0167] In this embodiment, the following embodiment is described assuming that the vehicle 130 is in a case where the automatic driving is in progress.

[0168] The configuration of the information processing method, the information processing apparatus 150, and the information processing system 101 of the present embodiment is the same as that of Embodiment 1 unless otherwise specified, the same configuration is given the same symbol, and detailed description of the configuration is omitted.

[0169] Figure 7 is a block diagram illustrating the information processing system 101 to which Embodiment 2 is applied.

[0170] As Figure 7 indicated, the information processing system 101 includes the vehicle 130, the information processing apparatus 150, and the terminal apparatus 170.

[0171] The vehicle 130 includes a fifth obtaining unit 135, a sixth obtaining unit 136, a seventh obtaining unit 137, and a communication unit 138.

[0172] The fifth obtaining section 135 is a device that obtains vehicle state information indicating a state of the vehicle 130, such as an ECU (Electronic Control Unit) or the like. The vehicle state information includes information indicating a state of the vehicle 130, such as driving, temporary stop, emergency stop, and travel speed. The fifth obtaining section 135 transmits the vehicle state information to the information processing device 150 and the terminal device 170.

[0173] The sixth obtaining section 136 obtains vehicle position information (e.g., GPS information) indicating a current position of the vehicle 130. The sixth obtaining section 136 transmits the obtained vehicle position information to the information processing device 150 and the terminal device 170.

[0174] The seventh obtaining section 137 performs each of object detection processes. The object detection processes detect an object outside the vehicle during travel and a position of the object. The seventh obtaining section 137 obtains sensing information indicating a presence of a person inside the vehicle 130, a position of an object outside the vehicle 130, and the like. The seventh obtaining section 137 is an image sensor such as a camera that captures an inside and an outside of the vehicle 130, a distance sensor such as a LIDAR (Laser Imaging Detection and Ranging), and the like. The seventh obtaining section 137 transmits the sensing information to the information processing device 150 and the terminal device 170. The object outside the vehicle 130 is, for example, a person walking in front of the vehicle 130, a bicycle existing behind the vehicle 130, and the like. The seventh obtaining section 137 is an example of a sensor.

[0175] The communication section 138 is a communication interface that transmits the vehicle state information, the vehicle position information, and the sensing information to the information processing device 150. In addition, the communication section 138 receives a control command for remote operation when the vehicle 130 is in a state of being remotely operated by a remote operator.

[0176] The detection processing section 151 of the information processing device 150 obtains the sensing information from the seventh obtaining section 137 of the vehicle 130, and generates vehicle inside and outside information, such as a presence of a person inside the vehicle 130 and an object outside the vehicle 130, from the sensing information. The detection processing section 151 outputs the vehicle inside and outside information to the decision section 152.

[0177] When the inside and outside information of the vehicle is obtained from the detection processing section 151, the decision section 152 of the information processing device 150 obtains past vehicle data stored in the database of the storage control section 153. Specifically, the decision section 152 requests the storage control section 153 for past vehicle data similar to the inside and outside information of the vehicle, and obtains the requested past vehicle data from the storage control section 153. Alternatively, the decision section 152 can request the storage control section 153 for a learned model created based on the past vehicle data, and obtain the learned model from the storage control section 153, instead of the past vehicle data stored in the database of the storage control section 153.

[0178] The decision section 152 calculates at least one of the difficulty of autonomous driving of each of the plurality of vehicles and the external danger degree of the plurality of vehicles 130, respectively, based on the sensing information, the past vehicle data, and the like. The difficulty of autonomous driving is a parameter indicating the ease of travel of the vehicle 130. For example, the difficulty of autonomous driving is high in a case where the vehicle 130 travels on a road with a narrow width, in a case where the vehicle 130 travels in an environment where rain, fog, or the like occurs, and the like. The external danger degree of the plurality of vehicles 130 is a parameter indicating the likelihood of an accident occurring in the area where the vehicle 130 travels. For example, the danger degree is high in a case where there is a dangerous area where a vehicle suddenly exits a road, an area under construction, and the like. In the following, the danger degree will be mainly described in the present embodiment. Alternatively, the decision section 152 can calculate at least one of the difficulty of autonomous driving of each of the plurality of vehicles and the external danger degree of the plurality of vehicles 130, respectively, using the obtained learned model, instead of using the sensing information and the past vehicle data, and the like.

[0179] The storage control section 153 of the information processing device 150 is a storage device that stores past vehicle data. The vehicle data is information such as a label, a place corresponding to the label, a recommended operation corresponding to the label, a danger degree corresponding to the label, and the like. The label is an explanatory text for a dangerous area where a vehicle suddenly exits a road, an area under construction, or the like, with respect to a certain place. The storage control section 153 searches for past vehicle data similar to the inside and outside information of the vehicle, and transmits the similar past vehicle data to the decision section 152, in accordance with a request from the decision section 152. Alternatively, the danger degree can be arbitrarily changed, for example, in accordance with the number of people present outside the vehicle 130, and the danger degree can change. Alternatively, the storage control section 153 can store a learned model learned based on the past vehicle data, instead of storing the past vehicle data.

[0180] Figure 8A 、 Figure 8B and Figure 8C is a diagram illustrating past vehicle data related to Embodiment 2. As Figure 8AAs shown, the past vehicle data is composed of "place" where a dangerous situation has occurred, "label" which is an explanatory text of the dangerous situation, "recommended operation" corresponding to the label, and "danger degree" corresponding to the label. As shown in Figure 8B As shown, the past vehicle data also has "state" which occurs in the vehicle 130 such as activation of AEB (Autonomous Emergency Braking), temporary stop, and the like, "label" which is an explanatory text corresponding to the state, "recommended operation" corresponding to the state, and "danger degree" corresponding to the state. As shown in Figure 8C As shown, the past vehicle data also has "situation" of the outside of the vehicle 130 which occurs in the running of the vehicle 130 such as a pedestrian on the road, a bicycle behind the vehicle 130, and the like, "label" which is an explanatory text corresponding to the situation, "recommended operation" corresponding to the situation, and "danger degree" corresponding to the situation.

[0181] The presentation section 171 of the terminal device 170 receives, via the communication section 173, information on the vehicle 130 such as the priority of the vehicle 130, the state of the vehicle 130, the image of the inside of the vehicle 130, the image of the outside of the vehicle 130, the vehicle ID, the current position of the vehicle 130, and the like, and presents the received information for each vehicle 130.

[0182] The presentation section 171 of the terminal device 170 presents the vehicle 130 with the highest priority with a large image, or presents only the vehicle 130 with a priority above a threshold value from among a plurality of vehicles 130, or presents detailed information of the vehicle 130, or presents information on the vehicles 130 arranged in order of priority height with images of the same size, or presents in a manner of emphasizing the image of the vehicle 130 with the highest priority by attaching a color thereto.

[0183] [Action]

[0184] The action of the information processing method, the information processing device 150, and the information processing system 101 in the present embodiment will be described next. Specifically, the action of the information processing system 101 in the running of the vehicle 130 will be described.

[0185] Figure 9 is a sequence chart illustrating the action of the information processing system 101 according to Embodiment 2.

[0186] As shown in Figure 9 The vehicle 130 obtains vehicle state information, and transmits to the information processing device 150 and the terminal device 170 (step S201). The decision section 152 of the information processing device 150 receives the vehicle state information via the communication section 154.

[0187] Furthermore, the vehicle 130 obtains vehicle position information indicating the current position of the vehicle 130 and transmits the vehicle position information to the information processing device 150 and the terminal device 170 (step S202 ). The determination unit 152 of the information processing device 150 receives the vehicle position information via the communication unit 154 .

[0188] Vehicle 130 obtains sensor information indicating the appearance of people inside vehicle 130 and objects or their locations outside vehicle 130, and transmits the obtained sensor information to information processing device 150 and terminal device 170 (step S203). Determination unit 152 of information processing device 150 receives the sensor information via communication unit 154.

[0189] Based on the sensing information in step S203, the detection processing unit 151 generates vehicle interior and exterior information indicating the appearance of people inside the vehicle 130 and the positions of objects outside the vehicle 130 (step S204). The detection processing unit 151 outputs the vehicle interior and exterior information to the determination unit 152 (step S205).

[0190] The determination unit 152 requests the storage control unit 153 to obtain past vehicle data similar to the acquired vehicle interior and exterior information (step S206 ).

[0191] The storage control unit 153 searches for past vehicle data similar to the vehicle interior and exterior information in accordance with a request from the decision unit 152, and outputs the similar past vehicle data to the decision unit 152 (step S207). Figures 8A-8C For example, past vehicle data refers to information such as a location, a location corresponding to a tag, a recommended operation corresponding to the tag, a risk level corresponding to the location, a state of the vehicle 130, and a condition corresponding to the location.

[0192] The decision unit 152 obtains past vehicle data from the storage control unit 153 and sets a priority level for each vehicle 130 to be monitored (step S208). The decision unit 152 determines the priority level based on the risk level. Priority level determination will be described later. The decision unit 152 transmits information indicating the determined priority level for each vehicle 130 to the terminal device 170 (step S209).

[0193] Next, the terminal device 170 receives the vehicle state information and vehicle interior and exterior information acquired in step S201, and the information indicating the priority of each vehicle 130 determined in step S209. The terminal device 170 presents information according to the priority (step S210).

[0194] The terminal device 170 provides various forms of prompts according to the priority. Figures 10A-10F , illustrating the form of the prompt of the prompt unit 171.

[0195] Figure 10A is another diagram illustrating the presentation of the terminal device 170 according to Embodiment 2. As shown in the diagram, the presentation section 171 presents the images of the plurality of vehicles 130 in the order of the height of the priority. In this embodiment, the information about the vehicles 130 is presented in the order of the height of the priority. Figure 10A

[0196] Figure 10B is another diagram illustrating the presentation of the terminal device 170 according to Embodiment 2. As shown in the diagram, the presentation section 171 presents the images of the plurality of vehicles 130 in the order of the height of the priority. In this embodiment, the information about the vehicles 130 is presented in the order of the height of the priority. Figure 10B

[0197] Figure 10C is another diagram illustrating the presentation of the terminal device 170 according to Embodiment 2. As shown in the diagram, the presentation section 171 presents the images of the plurality of vehicles 130 in the order of the height of the priority. In this embodiment, the information about the vehicles 130 is presented in the order of the height of the priority. Figure 10C Figure 10C

[0198] Figure 10D Figure 10D Figure 10D

[0199] Figure 10E ​​​​​​​is another diagram illustrating the presentation of the terminal device 170 to which Embodiment 2 is applied. As shown in Figure 10E , the presentation part 171 presents the information on the vehicle 130 with the highest priority with the color emphasized.

[0200] Figure 10F is another diagram illustrating the presentation of the terminal device 170 to which Embodiment 2 is applied. As shown in Figure 10F , in the case where a plurality of vehicles 130 are continuously traveling in the longitudinal direction, these vehicles 130 are summarized as one, and presented as one vehicle 130.

[0201] In addition, Figures 10A-10F the presentation is only an example, and is not limited thereto.

[0202] Returning to Figure 9 the sequence diagram, the explanation will be continued.

[0203] The terminal device 170 accepts the action instruction of the operator (step S211). For example, the operator confirms the running state of each vehicle 130 from each of the vehicles 130 presented on the terminal device 170. Then, the operator inputs the action instruction to the terminal device 170. When the terminal device 170 accepts the action instruction of the operator, the terminal device 170 transmits the control command to each vehicle 130 presented on the terminal device 170 (step S212). Here, the control command refers to the command for controlling the running of the vehicle 130, such as the emergency stop of the vehicle 130, the temporary stop of the vehicle 130, the slow driving of the vehicle 130, the steering operation of the vehicle 130, and the like.

[0204] The vehicle 130, upon receiving the control command of step S212, performs the action corresponding to the control command and the display (step S213). In this way, the processing of the information processing system 101 ends.

[0205] The terminal device 170 generates the situation data indicating the evaluation corresponding to the control command transmitted in step S212. For example, in the case where the slow driving of the vehicle 130 is performed due to the sudden road exit behavior of the vehicle A, the terminal device 170 generates the situation data in which the place is "A place", the label is "danger of sudden road exit", the recommended operation is "slow driving", and the danger degree is "high". In addition, the situation data can be generated by the operator inputting via the operation part 172. Further, the situation data can be automatically generated by the terminal device 170. The terminal device 170 transmits the generated situation data to the information processing device 150 (step S214). The storage control part 153 of the information processing device 150 receives the situation data via the communication part 154. The situation data is stored by the storage control part 153, and the database of the storage control part 153 is updated.

[0206] Further, the decision section 152 generates, for example, travel information indicating the position where the vehicle 130 stopped, information of the object outside the vehicle 130 at that time, and the like, based on the vehicle state information, the vehicle position information, and the sensor information, and outputs the travel information to the storage control section 153 (step S215).

[0207] The storage control section 153 obtains the travel information, and updates the database of the storage control section 153 (step S216). Also, in the information processing system 101, this processing is ended.

[0208] Further, in the information processing system 101, the update of the database of the storage control section 153 in steps S214 to S216 is not a necessary processing, and such processing can not be performed.

[0209] Here, the priority setting in the information processing device 150 is specifically described. Figure 11 is a flowchart illustrating the process of deciding the priority using the database in the action of the information processing device 150 related to Embodiment 2. Further, instead of using the database, the priority can be decided using a learned model.

[0210] First, as shown in Figure 11 , the information processing device 150 obtains the vehicle state information, the sensor information, and the vehicle position information (step S261).

[0211] The decision section 152 of the information processing device 150 judges whether or not the vehicle 130 has an abnormality based on the vehicle state information (step S262). Specifically, the decision section 152 judges whether or not information indicating an action having high danger in the travel of the vehicle 130, such as AEB activation, is included in the vehicle state information.

[0212] In the case where the vehicle 130 has an abnormality (YES in step S262), the decision section 152 sets the priority of the vehicle 130 from which the vehicle state information was transmitted to "high" (step S263).

[0213] The decision section 152 transmits information indicating the priority of the vehicle 130 set in step S263 to the terminal device 170 (step S264).

[0214] In the case where the vehicle 130 does not have an abnormality (NO in step S262), the decision section 152 refers to the database of the storage control section 153 stored in the storage control section 153 with respect to the region (step S265). Specifically, the decision section 152 refers to whether or not the place (region) where the vehicle 130 is now traveling exists in the database based on the vehicle position information.

[0215] The decision section 152 determines whether there is a dangerous place in the region in which the vehicle is currently traveling, based on the vehicle position information (step S266). The dangerous place refers to, for example, a place where there is a risk of suddenly running off the road, a construction site, and the like.

[0216] In a case where there is a dangerous place in the region in which the vehicle is currently traveling (YES in step S266), the decision section 152 sets the priority of the vehicle 130 that has transmitted the information in step S261 to "high" or "medium" (step S267), and proceeds to step S264.

[0217] In a case where there is no dangerous place in the region in which the vehicle is currently traveling (NO in step S266), the decision section 152 refers to the database on dangerous scenarios stored in the storage control section 153, based on the vehicle position information and the sensing information (step S268). Specifically, the decision section 152 refers to whether the region in which the vehicle 130 is currently traveling is within the database of the storage control section 153, based on the vehicle position information and the sensing information. Here, the dangerous scenario refers to a scenario in which the vehicle 130 is predicted to have a risk of suddenly running off the road, in a region in which it is difficult for the vehicle 130 to perform automatic driving. For example, the dangerous scenario refers to a case where a bicycle is traveling on the lane in which the vehicle 130 is traveling, a case where a bicycle is approaching the vehicle 130 from the rear, and a case where there is a risk of suddenly running off the road because there are many dead angles in which a vehicle is stopped on the road, and the like.

[0218] The decision section 152 determines whether the dangerous scenario is included in the region in which the vehicle is currently traveling, based on the vehicle position information and the sensing information (step S269).

[0219] In a case where the dangerous scenario is included in the region in which the vehicle is currently traveling (YES in step S269), the decision section 152 sets the priority of the vehicle 130 that has transmitted the information in step S261 to "high" or "medium" (step S270), and proceeds to step S264.

[0220] In a case where the dangerous scenario is not included in the region in which the vehicle is currently traveling (NO in step S269), the decision section 152 sets the priority of the vehicle 130 that has transmitted the information in step S261 to "low" (step S271), and proceeds to step S264. In this way, the information processing apparatus 150 determines the priority, and ends this process. In the present embodiment, the priority is divided into three levels of "high", "medium", and "low", but can be two levels, or four or more levels.

[0221] In addition, the degree of danger differs depending on the area in which the vehicle 130 is traveling, the climate of the area, and the like. Therefore, for example, in a case where the vehicle 130 is traveling on a road on which the width of the road surface is narrow, in a case where rain or fog occurs, and the like, the determination section 152 can set the priority of step S267 and step S270 to "high", can set the priority of step S263 to "super high", and can set the priority of step S267 to "medium", for example. The determination section 152 can obtain climate information of the area in which the vehicle is traveling from an external server via the communication section 154.

[0222] Thus, by the information processing method, the information processing apparatus 150, and the information processing system 101, when the operator monitors a plurality of vehicles 130 that are traveling, the degree of danger is calculated for each vehicle 130 based on the sensing information that indicates the state inside and outside each vehicle 130. Based on this degree of danger, the information processing apparatus 150 presents the vehicle 130 for which the degree of danger is higher as a vehicle 130 that is more highly prioritized for monitoring. The terminal apparatus 170 presents the image of the vehicle 130 for which the priority is high with a large area, or presents only the vehicle 130 for which the priority is above a threshold value, or presents detailed information of all the vehicles 130, or presents as much information of the vehicle 130 as possible, or presents the image of the vehicle 130 for which the priority is the highest with a color that emphasizes the vehicle 130. Therefore, the operator only needs to monitor the plurality of vehicles 130 that are presented on the terminal apparatus 170, in other words, can more accurately grasp the vehicle 130 that needs to be monitored. As a result, by monitoring a plurality of vehicles, the safety of the plurality of vehicles can be further ensured.

[0223] (Other Modified Examples)

[0224] The information processing method, the information processing apparatus 5, 150, and the information processing system 1, 101 according to the embodiments 1, 2 of the present disclosure are described above, but are not limited to the above-described embodiments 1, 2.

[0225] For example, in the information processing method, the information processing apparatus 5, and the information processing system 1 according to the above-described embodiment 1, an example in which the form of presentation for monitoring or controlling the vehicle 3 is determined for each passenger is described, but the form of presentation can also be determined for each degree of congestion of the vehicle 3. For example, the detection processing section 51 generates congestion information by detecting the number of passengers in the vehicle 3, the number of empty seats, and the like based on the inside and outside images obtained from the fourth obtaining section 34. The determination section 52 determines the priority based on the congestion information. The presentation control section 55 controls the form of presentation based on the priority.

[0226] Further, in the information processing method, the information processing apparatus 5, 150, and the information processing system 1, 101 according to the above-described Embodiments 1, 2, in a case where a plurality of vehicles 3, 130 are determined to have the same priority, the information processing apparatus 5, 150 can notify the vehicles 3, 130 of the message, and broadcasting can be performed in the vehicles 3, 130 according to the notification.

[0227] Further, in the information processing method, the information processing apparatus 5, 150, and the information processing system 1, 101 according to the above-described Embodiments 1, 2, an image prompted according to the form of the priority can be emphasized on a portion related to information used when the priority is determined, or can be prompted in a distinguishable manner. Specifically, the information used when the priority is determined can be the detected object. For example, in Embodiment 1, a passenger on the displayed image can be emphasized. Further, in Embodiment 2, a pedestrian or a vehicle on the displayed image can be emphasized. As a form of emphasized display, it can be the color, pattern, or addition of a frame of the object, or the blinking of the object, and the like.

[0228] Further, each processing section included in the information processing method, the information processing apparatus 5, 150, and the information processing system 1, 101 according to the above-described Embodiments 1, 2 is typically implemented as an integrated circuit, that is, an LSI. It can be implemented in a manner of being monolithic, or including a part or all of the sections.

[0229] Further, the integrated circuit is not limited to the LSI, and can be implemented with a dedicated circuit or a general-purpose processor. It can also be implemented using an FPGA (Field Programmable Gate Array) that is programmable after the LSI is manufactured, or a reconfigurable processor in which a connection and a setting of circuit units inside the LSI are reconfigurable.

[0230] In addition, in each of the above-described Embodiments 1, 2, each constituent element is constituted by a dedicated hardware or implemented by executing a software program appropriate for each constituent element. Each constituent element can be implemented by a program execution section such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

[0231] Further, the numbers employed in the above-described Embodiments 1, 2 are merely examples for specifically describing the present disclosure, and the Embodiments 1, 2 of the present disclosure are not limited by the numbers shown.

[0232] Further, the division of the functional blocks in the block diagrams is an example, and a plurality of functional blocks can be implemented as one functional block, or one functional block can be divided into a plurality of functional blocks, or a part of the functions can be transferred to other functional blocks. Further, the functions of a plurality of functional blocks having similar functions can be processed in parallel or time-division by a single hardware or software.

[0233] Further, the order of execution of the respective steps in the flowchart is an example shown for the sake of concreteness of the present disclosure, and the order can be other than described. Further, a part of the steps can be executed simultaneously (in parallel) with other steps.

[0234] The information processing method, the information processing apparatus 5, 150, and the information processing system 1, 101 according to the one or more aspects described above are described according to Embodiments 1, 2, but the Embodiments 1, 2 of the present disclosure are not limited to the forms. Various modifications conceived by those skilled in the art within the scope of the object of the present disclosure are included in the one or more aspects, or forms in which the constituent elements in different embodiments are combined and constructed.

[0235] In addition, the processing performed by the detection processing section 51, 151 and the decision section 52, 152 can use machine learning, as described in Embodiments 1, 2. The machine learning can include, for example, teacher learning that learns the relationship between input and output using teacher data to which a label (output information) is assigned with respect to input information, unsupervised learning that constructs a data structure by only input without a label, semi-supervised learning that uses either of a labeled and an unlabeled, reinforcement learning that obtains feedback (reward) with respect to an action selected based on an observation result, and thus can obtain a continuous action that can obtain the most reward, and the like. Further, as a specific method of machine learning, there are neural networks (including deep learning using a multi-layered neural network), genetic programming, decision trees, Bayesian networks, support vector machines (SVM), and the like. In the present disclosure, any one of the specific examples described above can be used.

[0236] The present disclosure can be applied to an apparatus for remotely operating an autonomous vehicle, a terminal apparatus for prompting a state of an autonomous vehicle, or a system including the same. In particular, it is used for an autonomous bus, an autonomous taxi, an autonomous trolley, and the like.

Claims

1. An information processing method comprising: obtaining section obtains sensing data and past vehicle data, the sensing data being data obtained from sensors of vehicles, the past vehicle data being data stored in a database; an object detection processing section performs object detection processing on each of the vehicles using each of the sensing data, and a decision section decides a form of presentation for monitoring or controlling the vehicles by an operator using results of the object detection processing and the past vehicle data, and controls the presentation by a presentation control section in the decided form, wherein in each of the object detection processing on each of the vehicles, an object outside a vehicle in motion and a position of the object are detected, and in the decision of the form of the presentation, the decision section calculates a degree of danger outside the vehicle for each of the vehicles using the detected object and the position of the object, decides a priority of the vehicles according to the degree of danger outside the vehicle calculated for each of the vehicles to preferentially present the presentation to the operator, and decides the form of the presentation to the operator according to the decided priority, the form being a form of presenting a vehicle with a higher calculated degree of danger among the vehicles as a vehicle to be preferentially monitored, the past vehicle data including a place where a dangerous situation has occurred, an explanatory text of the dangerous situation, a recommended operation corresponding to the explanatory text, and a degree of danger corresponding to the explanatory text.

2. An information processing apparatus comprising: an obtaining section obtains sensing data and past vehicle data, the sensing data being data obtained from sensors of vehicles, the past vehicle data being data stored in a database; an object detection processing section performs object detection processing on each of the vehicles using each of the sensing data, and a decision section decides a form of presentation for monitoring or controlling the vehicles by an operator using results of the object detection processing and the past vehicle data, and a presentation control section that controls the presentation by the presentation device in accordance with the form decided by the deciding section, wherein in each of the object detection processes for each of the vehicles, an object outside the vehicle and a position of the object are detected, the deciding section calculates a degree of danger outside the vehicle using the detected object and the position of the object for each of the vehicles, the deciding section decides a priority of the vehicles in accordance with the degree of danger outside the vehicle calculated for each of the vehicles to preferentially present the object to the operator, and the deciding section decides the form of the presentation to the operator in accordance with the decided priority, the form being a form in which a vehicle with a higher calculated degree of danger among the vehicles is presented as a vehicle to be preferentially monitored, and the past vehicle data includes a place where a dangerous situation has occurred, a description of the dangerous situation, i.e., a label, a recommended operation corresponding to the label, and a degree of danger corresponding to the label.

Images