Information processing method and information processing device

The information processing method prioritizes vehicles based on risk levels using sensing data from multiple vehicles, addressing the challenge of simultaneous remote control and monitoring, thereby reducing operator burden and ensuring safety.

JP7576781B2Active Publication Date: 2024-11-01PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023076684
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-07-19
Filing Date
2023-05-08
Publication Date
2024-11-01
Estimated Expiration
2039-03-14

AI Technical Summary

Technical Problem

Conventional technologies face challenges in safely managing multiple vehicles remotely due to the difficulty of a single operator controlling or monitoring multiple vehicles simultaneously, as existing systems are designed for single-vehicle operation or fail to prioritize vehicles effectively.

Method used

An information processing method and device that utilize sensing data from multiple vehicles to perform object detection, determine priorities based on risk levels, and present monitoring information to operators in a prioritized manner, reducing the burden on operators while ensuring safety.

Benefits of technology

The method enables efficient monitoring and control of multiple vehicles by prioritizing high-risk vehicles, thereby reducing operator workload and enhancing safety by ensuring critical vehicles are addressed promptly.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007576781000001
    Figure 0007576781000001
  • Figure 0007576781000002
    Figure 0007576781000002
  • Figure 0007576781000003
    Figure 0007576781000003
Patent Text Reader

Abstract

To provide an information processing method and information processor capable of alleviating a load on an operator who implements the monitoring or controlling while ensuring safety for a plurality of vehicles by monitoring or controlling the plurality of vehicles.SOLUTION: An information processing method includes: obtaining, from each of a plurality of vehicles, each of sensor data obtainable from sensors provided in the plurality of vehicles; performing object detection processes each using each of the sensor data; determining a mode of a presentation for monitoring or controlling the vehicles, using respective results of the object detection processes; and controlling the presentation on a terminal device 7 according to the determined mode.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an information processing method and an information processing device. [Background technology]

[0002] In recent years, technologies that allow vehicles to operate autonomously have emerged, and research and development into this technology has become more active. With the development of autonomous driving technology in the future, unmanned autonomous vehicles are expected to become more widespread. From the perspective of safety, the operation of unmanned autonomous vehicles requires that the vehicle be monitored by an operator in a remote location. Several technologies for monitoring vehicles have been proposed.

[0003] For example, Patent Document 1 discloses a technology in which, when a vehicle enters a remote control target area, a remote control request is made to a control center, and the control center determines a remote control device to remotely control the vehicle that issued the remote control request.

[0004] Moreover, Patent Document 2 discloses a technique for correcting the departure time of a train based on the number of passengers entering a station. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-147626 A [Patent Document 2] JP 2015-182623 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the conventional technology disclosed in Patent Document 1, when multiple vehicles enter an area subject to remote control, the multiple vehicles become remote control targets at the same time. However, it is difficult for one operator to remotely control multiple vehicles at the same time.

[0007] Furthermore, the conventional technology disclosed in Patent Document 2 is a technology aimed at an operator who is in a single vehicle, and does not envisage a single operator managing multiple vehicles at the same time.

[0008] Therefore, the present disclosure provides an information processing method and information processing device that can ensure the safety of multiple vehicles by monitoring or controlling the multiple vehicles, while reducing the burden on the operator who performs the monitoring or control. [Means for solving the problem]

[0009] In order to achieve the above object, an information processing method according to an embodiment of the present disclosure is provided, , and stored in the database Each of the past vehicle data The acquisition department and performing an object detection process for each of the plurality of vehicles using one of the sensing data. The detection processing unit and determining a presentation manner for an operator who monitors or controls the plurality of vehicles using the results of each of the object detection processes and the past vehicle data. The decision section and performing the presentation of the presentation device according to the determined aspect. The presentation control unit control This includes: External objects and positions of the objects in each of the plurality of vehicles while the vehicles are moving When the determination unit detects Calculating a degree of danger outside the plurality of vehicles for each of the plurality of vehicles using the object and the position of the object; The determination unit is determining a plurality of priorities corresponding to each of the plurality of vehicles based on the risk calculated for each of the plurality of vehicles so that the presentation is given priority to the operator; The determination unit is According to the determined priority of each of the plurality of vehicles, the manner of presentation is determined such that the vehicle having a higher calculated risk level is given priority as a vehicle to be monitored. The past vehicle data includes a location where a dangerous situation occurred, a label that is a description of the dangerous situation, a recommended operation corresponding to the label, and a risk level corresponding to the label. .

[0010] Furthermore, these comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized by any combination of the system, the method, the integrated circuit, the computer program, and the recording medium. Effect of the Invention

[0011] According to the information processing method etc. disclosed herein, it is possible to ensure the safety of a plurality of vehicles by monitoring or controlling the plurality of vehicles, while reducing the burden on the operator who performs the monitoring or control. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating an information processing system according to the first embodiment. [Figure 2A] FIG. 2A is a diagram illustrating priority information according to the first embodiment. [Figure 2B] FIG. 2B is a diagram illustrating a database according to the first embodiment. [Figure 2C] FIG. 2C is a diagram illustrating a database according to the first embodiment. [Diagram 3] FIG. 3 is a sequence diagram illustrating an operation of the information processing system according to the first embodiment. [Figure 4A] FIG. 4A is a diagram illustrating a presentation by the terminal device according to the first embodiment. [Figure 4B] FIG. 4B is a diagram illustrating a presentation by the terminal device according to the first embodiment. [Figure 4C] FIG. 4C is a diagram illustrating a presentation by the terminal device according to the first embodiment. [Figure 4D] FIG. 4D is a diagram illustrating a presentation by the terminal device according to the first embodiment. [Figure 4E] FIG. 4E is a diagram illustrating a presentation by the terminal device according to the first embodiment. [Diagram 5]FIG. 5 is a flow diagram illustrating a process of determining a priority in the operation of the information processing device according to the first embodiment. [Figure 6] FIG. 6 is a flow diagram illustrating calculation of the boarding / alighting completion time by using the database in the operation of the information processing device according to the first embodiment. [Figure 7] FIG. 7 is a block diagram illustrating an information processing system according to the second embodiment. [Figure 8A] FIG. 8A is a diagram illustrating a database according to the second embodiment. [Figure 8B] FIG. 8B is a diagram illustrating an example of a database according to the second embodiment. [Figure 8C] FIG. 8C is a diagram illustrating an example of a database according to the second embodiment. [Figure 9] FIG. 9 is a sequence diagram illustrating an operation of the information processing system according to the second embodiment. [Figure 10A] FIG. 10A is a diagram illustrating a presentation by a terminal device according to the second embodiment. [Figure 10B] FIG. 10B is a diagram illustrating a presentation by the terminal device according to the second embodiment. [Figure 10C] FIG. 10C is a diagram illustrating a presentation by the terminal device according to the second embodiment. [Figure 10D] FIG. 10D is a diagram illustrating a presentation by the terminal device according to the second embodiment. [Figure 10E] FIG. 10E is a diagram illustrating a presentation by the terminal device according to the second embodiment. [Figure 10F] FIG. 10F is a diagram illustrating a presentation by the terminal device according to the second embodiment. [Figure 11] FIG. 11 is a flow diagram illustrating an operation of determining a priority in the operation of the information processing device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] An information processing method according to one embodiment of the present disclosure acquires sensing data obtained from sensors in a plurality of vehicles from each of the plurality of vehicles, performs an object detection process using each of the sensing data, determines a presentation mode for monitoring or controlling the plurality of vehicles using a result of each of the object detection processes, and controls the presentation of a presentation device in accordance with the determined mode.

[0014] In this way, by presenting a method for monitoring or controlling (hereinafter collectively referred to as simply monitoring) a plurality of vehicles in a manner determined using the results obtained from each of the object detection processes, an operator monitoring a plurality of vehicles can easily grasp the vehicles to be monitored. In other words, the number of vehicles that the operator must simultaneously monitor can be reduced while reducing the risk of the operator overlooking a vehicle that he or she should monitor. Thus, by monitoring a plurality of vehicles, the safety of the plurality of vehicles can be ensured while the burden on the operator performing the monitoring can be reduced.

[0015] In addition, an information processing device according to one embodiment of the present disclosure includes an acquisition unit that acquires sensing data obtained from sensors in a plurality of vehicles from each of the plurality of vehicles, a detection processing unit that executes each of object detection processes using each of the sensing data, a determination unit that determines a presentation mode for monitoring or controlling the plurality of vehicles using a result of each of the object detection processes, and a presentation control unit that controls the presentation of a presentation device in accordance with the determined mode.

[0016] An information processing system according to an embodiment of the present disclosure includes an information processing device and the presentation device.

[0017] These information processing devices and information processing systems also provide the same advantageous effects as those described above.

[0018] In addition, an information processing method according to one embodiment of the present disclosure determines multiple priorities corresponding to the multiple vehicles respectively based on the results of each of the object detection processes, and determines the presentation manner in accordance with the multiple priorities.

[0019] In this way, the presentation device presents information for monitoring or controlling each vehicle according to the priority of each vehicle, so that an operator who monitors multiple vehicles can grasp which of the multiple vehicles should be given priority for monitoring. Therefore, the operator can monitor a vehicle with a higher priority than other vehicles, and the burden on the operator can be reduced while ensuring the safety of the vehicles.

[0020] In addition, in an information processing method according to one embodiment of the present disclosure, the states of the multiple vehicles are obtained from each of the multiple vehicles, and the multiple priorities are determined based on the results of each of the object detection processes and each of the states of the multiple vehicles.

[0021] In this way, by using not only the results of each object detection process but also the states of the multiple vehicles, the priority of each vehicle can be more appropriately determined. This makes it easier for an operator monitoring multiple vehicles to know which of the multiple vehicles should be monitored first. This makes it easier for the operator to monitor vehicles with higher priorities than other vehicles, making it possible to reduce the burden on the operator while ensuring the safety of the vehicles.

[0022] In addition, in an information processing method according to one embodiment of the present disclosure, the object detection process detects at least one of a passenger and an alighting passenger in a stopped vehicle, and determines the presentation manner depending on the detected at least one of the passenger and the alighting passenger.

[0023] Conventionally, it is difficult to control the timing of departure by determining that boarding or disembarking from a vehicle is complete. For example, it is difficult to determine from sensing data whether a person near a vehicle is a person planning to board. If an erroneous determination occurs, there is a risk of danger to passengers. On the other hand, it is possible for an operator who performs monitoring to control the timing of departure, but constantly monitoring the boarding and disembarking status of the vehicle increases the burden on the operator. In contrast, according to the present configuration, the mode of presentation for monitoring is controlled according to the passengers, so that the operator can easily grasp the vehicle in a situation that needs to be monitored. Therefore, when passengers board or disembark, the safety of multiple vehicles can be ensured while the burden on the operator can be reduced. Note that this configuration is effective even when departure is automatically controlled. This is because, as described above, departure is a situation in which danger may occur, and therefore the operator is required to monitor with priority.

[0024] In addition, in an information processing method according to one embodiment of the present disclosure, the object detection process detects an attribute or number of at least one of the boarding passengers and the disembarking passengers, and determines the presentation manner according to the detected attribute or number.

[0025] In this way, by presenting information in a manner that corresponds to factors related to the priority of monitoring when getting on and off, the operator can more accurately grasp the vehicles that need to be monitored.

[0026] In addition, an information processing method according to one embodiment of the present disclosure calculates, using the attribute or the number, a time until at least one of the passenger boarding the stopped vehicle and the disembarking passenger disembarking from the stopped vehicle is completed, and determines the presentation manner according to the calculated time.

[0027] In this way, by presenting the vehicle in a manner according to the boarding and alighting time, the vehicle can be presented to the operator according to the timing to be monitored. Therefore, the vehicle to be monitored can be presented in a more timely manner. For example, the vehicle with a shorter calculated time can be presented as the vehicle to be monitored preferentially over the vehicle with a longer calculated time.

[0028] In addition, in an information processing method according to one embodiment of the present disclosure, the object detection process detects an object outside the vehicle while it is moving and the position of the object, and determines the presentation mode according to the detected object and the position of the object.

[0029] Conventionally, there is a technology that controls the operation of a vehicle according to the state of an object outside the vehicle. However, the state of the object is diverse, and it is difficult to control the vehicle while assuming all possible situations. On the other hand, it is possible for the operator performing the monitoring to check the outside of the vehicle, but the burden increases if the operator constantly monitors the outside of multiple vehicles. In contrast, according to the present configuration, the mode of presentation for monitoring is controlled according to the object outside the vehicle, so that the operator can easily grasp the vehicle that is in the situation that needs to be monitored. Therefore, when an object is present outside the vehicle, the burden on the operator can be reduced while ensuring the safety of the multiple vehicles.

[0030] In addition, an information processing method according to one embodiment of the present disclosure uses the object and the object's position to calculate at least one of the difficulty of autonomous driving and the degree of danger outside the plurality of vehicles for each of the plurality of vehicles, and determines the presentation manner based on at least one of the calculated difficulty and danger.

[0031] In this way, by presenting the vehicles according to at least one of the degree of difficulty of autonomous driving and the degree of danger outside the vehicle, the operator can more accurately grasp the vehicles to be monitored. For example, vehicles with a relatively high degree of difficulty of autonomous driving or the degree of danger outside the vehicle can be presented as vehicles to be monitored with priority.

[0032] In addition, in an information processing method according to an embodiment of the present disclosure, the presentation includes displaying an image showing at least one of an entrance and an exit of each of the plurality of vehicles.

[0033] This allows the operator to easily check for safety when getting on and off the vehicle.

[0034] In addition, in the information processing method according to one embodiment of the present disclosure, the presentation includes displaying an image showing the outside of each of the plurality of vehicles.

[0035] This allows the operator to easily check the safety of the exterior of the vehicle.

[0036] In addition, in the information processing method according to the embodiment of the present disclosure, determining the presentation manner includes increasing visibility of the presentation as the priority becomes higher.

[0037] In this way, by increasing the visibility of a vehicle with a higher priority, the operator can quickly check the information of the vehicle to be monitored.

[0038] In addition, in the information processing method according to an embodiment of the present disclosure, determining the presentation manner includes increasing an amount of the presented information as the priority becomes higher.

[0039] In this way, by presenting a larger amount of information the higher the priority, the operator can get a detailed understanding of the information about the vehicle to be monitored.

[0040] In addition, since the amount of information presented is smaller as the priority is lower, it is possible to lower the frame rate or image quality of the vehicle information presented to the operator, thereby reducing the network load between the device that realizes this information processing method, the vehicle, and the presentation device.

[0041] In addition, in the information processing method according to an embodiment of the present disclosure, determining the presentation manner includes emphasizing the presentation as the priority is higher.

[0042] In this way, by emphasizing the display of higher priorities, the operator can quickly identify vehicles to be monitored.

[0043] In addition, in the information processing method according to the embodiment of the present disclosure, determining the presentation mode includes starting the presentation when the priority becomes higher than a threshold.

[0044] In this way, by presenting only vehicles whose priority is equal to or exceeds a threshold, the operator can focus on the vehicles that should be monitored.

[0045] In addition, in the information processing method according to the embodiment of the present disclosure, determining the presentation mode includes starting the presentation when the priority becomes higher than a threshold.

[0046] In this way, by presenting vehicles in descending order of priority, the operator can quickly identify the vehicle to be monitored. Also, the operator can check in advance the next vehicle to be monitored, which makes the monitoring work more efficient.

[0047] In addition, in an information processing method according to one embodiment of the present disclosure, the calculation of the priority further includes at least one of the vehicle position, the time corresponding to the vehicle position, a history of the time required for passengers and disembarkers to complete boarding and disembarking, and weather information corresponding to the vehicle position.

[0048] This makes it possible to more appropriately set the priorities of the multiple vehicles, thereby allowing the presentation device to more appropriately present the vehicles to be monitored.

[0049] Furthermore, these comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized by any combination of the system, the method, the integrated circuit, the computer program, and the recording medium.

[0050] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, components, arrangement and connection forms of the components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in an independent claim showing a top concept are described as optional components.

[0051] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In each drawing, the same reference numerals are used for substantially the same configurations, and duplicated explanations are omitted or simplified.

[0052] An information processing method, an information processing device, and an information processing system according to embodiments of the present disclosure will be described below.

[0053] (Embodiment 1) [composition] FIG. 1 is a block diagram illustrating an information processing system 1 according to the first embodiment.

[0054] The information processing system 1 includes a vehicle 3, an information processing device 5, and a terminal device 7.

[0055] The vehicle 3 is an autonomous vehicle 3 that can travel by remote control from a person remote from the vehicle 3, and can travel autonomously when not remotely controlled. The vehicle 3 is, for example, a bus, a taxi, a train, etc. In this embodiment, a bus is assumed as an example of the vehicle 3, and the following embodiment is described assuming that the vehicle 3 is stopped at a stop (bus stop). In the following, a person who gets on or plans to get on the vehicle 3 is referred to as a passenger, and a person who gets off or plans to get off the vehicle 3 is referred to as a disembarking passenger. The term "boarding and disembarking passengers" refers to both boarding passengers and disembarking passengers.

[0056] The vehicle 3 has a first acquisition unit 31, a second acquisition unit 32, a third acquisition unit 33, a fourth acquisition unit 34, and a communication unit 35.

[0057] The first acquisition unit 31 is a device that acquires information indicating boarding / alighting start times, and is, for example, a clock. Specifically, the boarding / alighting start time indicates the time when passengers start boarding / alighting on the vehicle 3 while the vehicle 3 is stopped at a stop. The first acquisition unit 31 transmits information indicating the boarding / alighting start times to the information processing device 5. Note that the boarding / alighting start time may be the time when the vehicle 3 stops at the stop by autonomous driving.

[0058] The second acquisition unit 32 acquires vehicle position information (e.g., Global Positioning System (GPS) information) indicating the current position of the vehicle 3. The second acquisition unit 32 transmits the acquired vehicle position information to the information processing device 5 and the terminal device 7. The current position of the vehicle 3 is an example of the position of the vehicle 3.

[0059] The third acquisition unit 33 is a device that acquires passenger images showing images of passengers getting on and off the vehicle 3, and is, for example, a camera. Specifically, the third acquisition unit 33 generates passenger images by capturing images of people (e.g., passengers) waiting at a stop of the vehicle 3 or capturing images of people (e.g., disembarking passengers) standing up inside the vehicle. The third acquisition unit 33 transmits the passenger images to the information processing device 5. The third acquisition unit 33 is an example of a sensor. The passenger images are also an example of sensing data.

[0060] The fourth acquisition unit 34 is a device that acquires vehicle interior and exterior images showing the inside and outside of the vehicle, and is, for example, a camera. The vehicle interior and exterior images include information on the outside of the vehicle, i.e., pedestrians, vehicles, bicycles, etc. outside the vehicle 3, information on passengers inside the vehicle, and information on passengers getting on and off near the boarding and alighting doors. The fourth acquisition unit 34 transmits the vehicle interior and exterior images to the terminal device 7. Note that the fourth acquisition unit 34 may be shared with the third acquisition unit 33, and the passenger images may be vehicle interior and exterior images.

[0061] The communication unit 35 is a communication interface that transmits information indicating boarding / alighting start times, vehicle position information, and passenger images to the information processing device 5, and transmits information indicating the vehicle position information and vehicle interior / exterior images to the terminal device 7. In addition, when the vehicle 3 is remotely operated by a remote person, the communication unit 35 receives a control command for remote operation. The communication unit 35 outputs the control command to actuators, displays, and the like (not shown) that control the vehicle 3.

[0062] Here, since there are a plurality of vehicles 3, each vehicle 3 has a vehicle ID for identifying the vehicle 3. The communication unit 35 transmits information indicating the boarding / alighting start time, vehicle position information, and passenger images, each of which is assigned with the vehicle ID, to the information processing device 5, and transmits vehicle interior / exterior images, each of which is assigned with the vehicle ID, to the terminal device 7.

[0063] The information processing device 5 is a device capable of remotely controlling a plurality of autonomously driving vehicles 3 based on instructions from an operator at a remote location. The information processing device 5 has a detection processing unit 51, a determination unit 52, a memory control unit 53, a communication unit 54, and a presentation control unit 55.

[0064] The detection processing unit 51 executes each of the object detection processes. The object detection processes are processes for detecting at least one of passengers getting on and off the vehicle 3 that is stopped, and for detecting the attributes or number of at least one of the passengers getting on and off. Specifically, the detection processing unit 51 acquires passenger images from the third acquisition unit 33 of the vehicle 3, detects the number of passengers getting on and off, the attributes of each passenger, etc. based on the passenger images, and generates passenger information for each vehicle 3. The attributes of passengers indicate information such as adults, children, elderly people, etc. The detection processing unit 51 outputs the passenger information to the determination unit 52.

[0065] The determination unit 52 uses the results of each of the object detection processes to determine the presentation mode for monitoring or controlling the multiple vehicles 3. The presentation mode will be described in detail later.

[0066] Furthermore, when the determination unit 52 acquires each passenger information from the detection processing unit 51 and acquires the boarding / alighting start times and vehicle position information from the first acquisition unit 31 and the second acquisition unit 32 of the vehicle 3, respectively, the determination unit 52 acquires past boarding / alighting data from the database stored in the memory control unit 53. Specifically, the determination unit 52 requests past boarding / alighting data having similar passenger information, boarding / alighting start times, and vehicle position information from the memory control unit 53, respectively, and acquires similar past boarding / alighting data from the memory control unit 53.

[0067] The determination unit 52 acquires past boarding and alighting data from the storage control unit 53, and determines a priority level, which is an index for monitoring each vehicle 3 with priority. As shown in Fig. 2A, the determination unit 52 transmits each piece of priority level information, in which each calculated priority level is associated with a vehicle ID acquired from each vehicle 3, to the terminal device 7. Fig. 2A is a diagram illustrating an example of priority level information according to the first embodiment.

[0068] The memory control unit 53 stores past boarding and alighting data organized into a database. The boarding and alighting data is information such as boarding and alighting start times, stops, attributes of boarders and alighting passengers and the number of boarders and alighting passengers for each attribute, the time required for all passengers to complete boarding and alighting, and boarding and alighting times according to the attributes of boarders and alighting passengers. In response to a request from the determination unit 52, the memory control unit 53 searches for past boarding and alighting data similar to the boarding and alighting passenger information, and transmits the similar past boarding and alighting data to the determination unit 52. Note that the memory control unit 53 may store a trained model trained from past boarding and alighting data instead of the trained past boarding and alighting data. In this case, the memory control unit 53 transmits the trained model to the determination unit 52 in response to a request from the determination unit 52.

[0069] The boarding and alighting data is shown in Fig. 2B and Fig. 2C. Fig. 2B and Fig. 2C are diagrams illustrating boarding and alighting data according to the first embodiment. As shown in Fig. 2B, the boarding and alighting data is composed of the "time" when the vehicle 3 stops at the stop location, the "location" where the vehicle stops, the "attributes and number of passengers" and the "time required" for all passengers to complete boarding and alighting. As shown in Fig. 2C, the boarding and alighting data further includes the "boarding and alighting time" per person in the attributes of the passengers.

[0070] 1, the communication unit 54 is a communication interface that receives information indicating boarding / alighting start times, vehicle position information, and passenger images, and transmits priority information to the terminal device 7. The communication unit 54 is an example of an acquisition unit.

[0071] The presentation control unit 55 controls the mode of presentation of the terminal device 7 in accordance with the mode of presentation determined by the determination unit 52. That is, the presentation control unit 55 outputs a command to control the presentation of the terminal device 7 so that the terminal device 7 performs presentation in a mode based on the determined priority.

[0072] The terminal device 7 is an operation terminal for an operator to monitor the traveling state of the vehicle 3 and remotely operate the vehicle 3, and is, for example, a personal computer, a tablet terminal, etc. The terminal device 7 has a presentation unit 71, an operation unit 72, a timing unit 73, and a communication unit 74.

[0073] The presentation unit 71 is a monitor that presents vehicle interior and exterior images, vehicle position information, etc., for each vehicle 3. Specifically, the presentation unit 71 presents, for each vehicle 3, information related to the vehicle 3, such as an image of the interior of the vehicle, an image of the boarding / alighting entrance of the vehicle 3, an image of the exterior of the vehicle 3 (around the vehicle 3), the priority of the vehicle 3, the boarding / alighting completion time, the vehicle ID, the stopping location, etc.

[0074] The presentation unit 71 presents information about the vehicle 3 in various ways. For example, the presentation unit 71 presents information about the vehicle 3 with the highest priority using a large image, presents only information about vehicles 3 with priorities equal to or higher than a threshold among a plurality of vehicles 3, presents detailed information about vehicles 3 with high priority, presents information about vehicles 3 with the same size of images arranged in order of priority, and presents information about the vehicle 3 with the highest priority by highlighting the image with color.

[0075] The operation unit 72 is an input unit that accepts remote operation of the vehicle 3 by an operator. The operator operates the operation unit 72 when confirming that all passengers have boarded and alighted (checking that boarding and alighting have been completed) and departing the vehicle 3. The operation unit 72 is, for example, a touch panel, a keyboard, or the like. The operation unit 72 transmits a control command for remote operation to the vehicle 3 via the communication unit 74.

[0076] The timing unit 73 is, for example, a clock, which measures the time when the operator operates the operation unit 72 to transmit a control command to start the vehicle 3.

[0077] The communication unit 74 is a communication interface that transmits control commands to the vehicle 3 .

[0078] In the present embodiment, the clock unit 73 of the terminal device 7 is not an essential component of the terminal device 7, and the second acquisition unit 32 of the vehicle 3 is not an essential component of the vehicle 3. For this reason, the clock unit 73 and the second acquisition unit 32 may not be present, and either one of them may be provided.

[0079] [Operation] Next, a description will be given of the information processing method, the information processing device 5, and the operation of the information processing system 1 in this embodiment. Specifically, a description will be given of the operation of the information processing system 1 performed from when the vehicle 3 stops at a bus stop until the vehicle 3 departs after a certain period of time.

[0080] FIG. 3 is a sequence diagram illustrating the operation of the information processing system 1 according to the first embodiment.

[0081] 3, the vehicle 3 acquires information indicating boarding / alighting start times, and transmits the acquired information to the information processing device 5 (step S101). The vehicle 3 also acquires vehicle position information indicating the current position of the vehicle 3, and transmits the acquired information to the information processing device 5 and the terminal device 7 (step S102). The determination unit 52 of the information processing device 5 receives the information indicating the boarding / alighting start times and the vehicle position information received via the communication unit 54.

[0082] Furthermore, the vehicle 3 acquires passenger images indicating images capturing people getting on and off the vehicle 3, and transmits the acquired passenger images to the information processing device 5 (step S103). The detection processing unit 51 of the information processing device 5 receives the passenger images via the communication unit 54.

[0083] Meanwhile, the vehicle 3 acquires images of the interior and exterior of the vehicle, and transmits the images of the interior and exterior of the vehicle to the terminal device 7 (step S104).

[0084] The detection processing unit 51 generates passenger information by detecting the number of passengers, attributes of the passengers, and the like based on the passenger images obtained in step S103 (step S105). The detection processing unit 51 outputs the passenger information to the determination unit 52 (step S106).

[0085] The determination unit 52 requests the memory control unit 53 for past boarding / alighting data similar to the acquired boarding / alighting passenger information, boarding / alighting start times, and vehicle position information (step S107).

[0086] In response to a request from the determination unit 52, the storage control unit 53 searches for past boarding / alighting data similar to the boarding / alighting passenger information, boarding / alighting start time, and vehicle position information, and outputs the similar past boarding / alighting data to the determination unit 52 (step S108). The boarding / alighting data is composed of information such as the time when the vehicle 3 stops at a stop location, the stop location, the attributes of the boarding / alighting passengers, the number of boarding / alighting passengers for each attribute, the time required from when all boarding / alighting passengers start to when they finish boarding / alighting, and the boarding / alighting time for each attribute of the boarding / alighting passengers, as exemplified in Figs. 2B and 2C. Note that the storage control unit 53 may transmit a learned model to the determination unit 52 in response to a request from the determination unit 52, instead of outputting similar past boarding / alighting data to the determination unit 52 in response to a request from the determination unit 52.

[0087] The determination unit 52 acquires past boarding and alighting data from the memory control unit 53, and sets a monitoring priority for each vehicle 3 (step S109). The determination unit 52 determines a priority for each vehicle 3 based on passenger information, boarding and alighting start times, and vehicle position information. The priority will be described later. The determination unit 52 transmits priority information in which the set priority is linked to the vehicle ID to the terminal device 7 (step S110). Note that the determination unit 52 may acquire a trained model from the memory control unit 53, instead of acquiring past boarding and alighting data from the memory control unit 53, and set a monitoring priority for each vehicle 3. In this case, the determination unit 52 determines a priority for each vehicle 3 using the acquired trained model.

[0088] Here, the determination unit 52 determines the manner in which the information on the vehicle 3 to be presented by the terminal device 7 is more visible as the priority level increases, determines the manner in which the amount of information is increased as the priority level increases, or determines the manner in which the information is emphasized as the priority level increases. The determination unit 52 also determines the manner in which the information on the vehicle 3 is presented in order of priority. The presentation control unit 55 controls the presentation of the terminal device 7 in accordance with the determined manner, that is, causes the terminal device 7 to start presenting the information on the vehicle 3 when the priority level becomes higher than a threshold value.

[0089] Next, the terminal device 7 receives the vehicle interior / exterior images acquired in step S104 and the priority information acquired in step S110. The terminal device 7 performs presentation according to the priority (step S111). The terminal device 7 performs presentation in various modes according to the priority. Figs. 4A to 4E show examples of presentation modes to the presentation unit 71.

[0090] FIG. 4A is a diagram illustrating the presentation of the terminal device 7 according to the first embodiment. As shown in FIG. 4A, the presentation unit 71 presents the vehicle 3 with the highest priority in a large-area image (presented in a highly visible manner). Specifically, only the vehicle 3 with a predetermined priority (threshold) or higher among the multiple vehicles 3 is presented, and a large-area image is presented for the vehicle 3 with the highest priority. The information on the vehicle 3 with the first priority indicates the time required for all people to complete boarding and disembarking from the vehicle 3, an image related to the vehicle 3, a vehicle ID, and a stop location. The image related to the vehicle 3 is an image of the inside of the vehicle, an image of the boarding and disembarking door of the vehicle 3, an image of the outside of the vehicle, and the like. For example, for a vehicle 3 with a second or higher priority, simpler information than that of the vehicle 3 with the first priority, for example, only one type of image related to the vehicle 3, is presented, and for example, the frame rate of the image is lowered or the image quality is lowered. For a vehicle 3 with a priority lower than a predetermined priority, in this embodiment, for a vehicle 3 with a priority of 6 or higher, information related to the vehicle 3 is not presented to the presentation unit 71.

[0091] Fig. 4B is a diagram illustrating the presentation of the terminal device 7 according to the first embodiment. As shown in Fig. 4B, the presentation unit 71 presents only information about vehicles 3 with a predetermined priority or higher among a plurality of vehicles 3. In Fig. 4B, only information about vehicles 3 with the boarding / disembarking completion time approaching is presented preferentially. In this embodiment, information about two vehicles 3 is presented, and information about other vehicles 3 with third or lower priority is not presented.

[0092] Fig. 4C is a diagram illustrating a presentation of the terminal device 7 according to the first embodiment. As shown in Fig. 4C, the presentation unit 71 presents information about the vehicle 3 with the highest priority in a large-area image, and presents only the vehicles 3 with a predetermined priority or higher among the multiple vehicles 3. In Fig. 4C, an image related to the vehicle 3 is presented.

[0093] Fig. 4D is a diagram illustrating a presentation by the terminal device 7 according to the first embodiment. As shown in Fig. 4D, the presentation unit 71 presents all images related to the vehicles 3 (presenting an increased amount of information) in a size large enough that the operator can visually recognize the information related to each vehicle 3 by the presentation unit 71. In Fig. 4D, information related to the vehicles 3 arranged in order of priority is presented.

[0094] Fig. 4E is a diagram illustrating an example of a presentation by the terminal device 7 according to embodiment 1. As shown in Fig. 4E, the presentation unit 71 presents information about the vehicle 3 with the highest priority by highlighting it in color.

[0095] 4A to 4E are merely examples and are not limited thereto. For example, the presentation includes displaying an image showing at least one of the boarding entrance and the exit entrance of each of the multiple vehicles 3. The presentation also includes displaying an image showing the outside of each of the multiple vehicles 3.

[0096] Returning to the explanation of the sequence diagram in FIG.

[0097] When all passengers have boarded and disembarked from the vehicle 3, the terminal device 7 accepts a departure instruction from the operator (step S112). For example, the operator confirms that boarding and disembarking are complete via the terminal device 7. Then, if the operator confirms that boarding and disembarking are complete, he or she inputs a departure instruction to the terminal device 7. The terminal device 7 accepts the input departure instruction. Note that the confirmation of boarding and disembarking completion may be performed by the terminal device 7 using image recognition to confirm whether or not people are present near the boarding and disembarking door.

[0098] When the terminal device 7 receives a departure instruction from the operator, the terminal device 7 transmits a control command to the vehicle 3 to cause the vehicle 3 to depart (step S113).

[0099] Upon receiving the control command in step S113, the vehicle 3 starts moving (step S114).

[0100] Furthermore, when the terminal device 7 accepts the departure instruction in step S112, the timing unit 73 measures the boarding / alighting completion time in response to the acceptance of the departure instruction, and transmits information indicating the boarding / alighting completion time to the information processing device 5 (step S115).

[0101] Next, the determination unit 52 stores the information indicating the boarding / alighting start time received in step S101 in the memory control unit 53 (step S116), and updates the database of the memory control unit 53 (step S118).

[0102] Furthermore, the determination unit 52 calculates the required time from the information indicating the boarding / alighting start time received in step S116 and the information indicating the boarding / alighting completion time in step S115, stores it in the memory control unit 53, and updates the database of the memory control unit 53 (step S118). Note that instead of updating the database of the memory control unit 53 described above, the determination unit 52 may update the trained model using boarding / alighting data such as the time, location, attributes and number of boarding / alighting passengers, and the required time.

[0103] Furthermore, the determination unit 52 stores the passenger information acquired in step S106 in the memory control unit 53 (step S117), and updates the database of the memory control unit 53 (step S118).

[0104] This ends the processing of the information processing system 1. Note that in this information processing system 1, the updating of the database and the trained model of the storage control unit 53 in steps S115 to S118 is not essential processing, and such processing does not have to be performed.

[0105] Next, the operation of the information processing device 5 will be described with reference to Fig. 5. Here, the flow is performed when the vehicle 3 stops at a predetermined position. Fig. 5 is a flow diagram illustrating the determination of priority in the operation of the information processing device 5 according to the first embodiment.

[0106] 5, the detection processing unit 51 of the information processing device 5 receives passenger images from the vehicle 3 (step S161). The detection processing unit 51 detects the number of passengers, attributes of the passengers, and the like based on the passenger images, and generates passenger information. The detection processing unit 51 outputs the passenger information to the determination unit 52.

[0107] Moreover, the determination unit 52 of the information processing device 5 receives information indicating the boarding / alighting start time and vehicle position information from the vehicle 3 (step S162).

[0108] The determination unit 52 determines whether or not there are passengers getting on or off the vehicle 3 based on the passenger information (step S163). Specifically, the determination unit 52 determines whether or not the number of passengers included in the passenger information is one or more.

[0109] When passengers are present in the passenger information (Yes in step S163), the determination unit 52 calculates the boarding / alighting completion time. Specifically, based on the passenger information, past boarding / alighting data, etc., the determination unit 52 calculates the required time from when all passengers start to board / alight until they complete it.

[0110] The determination unit 52 adds the calculated required time to the boarding / alighting start time acquired in step S162, and calculates the boarding / alighting completion time for each vehicle 3 (step S164).

[0111] Next, the determination unit 52 sets a higher priority for the multiple vehicles 3 in the order of the boarding / alighting completion time closest to the current time. The determination unit 52 sets a priority for each vehicle 3 every time the vehicle 3 stops at a bus stop, and updates the priority of each vehicle 3 in the order of boarding / alighting completion time (step S165). The current time is an example of time.

[0112] On the other hand, if there are no passengers in the passenger information (No in step S163), the determination unit 52 sets the vehicle 3 in which it is determined that there are no passengers as the vehicle 3 with the highest priority (step S167). If there are multiple vehicles 3 with no passengers, the determination unit 52 sets all of those vehicles 3 as the vehicle 3 with the highest priority. The determination unit 52 calculates the boarding / alighting completion time in the same manner as in step S164, and then sets a priority for each of the other vehicles 3 and updates the priority of each vehicle 3 (step S168).

[0113] Next, the determination unit 52 transmits information indicating the priority of each vehicle 3 to the terminal device 7 via the communication unit 54 (step S166). Then, this flow of the information processing device 5 is terminated. In addition, the same process is performed at the next predetermined position where the vehicle 3 stops.

[0114] A flow for calculating the boarding / alighting completion time in step S164 in Fig. 5 will be specifically described with reference to Fig. 6. Fig. 6 is a flow diagram illustrating calculation of the boarding / alighting completion time using a database in the operation of the information processing device 5 according to the first embodiment. Note that the boarding / alighting completion time may be calculated using a trained model instead of using the database.

[0115] 6, the determination unit 52 refers to past boarding and alighting data stored in the memory control unit 53 (step S171). Specifically, the determination unit 52 requests the memory control unit 53 for past boarding and alighting data having similar passenger information, boarding and alighting start times, and vehicle position information, and obtains from the memory control unit 53 the past boarding and alighting data corresponding to the request.

[0116] The determination unit 52 judges whether or not there is past boarding / alighting data that is similar to the boarding / alighting passenger information, the boarding / alighting start time, and the vehicle position information (step S172). For example, if there is similarity, it means that at least one of the boarding / alighting passenger information, the boarding / alighting start time, and the vehicle position information matches the past boarding / alighting data, for example, the boarding / alighting start time is close in time, such as within one hour from the boarding / alighting start time.

[0117] If there is past boarding / alighting data similar to the passenger information (Yes in step S172), the decision unit 52 adopts the time required from when all passengers start boarding / alighting to when they finish (past required time) as the current time required from when all passengers start boarding / alighting to when they finish (step S173).

[0118] The determination unit 52 adds the required time adopted in step S173 to the boarding start time to calculate the boarding / alighting completion time (step S174), and then this flow ends.

[0119] On the other hand, if there is no past boarding / alighting data similar to the passenger information (No in step S172), the determination unit 52 extracts the attributes of the passengers and the boarding / alighting times corresponding to each of the attributes of the passengers from the past boarding / alighting data stored in the memory control unit 53 (step S175).

[0120] The determination unit 52 calculates the time required for all passengers to complete boarding / alighting from the start of boarding / alighting based on step S175 (step S176). The determination unit 52 proceeds to step S174 and calculates the boarding / alighting completion time. Then, this flow ends.

[0121] As described above, in the information processing method, the information processing device 5, and the information processing system 1, when an operator monitors a plurality of stopped vehicles 3, the time required from when all passengers start to board and disembark until completion can be calculated for each vehicle 3 based on the number of passengers and disembarking passengers and their attributes. The information processing device 5 assigns a higher priority to a vehicle 3 that takes a shorter time to complete boarding and disembarking based on the boarding and disembarking start time and the required time. The terminal device 7 presents vehicles 3 with high priority in a large image, presents only vehicles 3 with a priority equal to or higher than a threshold, presents information on vehicles 3 arranged in order of priority, presents information on vehicles 3 in detail, highlights the image of the vehicle 3 with the highest priority by coloring, and the like. For this reason, the operator can monitor only the plurality of vehicles 3 presented on the terminal device 7, that is, easily grasp the vehicles to be monitored. This reduces the number of vehicles 3 that the operator must monitor simultaneously, thereby reducing the risk of the operator overlooking the plurality of vehicles 3 to be monitored. As a result, the burden on the operator who monitors the plurality of vehicles 3 can be reduced while ensuring the safety of the plurality of vehicles 3 by monitoring the plurality of vehicles 3.

[0122] (Embodiment 2) [composition] In this embodiment, the following description will be given assuming that the vehicle 130 is performing autonomous driving.

[0123] Unless otherwise specified, the information processing method, information processing device 150, and information processing system 101 of this embodiment are the same as those of embodiment 1, and the same components are given the same symbols and detailed descriptions of the components are omitted.

[0124] FIG. 7 is a block diagram illustrating an information processing system 101 according to the second embodiment.

[0125] As shown in FIG. 7, the information processing system 101 includes a vehicle 130, an information processing device 150, and a terminal device 170.

[0126] The vehicle 130 has a fifth acquisition unit 135, a sixth acquisition unit 136, a seventh acquisition unit 137, and a communication unit 138.

[0127] The fifth acquisition unit 135 is a device that acquires vehicle state information indicating the state of the vehicle 130, and is, for example, an ECU (Electronic Control Unit) or the like. The vehicle state information includes information indicating the state of the vehicle 130, such as running, temporary stop, emergency stop, running speed, etc. The fifth acquisition unit 135 transmits the vehicle state information to the information processing device 150 and the terminal device 170.

[0128] The sixth acquisition unit 136 acquires vehicle position information (for example, GPS information) indicating the current position of the vehicle 130. The sixth acquisition unit 136 transmits the acquired vehicle position information to the information processing device 150 and the terminal device 170.

[0129] The seventh acquisition unit 137 executes each of the object detection processes. The object detection process detects an object and the position of the object outside the vehicle while it is moving. The seventh acquisition unit 137 acquires sensing information indicating the state of a person inside the vehicle 130, the position of an object outside the vehicle 130, and the like. The seventh acquisition unit 137 is an image sensor such as a camera that captures the inside and outside of the vehicle 130, a distance sensor such as a LIDAR (Laser Imaging Detection and Ranging), and the like. The seventh acquisition unit 137 transmits the sensing information to the information processing device 150 and the terminal device 170. The object outside the vehicle 130 is, for example, a person walking in front of the vehicle 130, a bicycle present behind the vehicle 130, and the like. The seventh acquisition unit 137 is an example of a sensor.

[0130] The communication unit 138 is a communication interface that transmits vehicle state information, vehicle position information, and sensing information to the information processing device 150. In addition, when the vehicle 130 is in a state where it is remotely operated by a remote person, the communication unit 138 receives a control command for remote operation.

[0131] The detection processing unit 151 of the information processing device 150 acquires sensing information from the seventh acquisition unit 137 of the vehicle 130, and generates vehicle interior / exterior information indicating the state of people inside the vehicle 130 and objects outside the vehicle 130, based on the sensing information. The detection processing unit 151 outputs the vehicle interior / exterior information to the determination unit 152.

[0132] When the determination unit 152 of the information processing device 150 acquires the vehicle interior / exterior information from the detection processing unit 151, the determination unit 152 acquires past vehicle data stored in the database of the storage control unit 153. Specifically, the determination unit 152 requests past vehicle data similar to the vehicle interior / exterior information from the storage control unit 153, and acquires the past vehicle data corresponding to the request from the storage control unit 153. Note that the determination unit 152 may request a trained model created based on the past vehicle data, instead of the past vehicle data stored in the database of the storage control unit 153, from the storage control unit 153, and acquire the trained model from the storage control unit 153.

[0133] The determination unit 152 calculates at least one of the difficulty of autonomous driving for each of the multiple vehicles and the external danger of the multiple vehicles 130 based on the sensing information, past vehicle data, and the like. The difficulty of autonomous driving is a parameter indicating the ease of driving the vehicle 130. For example, the difficulty of autonomous driving is high when the vehicle 130 is driving on a narrow road, or when the vehicle 130 is driving in an environment where rain, fog, or the like is occurring. The external danger of the multiple vehicles 130 is a parameter indicating the possibility that an accident will occur in the area in which the vehicle 130 is driving. For example, the danger is high in an area where there is a risk of people jumping out, an area under construction, or the like. Hereinafter, in this embodiment, the explanation will be mainly based on the danger. Note that the determination unit 152 may calculate at least one of the difficulty of autonomous driving for each of the multiple vehicles and the external danger of the multiple vehicles 130 using the acquired trained model instead of using the sensing information, past vehicle data, and the like.

[0134] The storage control unit 153 of the information processing device 150 is a storage device that stores past vehicle data. The vehicle data is information such as a label, a location corresponding to the label, an operation recommended corresponding to the label, and a risk level corresponding to the label. The label is an explanatory text such as "risk area for people to run out" or "under construction" for a location. In response to a request from the determination unit 152, the storage control unit 153 searches for past vehicle data similar to the vehicle interior / exterior information, and transmits the similar past vehicle data to the determination unit 152. Note that the risk level is changed arbitrarily, and for example, the risk level varies depending on the number of people present outside the vehicle 130. Note that the storage control unit 153 may store a learned model learned from past vehicle data instead of storing past vehicle data.

[0135] 8A, 8B, and 8C are diagrams illustrating past vehicle data according to the second embodiment. As shown in FIG. 8A, the past vehicle data is composed of a "location" where a dangerous situation occurred, a "label" which is an explanatory text of the dangerous situation, a "recommended operation" corresponding to the label, and a "risk level" corresponding to the label. As shown in FIG. 8B, the past vehicle data further includes a "state" that occurred in the vehicle 130, such as AEB (Autonomous Emergency Braking) activation or temporary stop, a "label" which is an explanatory text corresponding to the state, a "recommended operation" corresponding to the state, and a "risk level" corresponding to the state. As shown in FIG. 8C, the past vehicle data further includes a "situation" outside the vehicle 130 that occurred while the vehicle 130 was traveling, such as a pedestrian on the road or a bicycle behind the vehicle 130, a "label" which is an explanatory text corresponding to the state, a "recommended operation" corresponding to the state, and a "risk level" corresponding to the state.

[0136] As shown in FIG. 7, the presentation unit 171 of the terminal device 170 receives information about the vehicle 130, such as the priority of the vehicle 130, the status of the vehicle 130, an image of the interior of the vehicle 130, an image of the exterior of the vehicle 130, the vehicle ID, the current location of the vehicle 130, etc., via the communication unit 173, and presents the received information for each vehicle 130.

[0137] The presentation unit 171 of the terminal device 170 presents the vehicle 130 with the highest priority as a large-area image, presents only those vehicles 130 among multiple vehicles 130 whose priority is equal to or higher than a threshold, presents detailed information about the vehicles 130, presents information about the vehicles 130 arranged in order of priority using images of the same size, or highlights the image of the vehicle 130 with the highest priority by coloring it.

[0138] [Operation] Next, a description will be given of the information processing method, the information processing device 150, and the information processing system 101 according to this embodiment. Specifically, a description will be given of the operation of the information processing system 101 performed while the vehicle 130 is traveling.

[0139] FIG. 9 is a sequence diagram illustrating the operation of the information processing system 101 according to the second embodiment.

[0140] 9, the vehicle 130 acquires vehicle state information and transmits it to the information processing device 150 and the terminal device 170 (step S201). The decision unit 152 of the information processing device 150 receives the vehicle state information via the communication unit 154.

[0141] Furthermore, the vehicle 130 acquires vehicle position information indicating the current position of the vehicle 130, and transmits the vehicle position information to the information processing device 150 and the terminal device 170 (step S202). The determination unit 152 of the information processing device 150 receives the vehicle position information via the communication unit 154.

[0142] The vehicle 130 acquires sensing information indicating the state of people inside the vehicle 130, objects outside the vehicle, i.e., outside the vehicle 130, and positions of objects, and transmits the acquired sensing information to the information processing device 150 and the terminal device 170 (step S203). The determination unit 152 of the information processing device 150 receives the sensing information via the communication unit 154.

[0143] The detection processing unit 151 generates vehicle interior / exterior information indicating the state of people inside the vehicle 130, the position of objects outside the vehicle 130, and the like, based on the sensing information in step S203 (step S204). The detection processing unit 151 outputs the vehicle interior / exterior information to the determination unit 152 (step S205).

[0144] The determination unit 152 requests the storage control unit 153 for past vehicle data similar to the acquired vehicle interior / exterior information (step S206).

[0145] In response to a request from the determination unit 152, the storage control unit 153 searches for past vehicle data similar to the vehicle interior / exterior information, and outputs the similar past vehicle data to the determination unit 152 (step S207). The past vehicle data is exemplified in Fig. 8A to Fig. 8C, and is information such as a location, a location corresponding to a label, an operation recommended corresponding to the label, a risk level corresponding to the location, a state of the vehicle 130, and a situation corresponding to the location.

[0146] The determination unit 152 acquires past vehicle data from the memory control unit 153, and sets a priority for monitoring each vehicle 130 preferentially (step S208). The determination unit 152 sets the priority based on the degree of danger. The determination of the priority will be described later. The determination unit 152 transmits information indicating the determined priority of each vehicle 130 to the terminal device 170 (step S209).

[0147] Next, the terminal device 170 receives the vehicle state information and vehicle interior / exterior information acquired in step S201, and the information indicating the priority of each vehicle 130 determined in step S209. The terminal device 170 performs presentation according to the priority (step S210).

[0148] The terminal device 170 presents in various forms according to the priority. 10A to 10F show examples of the presentation on the presentation unit 171.

[0149] FIG. 10A is a diagram illustrating the presentation of the terminal device 170 according to the second embodiment. As shown in FIG. 10A, the presentation unit 171 presents the vehicle 130 with the highest priority in a large image. Specifically, only the vehicle 130 with a predetermined priority or higher among the multiple vehicles 130 is presented, and the vehicle 130 with the highest priority is presented in detail in a large image. The information on the vehicle 130 with the first priority indicates information on the vehicle 130 such as the priority of the vehicle 130, the state of the vehicle 130, an image of the inside of the vehicle 130, an image of the outside of the vehicle 130, a vehicle ID, and a current location of the vehicle 130. For example, for the vehicle 130 with the second or higher priority, simpler information than that of the vehicle 130 with the first priority, for example, only one type of image related to the vehicle 130, is presented. For the vehicle 130 with a priority lower than a predetermined priority, in this embodiment, the vehicle 130 with the sixth or higher priority is not presented with information on the vehicle 130 by the presentation unit 171.

[0150] Fig. 10B is a diagram illustrating a presentation by terminal device 170 according to embodiment 2. As shown in Fig. 10B, presentation unit 171 presents only information related to vehicles 130 having a predetermined priority or higher among a plurality of vehicles 130. In this embodiment, information related to two vehicles 130 is presented, and information related to other vehicles 130 having third or lower priority is not presented.

[0151] Fig. 10C is a diagram illustrating a presentation of the terminal device 170 according to the second embodiment. As shown in Fig. 10C, the presentation unit 171 presents information about the vehicle 130 with the highest priority in a large-area image, and presents only the vehicles 130 with a predetermined priority or higher among the multiple vehicles 130. In Fig. 10C, even detailed information about the vehicle 130 is presented.

[0152] Fig. 10D is a diagram illustrating a presentation by the terminal device 170 according to the second embodiment. As shown in Fig. 10D, the presentation unit 171 presents information on a plurality of vehicles 130 in a size that allows the operator to visually recognize the information on each vehicle 130 through the presentation unit 171. In Fig. 10D, information on the vehicles 130 arranged in order of priority is presented.

[0153] Fig. 10E is a diagram illustrating an example of a presentation by the terminal device 170 according to embodiment 2. As shown in Fig. 10E, the presentation unit 171 presents information related to the vehicle 130 with the highest priority by highlighting it in color.

[0154] Fig. 10F is a diagram illustrating an example of a presentation by terminal device 170 according to embodiment 2. As shown in Fig. 10F, when a plurality of vehicles 130 are traveling in a line, these vehicles 130 are presented collectively as one vehicle 130.

[0155] Note that the examples shown in FIGS. 10A to 10F are merely examples and are not limited to these.

[0156] Returning to the explanation of the sequence diagram in FIG.

[0157] The terminal device 170 receives an operation instruction from the operator (step S211). For example, the operator checks the operating status of each vehicle 130 from each vehicle 130 presented on the terminal device 170. Then, the operator inputs an operation instruction to the terminal device 170. When the terminal device 170 receives the operation instruction from the operator, it transmits a control command for each vehicle 130 presented on the terminal device 170 (step S212). The control command here is a command to control the traveling of the vehicle 130, such as an emergency stop of the vehicle 130, a temporary stop of the vehicle 130, a slowdown of the vehicle 130, and a steering operation of the vehicle 130.

[0158] When the vehicle 130 receives the control command in step S212, it operates and displays according to the control command (step S213). In this way, the processing of the information processing system 101 ends.

[0159] The terminal device 170 generates situation data indicating an evaluation according to the control command transmitted in step S212. For example, when slow driving is performed due to the jumping out at point A, the terminal device 170 generates situation data with the location as "point A", the label as "risk of jumping out", the recommended operation as "slow driving", and the risk level as "high". The situation data may be generated by an operator inputting through the operation unit 172. The situation data may also be automatically generated by the terminal device 170. The terminal device 170 transmits the generated situation data to the information processing device 150 (step S214). The storage control unit 153 of the information processing device 150 receives the situation data via the communication unit 154. The storage control unit 153 stores the situation data and updates the database of the storage control unit 153.

[0160] In addition, based on the vehicle state information, vehicle position information, and sensing information, the determination unit 152 generates driving information indicating, for example, the position where the vehicle 130 stopped and information about objects outside the vehicle 130 at that time, and outputs the driving information to the memory control unit 153 (step S215).

[0161] The storage control unit 153 acquires the travel information and updates the database of the storage control unit 153 (step S216). Then, the information processing system 101 ends this process.

[0162] In this information processing system 101, the update of the database of the storage control unit 153 in steps S214 to S216 is not essential processing, and such processing does not have to be performed.

[0163] Here, the setting of the priority in the information processing device 150 will be specifically described. Fig. 11 is a flow diagram illustrating the determination of the priority using a database in the operation of the information processing device 150 according to the second embodiment. Note that the priority may be determined using a trained model instead of using the database.

[0164] First, as shown in FIG. 11, the information processing device 150 acquires vehicle state information, sensing information, and vehicle position information (step S261).

[0165] The decision unit 152 of the information processing device 150 determines whether or not there is an abnormality in the vehicle 130 based on the vehicle state information (step S262). Specifically, the decision unit 152 determines whether or not information indicating a highly dangerous operation during the traveling of the vehicle 130, such as the activation of AEB, is included in the vehicle state information.

[0166] If there is an abnormality in the vehicle 130 (Yes in step S262), the decision unit 152 sets the priority of the vehicle 130 that transmitted the vehicle state information to "high" (step S263).

[0167] The determination unit 152 transmits information indicating the priority of the vehicle 130 set in step S263 to the terminal device 170 (step S264).

[0168] If there is no abnormality in the vehicle 130 (No in step S262), the determination unit 152 refers to a database of the memory control unit 153 related to the area stored in the memory control unit 153 (step S265). Specifically, the determination unit 152 refers to whether or not the location (area) where the vehicle 130 is currently traveling exists in the database, based on the vehicle position information.

[0169] The determination unit 152 determines whether or not there is a dangerous point in the area in which the vehicle is currently traveling, based on the vehicle position information (step S266). A dangerous point is, for example, a point where there is a risk of someone running out into the street, or a point under construction.

[0170] If there are any dangerous spots in the database in the area currently being traveled (Yes in step S266), the decision unit 152 sets the priority of the vehicle 130 that transmitted the information in step S261 to "high" or "medium" (step S267) and proceeds to step S264.

[0171] If there is no dangerous spot in the database in the area where the vehicle 130 is currently traveling (No in step S266), the decision unit 152 refers to a database related to dangerous scenes stored in the memory control unit 153 based on the vehicle position information and the sensing information (step S268). Specifically, the decision unit 152 refers to whether the area where the vehicle 130 is currently traveling is present in the database of the memory control unit 153 based on the vehicle position information and the sensing information. Here, the dangerous scene is an area where it is difficult for the vehicle 130 to perform automatic driving, and a scene where it is predicted that there is a risk of danger occurring to the vehicle 130. For example, the dangerous scene is a case where a bicycle is traveling on the roadway where the vehicle 130 is traveling, a case where a bicycle is approaching the vehicle 130 from behind, a case where there are many blind spots due to on-street parking and there is a risk of the bicycle jumping out, etc.

[0172] The decision unit 152 determines whether or not a dangerous scene is included in the area in which the vehicle is currently traveling, based on the vehicle position information and the sensing information (step S269).

[0173] If the sensing information includes a dangerous scene (Yes in step S269), the decision unit 152 sets the priority of the vehicle 130 that transmitted the information in step S261 to "high" or "medium" (step S270), and proceeds to step S264.

[0174] If the sensing information does not include a dangerous scene (No in step S269), the determination unit 152 sets the priority of the vehicle 130 that transmitted the information in step S261 to "low" (step S271), and proceeds to step S264. In this manner, the information processing device 150 determines the priority and ends this process. In this embodiment, the priority is divided into three levels, "high", "medium", and "low", but it may be two levels or four or more levels.

[0175] It is considered that the degree of danger differs depending on the area in which the vehicle 130 is traveling, the weather of the area, etc. For this reason, for example, when the vehicle 130 is traveling on a narrow road, or when rain, fog, etc. are occurring, the decision unit 152 may set the priority of steps S267 and S270 to "high", the priority of step S263 to "ultra-high", and the priority of step S267 to "medium". The decision unit 152 may obtain weather information of the area in which the vehicle is traveling from an external server via the communication unit 154.

[0176] As described above, in the information processing method, the information processing device 150, and the information processing system 101, when an operator monitors a plurality of vehicles 130 that are traveling, the degree of danger can be calculated for each vehicle 130 based on sensing information indicating the internal and external conditions of each vehicle 130. Based on this degree of danger, the information processing device 150 presents the vehicle 130 with a higher degree of danger outside the vehicle as a vehicle 130 that should be monitored with a higher priority. The terminal device 170 presents the vehicle 130 with a high priority by presenting it with a large image, presenting only the vehicles 130 with a priority equal to or higher than a threshold, presenting detailed information about all the vehicles 130, presenting information about the vehicles 130 as much as possible, or highlighting the image of the vehicle 130 with the highest priority by coloring it. For this reason, the operator can monitor only the plurality of vehicles 130 presented on the terminal device 170, that is, more accurately grasp the vehicle 130 that should be monitored. As a result, by monitoring a plurality of vehicles, the safety of the plurality of vehicles can be more assured.

[0177] (Other variations) The information processing methods, information processing devices 5, 150, and information processing systems 1, 101 according to the first and second embodiments of the present disclosure have been described above, but the first and second embodiments of the present disclosure are not limited to the above-described first and second embodiments.

[0178] For example, in the information processing method, information processing device 5, and information processing system 1 according to the above-mentioned first embodiment, an example has been described in which the mode of presentation for monitoring or controlling the vehicle 3 is determined depending on passengers getting on and off. However, the mode of presentation may be determined depending on the degree of congestion of the vehicle 3. For example, the detection processing unit 51 detects the number of passengers and the number of vacant seats in the vehicle 3 from the interior and exterior images acquired by the fourth acquisition unit 34, and generates congestion degree information. The determination unit 52 determines a priority based on the congestion degree information. The presentation control unit 55 controls the mode of the presentation based on the priority.

[0179] In addition, in the information processing methods, information processing devices 5, 150, and information processing systems 1, 101 relating to the above-mentioned embodiments 1 and 2, if the same priority is determined for multiple vehicles 3, 130, the information processing devices 5, 150 may notify the vehicles 3, 130 of this, and an announcement may be made within the vehicles 3, 130 based on the notification.

[0180] Furthermore, in the information processing methods, information processing devices 5, 150, and information processing systems 1, 101 according to the above-mentioned first and second embodiments, the image presented in a manner based on the priority may be presented with a portion related to the information used in determining the priority emphasized or distinguished. Specifically, the information used in determining the priority may be a detected object. For example, in the first embodiment, passengers getting on and off may be emphasized on the image. Furthermore, in the second embodiment, pedestrians or vehicles may be emphasized on the image. The manner of highlighting may include adding a color, pattern, or frame to the object, making the object blink, and the like.

[0181] Furthermore, the information processing methods, information processing devices 5 and 150, and information processing systems 1 and 101 according to the first and second embodiments are typically realized as LSIs, which are integrated circuits. These may be individually implemented as single chips, or may be integrated in part or in whole as a single chip.

[0182] The integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor. A field programmable gate array (FPGA) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections or settings of the circuit cells inside the LSI may also be used.

[0183] In each of the first and second embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0184] Furthermore, all the numbers used above are merely examples for specifically explaining the present disclosure, and the first and second embodiments of the present disclosure are not limited to the numbers shown as examples.

[0185] In addition, the division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as one functional block, one functional block may be divided into multiple blocks, or some functions may be transferred to another functional block. Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in a time-sharing manner by a single piece of hardware or software.

[0186] In addition, the order in which each step is performed in the flowchart is merely an example for specifically explaining the present disclosure, and an order other than the above may be used. In addition, some of the steps may be performed simultaneously (in parallel) with other steps.

[0187] The information processing method, information processing device 5, 150, and information processing system 1, 101 according to one or more aspects have been described based on the first and second embodiments, but the first and second embodiments of the present disclosure are not limited to the multiple aspects. As long as they do not deviate from the spirit of the present disclosure, various modifications conceived by a person skilled in the art to the first and second embodiments, and forms constructed by combining components in different embodiments may also be included within the scope of one or more aspects.

[0188] The processes performed by the detection processing units 51 and 151 and the determination units 52 and 152 may use machine learning as described in the first and second embodiments. Examples of machine learning include supervised learning, which learns the relationship between input and output using teacher data to which labels (output information) are assigned to input information, unsupervised learning, which builds a data structure only from unlabeled input, semi-supervised learning, which handles both labeled and unlabeled data, and reinforcement learning, which learns a continuous action that can obtain the most reward by obtaining feedback (reward) for an action selected from the observation result of a state. Specific examples of machine learning include neural networks (including deep learning using a multi-layered neural network), genetic programming, decision trees, Bayesian networks, and support vector machines (SVMs). In the present disclosure, any of the specific examples listed above may be used. [Industrial Applicability]

[0189] The present disclosure can be applied to a device for remotely operating an autonomous vehicle, a terminal device for displaying the status of an autonomous vehicle, or a system including these, particularly for autonomous buses, autonomous taxis, autonomous trains, etc. [Explanation of symbols]

[0190] 1, 101 Information Processing Systems 3, 130 vehicles 5, 150 Information processing equipment 7, 170 Terminal device (presentation device) 33 Third acquisition unit (sensor) 51, 151 Detection processing unit 52, 152 Decision Section 54, 154 Communication Department (Acquisition Department) 55, 155 Presentation control unit 137 7th Acquisition Unit (Sensor)

Claims

1. An acquisition unit acquires sensing data obtained from sensors installed in a plurality of vehicles and past vehicle data stored in a database, A detection processing unit executes an object detection process for each of the plurality of vehicles using one of the sensing data, A determination unit determines a presentation manner for an operator who monitors or controls the plurality of vehicles using each result of the object detection process and the past vehicle data; A presentation control unit controls the presentation of the presentation device in accordance with the determined aspect; When an external object and a position of the object are detected for each of the plurality of vehicles while the vehicle is traveling, the determination unit calculates a degree of risk outside the plurality of vehicles for each of the plurality of vehicles using the object and the position of the object; the determination unit determines a plurality of priorities corresponding to each of the plurality of vehicles based on the risk calculated for each of the plurality of vehicles so that the presentation is given priority to the operator; the determination unit determines the manner of presentation according to the priority of each of the determined plurality of vehicles, such that a vehicle having a higher calculated risk level is a vehicle to be monitored with a higher priority; The past vehicle data includes a location where a dangerous situation has occurred, a label that is a description of the dangerous situation, a recommended operation corresponding to the label, and a risk level corresponding to the label. Information processing methods.

2. The past vehicle data includes at least one of information regarding a recommended operation and information regarding a risk level. The information processing method according to claim 1 .

3. The past vehicle data is vehicle interior / exterior information generated based on the sensing data, and is past vehicle data similar to the vehicle interior / exterior information indicating the appearance of people present inside each of the plurality of vehicles and the appearance of objects present outside the vehicle. The information processing method according to claim 1 .

4. the presentation includes information regarding the recommended action; The information processing method according to claim 2 .

5. an acquisition unit that acquires sensing data obtained from sensors installed in a plurality of vehicles and past vehicle data stored in a database; a detection processing unit that executes an object detection process for each of the plurality of vehicles by using one of the sensing data; a determination unit that determines a presentation mode for an operator who monitors or controls the plurality of vehicles using each result of the object detection process and the past vehicle data; a presentation control unit that controls the presentation on the presentation device in accordance with the determined aspect; When an external object and a position of the object are detected for each of the plurality of vehicles while the vehicle is traveling, the determination unit calculates a degree of risk outside the plurality of vehicles for each of the plurality of vehicles using the object and the position of the object; the determination unit determines a plurality of priorities corresponding to each of the plurality of vehicles based on the risk calculated for each of the plurality of vehicles so that the presentation is given priority to the operator; the determination unit determines the manner of presentation according to the priority of each of the determined plurality of vehicles, such that a vehicle having a higher calculated risk level is a vehicle to be monitored with a higher priority; The past vehicle data includes a location where a dangerous situation has occurred, a label that is a description of the dangerous situation, a recommended operation corresponding to the label, and a risk level corresponding to the label. Information processing device.

Citation Information

Patent Citations

  • Particle width converting device for article conveyor

    JP1987027211A

  • Automatic train operation system, train operation support system, and train service management system

    JP2015182623A

  • Remote control system

    JP2017147626A

  • Vehicle control device, vehicle control method, and control program of vehicle control device

    JP2019101709A