Information processing method, information processing device, and information processing program

Abstract

To provide an information processing method, information processing device, and information processing program capable of assisting in identifying a factor that has necessitated remote handling.SOLUTION: An information processing method in accordance with the present disclosure causes a computer to acquire a first image showing a scene where a moving body is autonomously traveling normally and a second image showing a scene where the moving body is traveling under remote control because remote handling arises, and to output factor identification assist information, which assists in identifying a factor of remote handling, on the basis of the first image and second image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, an information processing device, and an information processing program. [Background technology]

[0002] A system for remotely controlling an autonomously moving object is known. For example, a system has been disclosed in which, when an exceptional event occurs that makes it difficult for the robot to move autonomously, the robot can be remotely controlled by a remote operator to deal with the exceptional event.

[0003] However, in the prior art, remote operators are not allocated in accordance with the needs of the service manager who is the main manager of the service using mobile devices. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-91134 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, a service manager who manages a service using a mobile device wants to minimize the number of remote control operations by a remote operator. However, in order to reduce the number of remote control operations, it is necessary to identify the cause of the need for remote response.

[0006] The present disclosure aims to provide an information processing method, an information processing device, and an information processing program that can assist in identifying the cause of the need for remote response. [Means for solving the problem]

[0007] In order to achieve the above object, the information processing method disclosed herein has a computer acquire a first image showing a scene in which a mobile object is autonomously driving normally and a second image showing a scene in which a remote response has occurred and the mobile object is driving by remote control, and output factor identification support information that supports identifying the factor of the remote response based on the first image and the second image. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to assist in identifying the cause of the need for remote response. Note that the effects described herein are not necessarily limited to those described herein, and may be any of the effects described in this specification. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a remote control system according to an embodiment. [Figure 2] FIG. 2 is a functional block diagram illustrating an example of a functional configuration of the moving body according to the embodiment. [Figure 3] FIG. 3 is a functional block diagram illustrating an example of the functional configuration of the service manager terminal according to the embodiment. [Figure 4] FIG. 4 is a functional block diagram illustrating an example of a functional configuration of the remote control terminal according to the embodiment. [Figure 5] FIG. 5 is a functional block diagram illustrating an example of a functional configuration of the server according to the embodiment. [Figure 6] FIG. 6 is a schematic diagram illustrating an example of a data configuration of mobile object management information according to the embodiment. [Figure 7] FIG. 7 is a schematic diagram illustrating an example of a data configuration of moving object speed information according to the embodiment. [Figure 8] FIG. 8 is a schematic diagram illustrating an example of heat map information according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram illustrating an example of first input information according to the embodiment. [Figure 10] FIG. 10 is a schematic diagram illustrating an example of a data configuration of remote control history information according to the embodiment. [Figure 11] FIG. 11 is a schematic diagram illustrating an example of second input information according to the embodiment. [Figure 12] FIG. 12 is a schematic diagram illustrating an example of cause identification support information according to the embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of the flow of information processing executed by the control unit of the server according to the embodiment. [Figure 14] FIG. 14 is a flowchart showing an example of the flow of information processing executed by the control unit of the server according to the embodiment. [Figure 15] FIG. 15 is a schematic diagram showing an example of cause identification support information according to the third modified example. [Figure 16] FIG. 16 is a schematic diagram showing an example of cause identification support information according to the fourth modified example. [Figure 17] FIG. 17 is a schematic diagram showing an example of cause identification support information according to the fifth modified example. [Figure 18] FIG. 18 is a schematic diagram showing an example of cause identification support information according to the sixth modified example. [Figure 19] FIG. 19 is a schematic diagram showing an example of cause identification support information according to the seventh modified example. [Figure 20] FIG. 20 is a block diagram showing an example of the hardware configuration of a remote control system according to the embodiment and the modification.

[0010] An information processing method, an information processing device, and an information processing program according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0011] Fig. 1 is a schematic diagram showing an example of a remote control system according to an embodiment. As shown in Fig. 1, the remote control system 1 includes a mobile object 10, a service administrator terminal 20, a remote control terminal 30, and a server 40. The mobile object 10, the service administrator terminal 20, the remote control terminal 30, and the server 40 are communicably connected via a network NW or the like.

[0012] The moving body 10 is capable of moving autonomously and is remotely controllable. The moving body 10 is, for example, an autonomously moving vehicle, a robot having a movement function and moving autonomously, etc. An autonomously moving vehicle is sometimes called an automatically driven vehicle.

[0013] In this embodiment, the mobile object 10 provides various services in a service provision area that is assigned to the mobile object 10 by a service manager P. For example, the mobile object 10 provides various services such as selling goods, road maintenance, construction work, etc. in a service provision area such as a factory, airport, or port. Furthermore, for example, the mobile object 10 provides services such as delivering goods in a service provision area such as a public road.

[0014] 1 shows an example in which multiple moving bodies 10 provide services in their respective predetermined areas. The moving bodies 10 autonomously travel within their respective areas and provide the predetermined services.

[0015] When the moving body 10 needs remote control by the remote operator Q, for example, when autonomous movement becomes difficult, the moving body 10 transmits remote control request information to the server 40. The remote control request information will be described in detail later.

[0016] The service manager terminal 20 is an information processing device operated by a service manager P. The service manager P is the entity that provides services using the mobile object 10. In other words, the service manager P is the provider or manager of the services provided by the remote control system 1. For example, when a mobile object 10 provides a transportation service for transporting goods, the service manager P creates and manages service specifications such as the area in which the mobile object 10 provides the service, the delivery points of goods within the area, the mobile object 10 that performs the service, and information about the driving rules of the mobile object 10, as well as managing the costs required for the service.

[0017] The remote control terminal 30 is an information processing device operated by a remote operator Q. The remote operator Q is an operator who remotely controls the moving body 10. The remote operator Q remotely controls the moving body 10 assigned to the remote operator Q by the server 40. Remote control tasks include, for example, obstacle avoidance and departure confirmation.

[0018] In this embodiment, the remote operator Q is a user who exists outside the local network to which the server 40 belongs, and is described as operating the remote control terminal 30 that exists outside the local network. However, the remote operator Q may also be a user who exists in the local network to which the server 40 belongs, and operate the remote control terminal 30 that exists in the local network.

[0019] The server 40 manages the mobile objects 10 and the remote operators Q included in the remote control system 1. Furthermore, based on remote control request information received from the mobile object 10, the server 40 determines the remote operator Q who will remotely control the mobile object 10 that is the sender of the remote control request information. Details of these processes will be described later.

[0020] Next, the functional configurations of the mobile object 10, the service administrator terminal 20, the remote control terminal 30, and the server 40 will be described.

[0021] FIG. 2 is a functional block diagram showing an example of the functional configuration of the moving object 10 according to the embodiment.

[0022] The mobile object 10 includes a communication unit 11, a sensor 12, a drive unit 13, a service providing unit 14, a memory unit 15, and a control unit 16. The communication unit 11, the sensor 12, the drive unit 13, the service providing unit 14, the memory unit 15, and the control unit 16 are connected to each other so as to be able to communicate with each other via a bus or the like.

[0023] The communication unit 11 is a communication interface with the remote control terminal 30, the server 40, and the remote control terminal 30, etc., via a network NW, etc.

[0024] The sensor 12 is a sensor that acquires information about the traveling environment of the mobile object 10. The traveling environment is, for example, observation information about the mobile object 10 and information about the surroundings of the mobile object 10. The sensor 12 is, for example, an internal sensor and an external sensor.

[0025] The internal sensor is a sensor that observes observation information. The observation information includes at least position information of the moving body 10. The position information may be a relative position or an absolute position. In this embodiment, a form in which the position information is an absolute position represented by latitude and longitude will be described as an example. The internal sensor includes, for example, a position sensor (GNSS (Global Navigation Satellite System), GPS (Global Positioning System)). The observation information may also include speed, acceleration, etc. In this case, the internal sensor further includes, for example, an inertial measurement unit (IMU), an acceleration sensor, a speed sensor, a rotary encoder, etc.

[0026] The external sensor observes information about the surroundings of the moving body 10. The external sensor is mounted on the moving body 10. The surrounding information is information that indicates the situation around the moving body 10. The surroundings of the moving body 10 is an area within a predetermined range from the moving body 10. This range is the observable range of the external sensor. This range may be set in advance.

[0027] The surrounding information is, for example, at least one of a captured image of the surroundings of the moving body 10 and distance information. The captured image is captured image data obtained by capturing an image. In the following description, the captured image data will be simply referred to as an image. The distance information is information indicating the distance from the moving body 10. The image includes a still image and a video.

[0028] The external sensor is, for example, an imaging device that captures an image by imaging, a distance sensor, etc. The distance sensor is, for example, a millimeter-wave radar, a laser sensor, a distance image sensor, etc. The captured image is, for example, digital image data in which a pixel value is defined for each pixel, or a depth map in which the distance from the sensor 12 is defined for each pixel, etc. The laser sensor is, for example, a two-dimensional LiDAR (Laser imaging Detection and Ranging) sensor installed parallel to a horizontal plane, or a three-dimensional LiDAR sensor.

[0029] The drive unit 13 is a device mounted on the mobile object 10 that drives the movement of the mobile object 10. The drive unit 13 is, for example, an engine, a motor, wheels, etc. The drive unit 13 is driven under the control of the control unit 16. For example, the control unit 16 determines the surrounding situation based on information obtained from the sensor 12, etc., and controls the amount of acceleration, the amount of braking, the steering angle, etc. For example, the control unit 16 controls the drive unit 13 of the mobile object 10 so that the mobile object 10 moves autonomously along a predetermined route within a predetermined area.

[0030] The service providing unit 14 is hardware for providing a service. For example, if the service provided by the mobile object 10 is a luggage transport service, the service providing unit 14 is a luggage locker.

[0031] The storage unit 15 stores various types of data. The storage unit 15 is, for example, a semiconductor memory element such as a RAM (Random Access Memory), a flash memory, a hard disk, an optical disk, etc. The storage unit 15 may be a storage device provided outside the mobile object 10. The storage unit 15 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN, the Internet, etc.

[0032] In this embodiment, the memory unit 15 stores a basic program for autonomous movement. The control unit 16 of the mobile object 10 executes control of the autonomous movement and the like using the basic program stored in the memory unit 15. The memory unit 15 also stores service information for the mobile object 10 to execute a service. For example, if the service is a parcel delivery service, the service information includes information such as a parcel pickup location and a parcel delivery destination. The control unit 16 of the mobile object 10 controls the service providing unit 14, the driving unit 13, and the like to execute the service using the service information stored in the memory unit 15.

[0033] The control unit 16 is a computer that executes information processing in the mobile object 10. The control unit 16 is realized by one or more processors. The control unit 16 may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. The control unit 16 may also be realized by a processor such as a dedicated IC, i.e., by hardware. The control unit 16 may also be realized by a combination of software and hardware. At least one of the one or more functional units included in the control unit 16 may be mounted on an external information processing device that is communicatively connected to the mobile object 10 via a network NW or the like.

[0034] The control unit 16 controls the driving unit 13 so that the moving body 10 moves autonomously.

[0035] For example, the control unit 16 controls the driving unit 13 to autonomously move along a route represented by route information within a predetermined area, using the captured image acquired by the sensor 12 and position information of the mobile object 10, etc. The route along which the mobile object 10 autonomously moves is specified in advance in mobile object management information (described later) managed by the server 40, for example, and map information of the real space including the route and the route is stored in advance in the storage unit 15 of the mobile object 10.

[0036] The control unit 16 controls the driving unit 13 so that the mobile object 10 travels along the route using the map information, information about the route, position information detected by the sensor 12, captured images acquired by the sensor 12, etc. The control unit 16 controls the driving unit 13, causing the mobile object 10 to travel autonomously along the route. The control unit 16 also provides services by controlling the service providing unit 14 based on service information stored in the memory unit 15.

[0037] Furthermore, when the control unit 16 detects a situation requiring remote control, it transmits remote control request information to the server 40.

[0038] A situation requiring remote control is, for example, a situation in which autonomous movement of the mobile object 10 is difficult. The control unit 16 can detect that a situation requiring remote control is present based on the detection results of the sensor 12, the driving state of the driving unit 13, and the provision state of the service providing unit 14. When the control unit 16 detects that a situation requiring remote control is present, it transmits remote control request information to the server 40.

[0039] The remote control request information includes a moving object ID (Identifier) ​​of the moving object 10, location information of the moving object 10, and request content information. The moving object ID is also called a vehicle ID.

[0040] The moving body ID is identification information of the moving body 10. The location information of the moving body 10 is information that indicates the location of the moving body 10. The location information of the moving body 10 is expressed, for example, by the latitude and longitude of the moving body 10. The location information of the moving body 10 may also be area identification information of the area in which the moving body 10 currently exists. The location information of the moving body 10 may also be information that can identify driving rules, such as driving speed, that are defined for the area in which the moving body 10 travels.

[0041] The request content information is information that indicates the content of the remote control request. For example, the request content information includes specific details of the situation in which autonomous movement is difficult, information that can identify travel rules such as travel speeds specified for the area in which the mobile object 10 travels, information that can be used by the server 40 to assign a remote operator Q, etc.

[0042] Examples of information that can be used by the server 40 to assign a remote operator Q include status information of the mobile body 10, the service status of the mobile body 10, the service delay status of the mobile body 10, and information regarding the content of remote control performed in response to a remote request.

[0043] The status information of the mobile body 10 is information that indicates, for example, the remaining battery powering the mobile body 10, whether or not the mobile body 10 has any cargo on board, etc. The service status of the mobile body 10 is information that indicates, for example, whether the mobile body 10 is moving to the delivery destination, moving to the delivery origin, or being sent out, etc. The service delay status of the mobile body 10 is information that indicates, for example, whether there is sufficient time, on time, delayed, delay time, etc., when the service provided by the mobile body 10 is cargo transport. Information regarding the content of remote control to be performed in response to a remote request is information that indicates, for example, obstacle avoidance, departure decision, etc.

[0044] The control unit 16 drives and controls the driving unit 13, the service providing unit 14, etc. in response to remote control by a remote control terminal 30 operated by an execution operator among the remote operators Q who is assigned as the remote operator Q by the server 40, which will be described later, to execute remote control of the mobile body 10. In detail, the control unit 16 drives and controls the driving unit 13, the service providing unit 14, etc. in response to commands related to remote control received from the remote control terminal 30. Through this control, the mobile body 10 is remotely controlled by the remote control terminal 30 operated by the remote operator Q.

[0045] FIG. 3 is a functional block diagram showing an example of the functional configuration of the service administrator terminal 20 according to the embodiment.

[0046] The service administrator terminal 20 includes a communication unit 21, an input unit 22, a display unit 23, a storage unit 24, and a control unit 25. The communication unit 21, the input unit 22, the display unit 23, the storage unit 24, and the control unit 25 are communicatively connected via a bus or the like. The service administrator terminal 20 is an example of an information processing device.

[0047] The communication unit 21 is a communication interface that communicates with the mobile object 10, the remote control terminal 30, and the server 40 via a network NW, etc. For example, the communication unit 21 transmits information about the service, such as service specifications, to the server 40.

[0048] Furthermore, the communication unit 21 transmits information related to the service to the mobile object 10. The communication unit 21 receives inquiry information related to remote control from the remote operator Q by communicating with the remote control terminal 30 operated by the remote operator Q. The communication unit 21 also communicates various information with external devices.

[0049] The input unit 22 accepts various operations by the service manager P. The input unit 22 is, for example, a keyboard, a pointing device such as a mouse, a microphone, etc. The display unit 23 displays or outputs various types of information. The display unit 23 is, for example, a display that displays various types of information, a speaker that outputs various types of audio, etc. The input unit 22 and the display unit 23 may be integrated into a touch panel. The input unit 22 and the display unit 23 may also be configured as separate entities from the service manager terminal 20. In this case, the input unit 22 and the display unit 23 may be configured to be communicably connected to the service manager terminal 20.

[0050] The storage unit 24 stores various types of data. The storage unit 24 is, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, an optical disk, or the like. The storage unit 24 may be a storage device provided outside the service administrator terminal 20. The storage unit 24 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN (Local Area Network), the Internet, or the like. In this embodiment, the storage unit 24 stores various types of information, such as information required to create service specifications.

[0051] The control unit 25 is a computer that executes information processing in the service administrator terminal 20. The control unit 25 may be realized by causing a processor such as a CPU to execute a program, i.e., by software. It may also be realized by a processor such as a dedicated IC, i.e., by hardware. The storage unit 15 may be realized by a combination of software and hardware. At least one of one or more functional units included in the control unit 25 may be mounted on an external information processing device that is communicatively connected to the service administrator terminal 20 via a network NW or the like.

[0052] FIG. 4 is a functional block diagram showing an example of the functional configuration of the remote control terminal 30 according to the embodiment.

[0053] The remote control terminal 30 includes a communication unit 31, an input unit 32, a display unit 33, a storage unit 34, and a control unit 35. The communication unit 31, the input unit 32, the display unit 33, the storage unit 34, and the control unit 35 are communicatively connected via a bus or the like.

[0054] The communication unit 31 is a communication interface that communicates with the mobile object 10, the remote control terminal 30, and the server 40 via a network NW or the like. For example, the communication unit 31 receives remote control assistance request information in response to remote control request information for the mobile object 10 from the server 40 via the network NW. The communication unit 31 also transmits information about the remote operator Q to the server 40. Furthermore, the communication unit 31 transmits commands related to remote control to the mobile object 10 that is the target of remote control. The communication unit 31 also communicates various information with external devices.

[0055] The input unit 32 accepts various operations by the remote operator Q. For example, the input unit 32 accepts input of commands for remote control of the moving body 10 by the remote operator Q. The input unit 32 is, for example, a pointing device such as a keyboard and a mouse, a microphone, a steering wheel, a joystick, an accelerator, a brake, a touch panel, etc.

[0056] The display unit 33 displays or outputs various types of information. For example, the display unit 33 displays captured image information of the mobile object 10 when the mobile object 10 is remotely controlled. The display unit 33 is a display that displays various types of information, a speaker that outputs various types of sound, etc. The input unit 32 and the display unit 33 may be integrated into a touch panel. The input unit 32 and the display unit 33 may also be configured as separate entities from the remote control terminal 30. In this case, the input unit 32 and the display unit 33 may be configured to be communicably connected to the remote control terminal 30.

[0057] The storage unit 34 stores various types of data. The storage unit 34 is, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, an optical disk, etc. The storage unit 34 may be a storage device provided outside the remote control terminal 30. The storage unit 34 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN, the Internet, etc.

[0058] In this embodiment, the storage unit 34 stores, for example, a remote control application. The remote control application is software for providing functions such as receiving remote control assistance request information from the server 40 and executing remote control on the mobile object 10. The remote control application operates, for example, online with the server 40. Note that the remote control application may also be configured to operate offline with respect to the server 40. The remote control application may be configured to be downloaded to the remote control terminal 30, or may be configured to realize the above functions on a browser without being downloaded.

[0059] The control unit 35 is a computer that executes information processing in the remote control terminal 30. The control unit 35 may be realized by causing a processor such as a CPU to execute a program, i.e., by software. The control unit 35 may also be realized by a processor such as a dedicated IC, i.e., by hardware. The control unit 35 may also be realized by a combination of software and hardware. At least one of one or more functional units included in the control unit 35 may be mounted on an external information processing device that is communicatively connected to the service administrator terminal 20 via a network NW or the like.

[0060] Meanwhile, a service manager P, who is an administrator of the services provided by the remote control system 1, has a need to reduce the number of times that the remote operator Q remotely controls the moving object 10 as much as possible. Therefore, in order to reduce the number of times that remote control is performed, the server 40 according to this embodiment identifies the cause of the need for remote response. Figure 5 is a functional block diagram showing an example of the functional configuration of the server 40 according to this embodiment.

[0061] The server 40 includes a communication unit 41, an input unit 42, a display unit 43, a storage unit 44, and a control unit 45. The communication unit 41, the input unit 42, the display unit 43, the storage unit 44, and the control unit 45 are communicatively connected via a bus or the like.

[0062] The communication unit 41 is a communication interface for communicating with the mobile object 10, the service manager terminal 20, and the remote control terminal 30.

[0063] The input unit 42 accepts various operations by the user. For example, the input unit 42 is a pointing device such as a keyboard and a mouse, a microphone, etc. The display unit 43 displays or outputs various types of information. For example, the display unit 43 is a display that displays various types of information, a speaker that outputs various types of sound, etc. The input unit 42 and the display unit 43 may be integrated into a touch panel.

[0064] The storage unit 44 stores various types of information. The storage unit 44 is, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, an optical disk, etc. The storage unit 44 may be a storage device provided outside the server 40. The storage unit 44 may also be a storage medium that stores or temporarily stores programs and various types of information downloaded via a LAN, the Internet, etc.

[0065] In this embodiment, the storage unit 44 stores mobile object management information 441, mobile object speed information 442, heat map information 443, first input information 444, remote control history information 445, second input information 446, normal driving history information 447, cause information 448, cause identification support information 449, etc. Note that the information stored in the storage unit 44 is not limited to these.

[0066] 6 is a schematic diagram showing an example of the data configuration of mobile object management information 441 according to the embodiment. The mobile object management information 441 is a table for managing information about each of the multiple mobile objects 10 included in the remote control system 1, which are managed by the server 40. Note that the data format of the mobile object management information 441 is not limited to a table.

[0067] 6 is a table that associates a vehicle ID, a status, a start position, a start time, and an end time. The vehicle ID is identification information of the mobile body 10. The status is information that indicates the traveling state of the mobile body 10. The traveling state includes, for example, manual traveling by a user, automatic traveling by autonomous traveling, remote traveling by operation of a remote operator Q, safety stop by autonomous traveling, etc.

[0068] The start position is position information indicating the position where the moving object 10 started traveling corresponding to the status. The start position is, for example, a two-dimensional coordinate position set within the range of the traveling area indicating the area where the moving object 10 is traveling. Note that the start position is not limited to this and may be expressed by latitude and longitude. The start time is the time when the moving object 10 started traveling corresponding to the status. The end time is the time when the moving object 10 ended traveling corresponding to the status.

[0069] Dynamically changed information in the mobile entity management information 441, such as location information and operating status, is received from the mobile entity 10 periodically or when a predetermined event occurs, and is updated by the control unit 45 of the server 40. Furthermore, static information in the mobile entity management information 441, such as the mobile entity ID, is not generally updated, but may be changed as appropriate in response to a user's operation instruction at the input unit 42 or a request from the service administrator terminal 20.

[0070] Note that the information stored in the mobile object management information 441 is not limited to these. For example, the mobile object management information 441 may be a table that further associates, for each mobile object ID of the mobile object 10, at least one of the following: the type of the mobile object 10, the travel area in which the mobile object 10 is traveling, the current location information of the mobile object 10, route information indicating the route traveled by the mobile object 10, the remaining battery charge of the mobile object 10, the content of the service provided by the mobile object 10, the status of the service provided by the mobile object 10, the failure status of the mobile object 10, the communication status of the mobile object 10, the hardware configuration of the mobile object 10, and the software configuration of the mobile object 10.

[0071] For example, the service content is information that indicates the transportation of goods, etc., performed by the mobile body 10. For example, the service status is information that indicates whether or not there is a delay in the service performed by the mobile body 10. For example, the failure status is information that indicates whether or not there is a failure in the mobile body 10. For example, the communication status of the mobile body 10 is information that indicates whether the communication of the mobile body 10 is good, has deteriorated, or is interrupted, etc. For example, the hardware configuration of the mobile body 10 is information that indicates the placement of cameras mounted on the mobile body 10, whether the drive source is electric or engine-powered, etc. For example, the software configuration of the mobile body 10 is information such as the type and version information of software installed on the mobile body 10.

[0072] 7 is a schematic diagram showing an example of the data configuration of the mobile object speed information 442 according to the embodiment. The mobile object speed information 442 is a table for managing speed information within an area where the mobile object 10 performs a service. Note that the data format of the mobile object speed information 442 is not limited to a table.

[0073] Mobile object speed information 442 in table T2 shown in FIG. 7 is a table that associates a vehicle ID, a start position, an acceleration detection time, and a speed detection time. The vehicle ID and start position are the same as the vehicle ID and start position shown in FIG. 6, so detailed description will be omitted. The acceleration detection time is information indicating the time when acceleration was detected at the start of travel corresponding to the start position. The speed detection time is information indicating the time when speed was detected at the start of travel corresponding to the start position. Note that mobile object speed information 442 is not limited to this.

[0074] The heat map information 443, first input information 444, remote control history information 445, second input information 446, normal driving history information 447, factor information 448, and factor identification support information 449 stored in the memory unit 44 will be described in detail later.

[0075] Returning to Figure 5, the control unit 45 includes a first acquisition unit 451, a first generation unit 452, an output unit 453, a second acquisition unit 454, a third acquisition unit 455, a fourth acquisition unit 456, a fifth acquisition unit 457, an identification unit 458, and a second generation unit 459.

[0076] The first acquisition unit 451, the first generation unit 452, the output unit 453, the second acquisition unit 454, the third acquisition unit 455, the fourth acquisition unit 456, the fifth acquisition unit 457, the identification unit 458, and the second generation unit 459 are realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above multiple units may be realized by a combination of software and hardware. When multiple processors are used, each of the multiple processors may realize one of the multiple units, or may realize two or more of the multiple units.

[0077] The processor realizes each of the above-mentioned multiple units by reading and executing a program stored in a ROM (Read Only Memory) or the like. Note that instead of storing the program in a ROM, the program may be directly embedded in the processor circuitry. In this case, the processor realizes each of the above-mentioned multiple units by reading and executing the program embedded in the circuitry.

[0078] The first acquisition unit 451 acquires mobile object management information 441 from the storage unit 44. The first acquisition unit 451 also acquires mobile object speed information 442 from the storage unit 44. Information including the mobile object management information 441 and the mobile object speed information 442 is also referred to as mobile object information. In other words, the first acquisition unit 451 acquires the mobile object information from the storage unit 44.

[0079] The first generation unit 452 generates heat map information 443 based on the moving object information. Specifically, the first generation unit 452 generates the heat map information 443 based on the moving object information including the moving object management information 441 and the moving object speed information 442 acquired by the first acquisition unit 451. The heat map information 443 is information that indicates, as a heat map, points on a predetermined route where autonomous movement of the moving object 10 becomes difficult and remote control is performed by the remote operator Q. Here, the heat map information 443 will be described with reference to FIG. 8.

[0080] FIG. 8 is a schematic diagram showing an example of heat map information 443 according to the embodiment. The heat map information 443 shown in FIG. 8 is information including an area 81 of the moving object 10, a travel route 82 actually traveled by the moving object 10, and a heat map 83 of the moving object 10. The heat map 83 shown in FIG. 8 shows a distribution of elevations according to the number of times remote control has been performed. That is, in the heat map 83 shown in FIG. 8, points where remote control has been performed more frequently have a higher distribution and are therefore displayed in darker colors. The first generation unit 452 stores the generated heat map information 443 in the storage unit 44.

[0081] 5 , the output unit 453 outputs the heat map information 443 to the service administrator terminal 20. Specifically, the output unit 453 outputs the heat map information 443 generated by the first generation unit 452. Furthermore, the communication unit 41 transmits the heat map information 443 stored in the storage unit 44 to the service administrator terminal 20. Then, the control unit 25 of the service administrator terminal 20 outputs the heat map information 443 received from the server 40 to the display unit 23.

[0082] The second acquisition unit 454 acquires first input information 444 input by the service manager P from the service manager terminal 20. Specifically, the communication unit 21 of the service manager terminal 20 transmits to the server 40 the first input information 444, of the heat map information 443 displayed on the display unit 23, for which the input unit 22 has accepted an operation by the service manager P. Then, the second acquisition unit 454 acquires the first input information 444 received by the communication unit 41. Here, the first input information 444 will be described with reference to FIG. 9. FIG. 9 is a schematic diagram showing an example of the first input information 444 according to the embodiment.

[0083] 9 is an area that designates a part of area 84 of heat map information 443 shown in FIG. 8 (corresponding to area 84 shown in FIG. 8). Area 84 is an area of ​​heat map information 443 that is colored darkly in heat map 83 and has been remotely controlled many times. Service manager P designates (inputs) area 84 to identify the cause of the need for remote response.

[0084] 5, the third acquisition unit 455 acquires the remote control history information 445. Specifically, the third acquisition unit 455 acquires the remote control history information 445 stored in the storage unit 44. Then, the third acquisition unit 455 acquires the remote control history information 445 corresponding to the first input information 444 acquired by the second acquisition unit 454. Here, the remote control history information 445 will be described with reference to FIG. 10.

[0085] 10 is a schematic diagram showing an example of the data configuration of remote control history information 445 according to the embodiment. The remote control history information 445 in table T3 shown in FIG. 10 is information obtained by extracting only the status "remote driving" from the mobile object management information 441 in table T1 shown in FIG. 6. That is, the remote control history information 445 in table T3 shown in FIG. 10 is a table that associates a vehicle ID, a remote driving status, a start position, a start time, and an end time. The third acquisition unit 455 stores the acquired remote control history information 445 in the storage unit 44.

[0086] 5, the output unit 453 outputs the remote control history information 445 acquired by the third acquisition unit 455. Furthermore, the communication unit 41 transmits the remote control history information 445 stored in the storage unit 44 to the service manager terminal 20. Then, the control unit 25 of the service manager terminal 20 outputs the remote control history information 445 received from the server 40 to the display unit 23.

[0087] The fourth acquisition unit 456 acquires second input information 446 input by the service manager P from the service manager terminal 20. The second input information 446 is information including a second image showing a scene in which a remote response has occurred and the mobile object 10 is traveling by remote control. The second image showing the scene in which the mobile object 10 is traveling by remote control is an image captured by an imaging device mounted on the mobile object 10.

[0088] Specifically, the communication unit 21 of the service administrator terminal 20 transmits to the server 40 second input information 446, which is the remote control history information 445 displayed on the display unit 23 and which is the operation by the service administrator P that the input unit 22 has accepted. Then, the fourth acquisition unit 456 acquires the second input information 446 received by the communication unit 41. Here, the second input information 446 will be described with reference to FIG. 11.

[0089] 11 is a schematic diagram showing an example of second input information 446 according to the embodiment. The second input information 446 is, for example, information that specifies at least one area 111 among the points that are included in the remote control history information 445 and that display the start position of the remote travel on a map, for the area 84 of the first input information 444 shown in FIG. The service manager P specifies (inputs) the area 111 in order to identify the cause of the need for remote response.

[0090] Then, based on the acquired second input information 446, the fourth acquisition unit 456 extracts the start position of the remote control history information 445 to be stored in the storage unit 44, which corresponds to the area 111 included in the second input information 446. As a result, for example, the information extracted by the fourth acquisition unit 456 becomes the area 101 shown in table T3 in Fig. 10, and the service manager P can extract a point for identifying the cause of the need for remote response.

[0091] Returning to FIG. 5, the fifth acquisition unit 457 acquires the normal driving history information 447. The normal driving history information 447 is information including a first image showing a scene in which the moving body 10 is autonomously driving normally. The first image showing a scene in which the moving body 10 is autonomously driving normally is an image captured by an imaging device mounted on the moving body 10. Specifically, the fifth acquisition unit 457 acquires the normal driving history information 447 from the storage unit 44 based on the remote control history information 445 extracted by the fourth acquisition unit 456.

[0092] For example, the fifth acquisition unit 457 extracts a first start time included in the field 101 of the second input information 446 from the remote control history information 445 stored in the storage unit 44. Next, the fifth acquisition unit 457 extracts a second start time from the remote control history information 445 corresponding to the extracted first start time, where the speed in the remote control history information 445 is "state ≠ 0," the status in the remote control history information 445 is "autonomous driving," and the status in the remote control history information 445 has remained "autonomous driving" for a predetermined time or more. The predetermined time is, for example, 30 seconds. Note that the predetermined time is not limited to 30 seconds and can be arbitrarily set by the server 40 so that the service manager P can identify the cause of the remote response. Note that the predetermined time may be set by input from the service manager terminal 20 so that the service manager P can identify the cause of the remote response.

[0093] Next, fifth acquisition unit 457 extracts a first start position from remote control history information 445 corresponding to the extracted second start time. Next, fifth acquisition unit 457 acquires a second start position that exists a predetermined time before the second start time from remote control history information 445 corresponding to the extracted first start position.

[0094] Next, fifth acquisition unit 457 acquires a third start position that exists in the same position as the first start position from remote control history information 445 corresponding to the extracted first start position, and extracts a third start time corresponding to the third start position. Next, fifth acquisition unit 457 extracts a fourth start position that exists a predetermined time before the third start time from remote control history information 445 corresponding to the extracted third start time, and extracts information corresponding to the third start position where the status is "automatic driving".

[0095] Then, when the first start location = the third start location and the second start location = the fourth start location, the fifth acquisition unit 457 extracts the fourth start location that exists a predetermined time before the third start time from the remote control history information 445 that corresponds to the extracted third start time, and acquires information in which the status corresponding to the third start location is "automatic driving" as normal driving history information 447. The fifth acquisition unit 457 also stores the acquired normal driving history information 447 in the storage unit 44.

[0096] The identification unit 458 identifies the cause of the remote response based on the second input information 446 and the normal driving history information 447. Specifically, the identification unit 458 identifies the cause of the remote response based on the second input information 446 acquired by the fourth acquisition unit 456 and the normal driving history information 447 acquired by the fifth acquisition unit 457.

[0097] For example, the identification unit 458 associates the position information of the start position of the remote control history information 445 extracted by the fourth acquisition unit 456 with the position information of the start position of the remote control history information 445 whose status is "autonomous driving" extracted by the fifth acquisition unit 457, and extracts a difference between a first image corresponding to the position information of the start position of the remote control history information 445 extracted by the fourth acquisition unit 456 and a second image corresponding to the position information of the start position of the remote control history information 445 whose status is "autonomous driving" extracted by the fifth acquisition unit 457. Since the difference is considered to be a factor in the remote response, the identification unit 458 identifies the factor of the remote response from the difference. Then, the identification unit 458 stores factor information 448 indicating the identified factor of the remote response in the storage unit 44.

[0098] The second generating unit 459 generates factor identification support information 449. Specifically, the second generating unit 459 generates factor identification support information 449 that supports identification of the factor of a remote response by associating information on the start position of the remote control history information 445 extracted by the fourth acquiring unit 456, information that the status of the remote control history information 445 is "autonomous driving" extracted by the fifth acquiring unit 457, and factor information 448 identified by the identifying unit 458. Then, the second generating unit 459 stores the generated factor identification support information 449 in the storage unit 44.

[0099] The output unit 453 outputs the second input information 446 acquired by the fourth acquisition unit 456, the normal driving history information 447 acquired by the fifth acquisition unit 457, and the factor identification support information 449 generated by the second generation unit 459. In addition, the communication unit 41 transmits the second input information 446, the normal driving history information 447, the factor information 448, and the factor identification support information 449 stored in the storage unit 44 to the service manager terminal 20.

[0100] Then, the control unit 25 of the service administrator terminal 20 outputs the second input information 446, normal driving history information 447, cause information 448, and cause identification support information 449 received from the server 40 to the display unit 23. Specifically, the control unit 25 of the service administrator terminal 20 acquires a first image included in the normal driving history information 447 and showing a scene in which the mobile object 10 is autonomously driving normally, and a second image included in the second input information 446 and showing a scene in which a remote response has occurred and the mobile object 10 is driving by remote control, and outputs cause identification support information 449 to the display unit 23 based on the first image and the second image.

[0101] Fig. 12 is a schematic diagram showing an example of cause identification support information 449 according to the embodiment. In Fig. 12, the form of the cause identification support information 449 output by the display unit 23 of the service manager terminal 20 will be described.

[0102] 12 includes, for example, the input unit 22, a first image 122 (corresponding to the normal driving history information 447 shown in FIG. 5) showing a scene in which the moving body 10 is autonomously driving normally, a second image 123 (corresponding to the second input information 446 shown in FIG. 5) showing a scene in which the moving body 10 is driving by remote control, and factor information 124 (factor information 48 shown in FIG. 5) that becomes a factor in the remote response and becomes a difference. Here, the difference shown in FIG. 12 is an obstacle in the driving of the moving body 10, such as a cardboard box.

[0103] Furthermore, the control unit 25 highlights the factor information 124 that is the difference in the output of the factor identification support information 121. For example, the control unit 25 highlights the difference area by surrounding it with a frame 125. Furthermore, the control unit 25 outputs the first image 122 and the second image 123 side by side.

[0104] For example, the control unit 25 outputs, to the display unit 23, for example, a first image 122 showing a scene in which the mobile object 10 is autonomously traveling normally, and then outputs, to the display unit 23, a second image 123 showing a scene in which a remote response has occurred and the mobile object 10 is traveling by remote control. After the video output of the first image 122 and the second image 123 is completed, the control unit 25 surrounds the difference area with a frame 125 and highlights it. In this way, the service administrator terminal 20 can output the cause identification support information 449 generated by the second generation unit 459 of the server 40 to the display unit 23, thereby supporting the service administrator P in identifying the cause that caused the mobile object 10 to require a remote response.

[0105] Next, an example of the flow of information processing executed by the control unit 45 of the server 40 according to the embodiment will be described.

[0106] 13 is a flowchart showing an example of the flow of information processing executed by the control unit 45 of the server 40 according to the embodiment. In FIG. 13, the content of processing by the server 40 to generate heat map information 443 will be described.

[0107] The first acquisition unit 451 acquires moving object information from the storage unit 44 (step S131). Subsequently, the first generation unit 452 generates heat map information 443 based on the moving object information including the moving object management information 441 and the moving object speed information 442 acquired by the first acquisition unit 451 (step S132). Subsequently, the output unit 453 outputs the heat map information 443 generated by the first generation unit 452.

[0108] When the processing of step S132 is completed, the communication unit 41 transmits the heat map information 443 stored in the storage unit 44 to the service administrator terminal 20. Then, the control unit 25 of the service administrator terminal 20 outputs the heat map information 443 received from the server 40 to the display unit 23.

[0109] 14 is a flowchart showing an example of the flow of information processing executed by the control unit 45 of the server 40 of this embodiment. In FIG. 14, the content of processing in which the server 40 outputs the cause identification support information 449 will be described.

[0110] The second acquisition unit 454 acquires the first input information 444 input by the service administrator P from the service administrator terminal 20 (step S141). Subsequently, the third acquisition unit 455 acquires the remote control history information 445 stored in the storage unit 44 (step S142). Then, the third acquisition unit 455 acquires the remote control history information 445 corresponding to the first input information 444 acquired by the second acquisition unit 454.

[0111] Next, the fourth acquisition unit 456 acquires the second input information 446 input by the service administrator P from the service administrator terminal 20 (step S143). Then, based on the acquired second input information 446, the fourth acquisition unit 456 extracts the start position of the remote control history information 445 to be stored in the storage unit 44, which corresponds to the area 111 included in the second input information 446.

[0112] Next, the fifth acquisition unit 457 acquires the normal driving history information 447 from the storage unit 44 based on the remote control history information 445 extracted by the fourth acquisition unit 456 (step S144). Next, the identification unit 458 identifies the cause of the remote response based on the second input information 446 acquired by the fourth acquisition unit 456 and the normal driving history information 447 acquired by the fifth acquisition unit 457 (step S145).

[0113] Next, the second generating unit 459 generates factor identification support information 449 that supports identification of the factor of the remote response, by associating the information on the start position of the remote control history information 445 extracted by the fourth acquiring unit 456, the information that the status of the remote control history information 445 is "automatic driving" extracted by the fifth acquiring unit 457, and the factor information 448 identified by the identifying unit 458 (step S146). Then, the second generating unit 459 stores the generated factor identification support information 449 in the storage unit 44.

[0114] When step S146 is completed, the communication unit 41 transmits the second input information 446, the normal driving history information 447, the cause information 448, and the cause identification support information 449 stored in the storage unit 44 to the service manager terminal 20. Then, the control unit 25 of the service manager terminal 20 outputs the second input information 446, the normal driving history information 447, the cause information 448, and the cause identification support information 449 received from the server 40 to the display unit 23.

[0115] As described above, the information processing method executed by the service administrator terminal 20 of this embodiment involves a computer acquiring a first image showing a scene in which the mobile body 10 is autonomously driving normally and a second image showing a scene in which a remote response has occurred and the mobile body 10 is driving by remote control, and outputting factor identification support information 449 that supports identifying the cause of the remote response based on the first image and the second image.

[0116] The cause is identified by associating location information indicating the location of the moving object 10 contained in the first image with location information in the second image, and extracting the difference between the first image and the second image. The cause identification support information 449 is then output by highlighting the difference. Furthermore, the cause identification support information 449 is output by arranging the first image and the second image side by side.

[0117] For this reason, the service manager terminal 20 of this embodiment outputs the cause identification support information 449 generated by the server 40 to the display unit 23. By checking the cause identification support information 449 output to the display unit 23, the service manager P can understand the cause that made it necessary to remotely respond to the mobile object 10.

[0118] Therefore, the service administrator terminal 20 and the server 40 of this embodiment can assist in identifying the cause of the need for remote support.

[0119] (First Modification) In the above embodiment, when highlighting differences, the area of ​​the difference is highlighted by surrounding it with a frame 125. For example, the control unit 25 of the service administrator terminal 20 may highlight differences by displaying explanatory text corresponding to the difference or by blinking the text.

[0120] (Second Modification) In the above embodiment, the cause identification support information 449 output by the control unit 25 of the service administrator terminal 20 is described as outputting the second image 123 after outputting the first image 122, but the present invention is not limited to this. For example, the control unit 25 of the service administrator terminal 20 may output the first image 122 and the second image 123 simultaneously. In this way, the service administrator terminal 20 can improve the convenience of video viewing for the service administrator P by outputting two images simultaneously.

[0121] (Third Modification) In the above embodiment, the cause identification support information 449 output by the control unit 25 of the service administrator terminal 20 is described as highlighting the differences, but the present invention is not limited to this. For example, the control unit 25 of the service administrator terminal 20 may output not only the differences but also environmental information during the remote response. The environmental information during the remote response is, for example, the status of communication between the mobile object 10 and the remote control terminal 30.

[0122] Fig. 15 is a schematic diagram showing an example of factor identification support information 131 according to the second modified example. The factor identification support information 131 shown in Fig. 15 is obtained by adding environmental information during remote support to a first image 122 and a second image 123, respectively, in addition to the factor identification support information 121 shown in Fig. 12, and outputting the added information. The environmental information shown in Fig. 15 is, for example, a state 132 of the communication status between the mobile object 10 and the remote control terminal 30 in the first image 122, and a state 133 of the communication status between the mobile object 10 and the remote control terminal 30 in the second image 123. The state 132 indicates a good communication status, and the state 133 indicates a poor communication status.

[0123] Furthermore, the environmental information may include, in addition to the communication status between the mobile object 10 and the remote control terminal 30, the status of the mobile object 10, the traveling position of the mobile object 10 in the area in which the mobile object 10 is traveling, speed information of the mobile object 10, etc. Furthermore, the environmental information may indicate the area in which the mobile object 10 is traveling on a map.

[0124] Furthermore, the environmental information may be displayed superimposed on the first image 122 and the second image 123. Furthermore, if the amount of environmental information is large, it may not be displayed superimposed on the first image 122 and the second image 123, but may be displayed next to the first image 122 and the second image 123, for example.

[0125] The control unit 25 may set the input unit 22 so that the service manager P can display or hide the environmental information. This allows the service manager terminal 20 to output not only obstacles but also other differences. Therefore, the service manager P can extract factors other than obstacles that have made remote response necessary.

[0126] (Fourth Modification) For example, the service manager P may select multiple items from the remote control history information 445 to identify the cause of the remote response. The identification unit 458 of the server 40 identifies the cause of the remote response based on the multiple items of second input information 446 and the normal driving history information, and then the control unit 25 of the service manager terminal 20 outputs the multiple items of second input information 446, normal driving history information 447, cause information 448, and cause identification support information 449 received from the server 40 to the display unit 23. This will be described with reference to FIG. 16 .

[0127] Fig. 16 is a schematic diagram showing an example of factor identification support information 449 according to the fourth modified example. The factor identification support information 141 shown in Fig. 16 is obtained by outputting a plurality of second images 142, 143, and 144 in addition to the factor identification support information 121 shown in Fig. 12.

[0128] In second image 142, factor information 145, which is a factor in remote response and is a difference corresponding to second image 142, and the difference area is shown by frame 146. In second image 143, factor information 147, which is a factor in remote response and is a difference corresponding to second image 143, and the difference area is shown by frame 148. In second image 144, factor information 149, which is a factor in remote response and is a difference, and the difference area is shown by frame 150. The difference in factor information 145 is a person such as a pedestrian. The difference in factor information 147 is a vehicle such as a truck. The difference in factor information 149 is a vehicle such as a passenger car.

[0129] The control unit 25 surrounds and highlights the difference areas of each second image with frames 125, 146, 148, and 150, and outputs first image 122, second image 123, second image 142, second image 143, and second image 144 side by side. For example, the control unit 25 may output first image 122 so that a playback control button is provided for each group during remote support. This is because it is assumed that the difference between each second image during remote support and the normal state exists at different times.

[0130] Furthermore, the control unit 25 may classify the objects that are the differences based on difference type information indicating the type of difference stored in the storage unit 44 of the server 40, and change the frame display mode for each classification so that it is possible to determine whether the differences are the same from the second image taken during remote response. For example, differences may be displayed in red frames for cardboard boxes, blue frames for pedestrians, yellow-green frames for trucks, pink frames for passenger cars, etc. In this way, the service manager terminal 20 can simultaneously output images taken during normal operation and multiple remote responses, allowing the service manager P to grasp multiple remote response factors that occurred in the same location and check their frequency.

[0131] (Fifth Modification) For example, the cause identification support information 449 generated by the second generating unit 459 of the server 40 may generate information including a breakdown of the causes. For example, the second generating unit 459 of the server 40 can generate breakdown information including a breakdown of the causes by acquiring or estimating statistics on obstacles that caused remote responses.

[0132] Fig. 18 is a schematic diagram showing an example of factor identification support information 449 according to the fifth modified example. Factor identification support information 151 shown in Fig. 18 is information including breakdown information 152, which is a breakdown of the differences that become factors in remote handling for second image 123, second image 142, second image 143, and second image 144 shown in Fig. 16, and second image 123, second image 142, second image 143, and second image 144.

[0133] The breakdown information 152 shows, for example, a pie chart showing the breakdown of obstacle objects corresponding to the difference. The breakdown of obstacle objects shown in the pie chart is 70% trucks, 20% cars, 8% pedestrians, and 2% cardboard boxes. Note that the breakdown information 152 is not limited to a pie chart and may include other forms such as a bar graph. In this way, the service manager terminal 20 can output statistical information showing the statistics of the second image during remote response, allowing the service manager P to consider countermeasures based on the most frequent causes.

[0134] (Sixth Modification) For example, when the mobile object 10 autonomously travels within a specific site at an airport or a factory, a container transported or towed by a specific vehicle or the specific vehicle may be tagged for identification. When the identification unit 458 of the server 40 identifies an obstacle that caused a remote response as a container or a stopped vehicle by performing a known image recognition process, the second generation unit 459 of the server 40 may generate, in addition to the cause identification support information 449, abandonment information indicating when the container or stopped vehicle was at a location that required a remote response, left that location, and was abandoned. Then, the control unit 25 of the service administrator terminal 20 may output the cause identification support information 449 and the abandonment information to the display unit 23.

[0135] Fig. 18 is a schematic diagram showing an example of factor identification support information 161 according to the sixth modified example. The factor identification support information 161 shown in Fig. 18 includes, for example, the input unit 22, a first image 122 showing a scene in which the moving object 10 is autonomously traveling normally, a second image 162 showing a scene in which the moving object 10 is traveling by remote control, factor information 163 that is a factor in a remote response and that is a difference, a frame 164 that highlights the factor information 163 and indicates the difference, and abandonment information 165. Here, the difference shown in Fig. 18 is an obstacle in the traveling of the moving object 10, such as a container.

[0136] 18 indicates, for example, the sentence "Time when the remote response occurred: yyyy / mm / dd 10:30" and the sentence "Abandonment period of the container that caused the problem: yyyy / mm / dd 10:00-11:00." In other words, the abandonment information 165 includes remote response occurrence time information indicating the occurrence time when the remote response occurred, and abandonment period information indicating the abandonment period.

[0137] For example, the second generating unit 459 of the server 40 generates remote response occurrence time information indicating the time when a remote response occurred, based on the remote control history information 445 stored in the storage unit 44. Furthermore, the second generating unit 459 generates abandonment period information indicating the period during which the container was abandoned, based on the tag of the container that is the obstacle identified by the identifying unit 458 and transportation information (not shown) associated with the tag. Furthermore, the second generating unit 459 generates abandonment information 165 by associating the remote response occurrence time information with the abandonment period information.

[0138] Furthermore, the identification unit 458 may identify who last abandoned the container or stopped vehicle based on the transportation information associated with the tag, and the second generation unit 459 may generate information including the person who abandoned the container or stopped vehicle in the abandonment information 165. Then, the control unit 25 of the service manager terminal 20 outputs the abandonment information 165 to the display unit 23. This allows the service manager P to know who abandoned the container or vehicle. For example, if the same person repeatedly abandons containers or vehicles in the same way, the service manager P can take individual measures, or if a similar incident occurs in the same location, the service manager P can take measures to change the operation itself.

[0139] For example, the communication unit 41 of the server 40 may output the abandoned period information generated by the second generation unit 459 as warning information to a mobile terminal or the like of the person who left the device abandoned. Furthermore, the communication unit 41 of the server 40 may output the abandoned period information generated by the second generation unit 459 as report information to a superior of the company to which the person who left the device belongs.

[0140] Furthermore, the identification unit 458 may identify the license plate of the stopped vehicle from an image using known image recognition processing instead of a tag, and identify vehicle identification information including the license plate, and the second generation unit 459 may generate information including the vehicle identification information in the abandoned vehicle information 165. The vehicle identification information includes the driver who drives the vehicle, the operations manager, and the operations manager's contact information.

[0141] Then, the control unit 25 of the service manager terminal 20 outputs the abandoned vehicle information 165 to the display unit 23. This allows the service manager P to know the driver of the vehicle, as well as the vehicle's operations manager and the operations manager's contact information. For example, the service manager P can contact the operations manager and request a change in operation so that the vehicle is not parked at the location.

[0142] Furthermore, the communication unit 41 of the server 40 may transmit the abandonment information generated by the second generation unit 459 to the remote control terminal 30. For example, by checking the abandonment information, such as that a remote response is occurring at the same location every day due to construction work within the autonomous driving area of ​​the mobile body 10, the remote operator Q who operates the remote control terminal 30 can understand that the abandoned container or stopped vehicle is interfering with the autonomous driving of the mobile body 10, causing a remote response.

[0143] This allows the service manager terminal 20 to not only identify the cause of the remote response, but also to take action to reduce the remote response or to take measures.

[0144] (Seventh Modification) For example, the first image of the normal driving history information may include an obstacle. This occurs when an obstacle is present in a position that does not affect the autonomous driving of the mobile object 10. Specifically, this occurs when there is no obstacle on the driving route of the mobile object 10, but an obstacle is present outside the driving route. For example, the identification unit 458 may extract the obstacle included in the normal driving history information by performing a known image recognition process based on the extracted difference.

[0145] FIG. 19 is a schematic diagram illustrating an example of factor identification support information 171 according to the seventh modified example. The factor identification support information 171 illustrated in FIG. 19 includes, for example, the input unit 22, a first image 172 illustrating a scene in which the moving object 10 is autonomously traveling normally, a second image 173 illustrating a scene in which the moving object 10 is traveling by remote control, obstacle information 174 illustrating an obstacle for which remote action is not required, factor information 175 illustrating a difference that may be a cause of remote action and requires remote action, a frame 176 illustrating a difference region in which the factor information 175 is highlighted, a state 177 of the first image 172, and a state 178 of the second image 173. The state 177 is information including the sentence "Normal: container present." The state 178 is information including the sentence "Abnormal: container present."

[0146] For example, the service manager P can recognize that an obstacle is not simply a cause of the remote response when the obstacle information is included in the first image 172. Also, for example, the service manager P can recognize that an obstacle is a cause of the remote response when the obstacle information is not included in the first image 172.

[0147] (Eighth Modification) For example, the first generating unit 452 of the server 40 may further generate heat map information 443 for an event in which the remote driving status is longer than a predetermined time and a remote response is required. An example of a case in which the remote driving status is longer than a predetermined time is a case in which remote control is being performed to avoid an obstacle or the like on a narrow road and to take a detour.

[0148] Then, the identification unit 458 of the server 40 may identify the cause of an event in which the remote driving status was longer than a predetermined time and a remote response was required. In this case, the identification unit 458 of the server 40 may extract events in which the remote response was required shorter than a predetermined time from among images with similar remote responses, and identify the events in which the remote response was required longer than a predetermined time and the events in which the remote response was required shorter than a predetermined time as cause identification support information.

[0149] This allows the service manager P to narrow down the focus on the remote response by the remote operator Q that has a significant impact on autonomous driving and lasts for a long period of time, and to devise measures to address the causes.

[0150] Next, an example of the hardware configuration of the mobile object 10, the service administrator terminal 20, the remote control terminal 30, and the server 40 in the above embodiment and modified example will be described.

[0151] 20 is a block diagram showing an example of the hardware configuration of the remote control system 1 according to the embodiment and the modified example. In the mobile object 10, the service administrator terminal 20, the remote control terminal 30, and the server 40 according to the embodiment and the modified example, a CPU 50, a ROM 52, a RAM 54, an I / F 56, etc. are interconnected via a bus 58, and the hardware configuration utilizes a normal computer.

[0152] The CPU 50 is a computing device that controls the mobile object 10, the service administrator terminal 20, the remote control terminal 30, and the server 40 in the above-described embodiment and modified examples. The ROM 52 stores programs and the like that realize information processing by the CPU 50. The RAM 54 stores data necessary for various processes by the CPU 50. The I / F 56 is an interface that is connected to a storage unit, an input unit, a display unit, a sensor, a communication unit, and the like, and is used to send and receive data.

[0153] In the mobile object 10, service administrator terminal 20, remote control terminal 30 and server 40 of the above embodiment and modified example, the CPU 50 reads a program from the ROM 52 onto the RAM 54 and executes it, thereby realizing each of the above functional units on the computer.

[0154] The programs for executing the above processes executed by the mobile object 10, the service administrator terminal 20, the remote control terminal 30, and the server 40 in the above-described embodiment and modified examples may be stored in an HDD (hard disk drive). Also, the programs for executing the above processes executed by the mobile object 10, the service administrator terminal 20, the remote control terminal 30, and the server 40 in the above-described embodiment and modified examples may be provided by being pre-installed in the ROM 52.

[0155] In addition, the programs for executing the above processes executed by the mobile body 10, service administrator terminal 20, remote control terminal 30, and server 40 in the above embodiments and modified examples may be stored in an installable or executable format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD) and provided as a computer program product.

[0156] Furthermore, the programs for executing the information processing executed by the mobile object 10, the service administrator terminal 20, the remote control terminal 30, and the server 40 in the above-described embodiment and modified examples may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Furthermore, the programs for executing the information processing executed by the mobile object 10, the service administrator terminal 20, the remote control terminal 30, and the server 40 in the above-described embodiment and modified examples may be provided or distributed via a network such as the Internet.

[0157] Although the above describes an embodiment, the embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its modifications are included within the scope and spirit of the invention, and are also included in the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0158] 10 Mobile 20 Service administrator terminal 23 Display section 25 Control Unit 30 Remote Control Terminal 40 servers 45 Control Unit 451 First acquisition part 452 1st generation part 453 Output Department 454 Part 2 455 Part 3 456 Part 4 457 Part 5 458 Specific Department 459 Second Generation Section

Claims

1. The computer Acquire a first image showing a scene in which a moving object is autonomously traveling normally, and a second image showing a scene in which a remote control has occurred and the moving object is traveling by remote control; outputting factor identification support information that supports identification of a factor of the remote response based on the first image and the second image; Information processing methods.

2. The factor is identified by associating position information indicating a position of the moving object included in the first image and the second image with each other, and extracting a difference between the first image and the second image. The information processing method according to claim 1 .

3. The output of the cause identification support information highlights the difference. The information processing method according to claim 2 .

4. The factor identification support information is output by outputting the first image and the second image side by side. The information processing method according to claim 3 .

5. The factor identification support information includes a breakdown of the factors. The information processing method according to claim 3 .

6. The cause identification support information includes a time when the remote response occurred. The information processing method according to any one of claims 1 to 5.

7. Acquire a first image showing a scene in which a moving object is autonomously traveling normally, and a second image showing a scene in which a remote control has occurred and the moving object is traveling by remote control; a control unit that outputs cause identification support information that supports identification of a cause of the remote response based on the first image and the second image; Information processing device.

8. Acquire a first image showing a scene in which a moving object is autonomously traveling normally, and a second image showing a scene in which a remote control has occurred and the moving object is traveling by remote control; outputting factor identification support information that supports identification of a factor of the remote response based on the first image and the second image; An information processing program that causes a computer to execute a process.

Images