Information processing device, information processing method, and program

Abstract

The present technology relates to an information processing device, an information processing method, and a program that make it possible to reduce the burden on an operator that monitors the surroundings of a plurality of systems. According to the present invention, an information processing device comprises a recognition unit and an image processing unit. On the basis of a plurality of captured images captured of different locations by a plurality of systems, the recognition unit extracts regions of caution that are regions that require caution in the surroundings of the systems, and, on the basis of the captured images that include the regions of caution, the image processing unit generates a first overhead image. The present technology can be applied, for example, to the monitoring of systems by an operator or to a server that mediates interventions at systems.

Description

Technical Field

[0001] The present technology relates to an information processing apparatus, an information processing method, and a program, and particularly relates to an information processing apparatus, an information processing method, and a program suitable for use when one operator monitors the surroundings of a plurality of systems.

Background Art

[0002] Conventionally, a technology has been proposed to share in real time an image captured by a system worn by a user at the scene with an operator not at the scene so that the user and the operator can communicate. By using this technology, for example, by sharing in real time images captured by systems worn by users existing at different positions with an operator not at the scene, it is possible for one operator to monitor the surroundings of a plurality of systems (users) (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when one operator monitors the surroundings of a plurality of systems, the visual load on the operator increases. Therefore, there is a risk that the operator may overlook a dangerous situation around the system.

[0005] The present technology has been made in view of such a situation, and is intended to reduce the load when an operator monitors the surroundings of a plurality of systems.

Means for Solving the Problems

[0006] One aspect of this technology is an information processing device comprising: a recognition unit that extracts areas requiring attention around each system based on multiple captured images taken at different locations by multiple systems; and an image processing unit that generates a first overhead image based on the captured images including the areas requiring attention.

[0007] One aspect of this technology is an information processing method in which an information processing device extracts areas requiring attention around each system based on multiple captured images taken at different locations by multiple systems, and generates an overhead image based on the captured images including the areas requiring attention.

[0008] One aspect of this technology involves a program that extracts areas requiring attention around each system based on multiple captured images taken at different locations by multiple systems, and then causes a computer to perform a process to generate an overhead image based on the captured images that include the areas requiring attention.

[0009] In one aspect of this technology, based on multiple captured images taken at different locations by multiple systems, a region requiring attention is extracted around each system, and an overhead image is generated based on the captured images including the region requiring attention. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing a first embodiment of an information processing system to which this technology is applied. [Figure 2] This is a block diagram showing an example configuration for a user support system. [Figure 3] This is a schematic diagram illustrating a specific example of a user assistance system. [Figure 4] This is a block diagram showing examples of vehicle configurations. [Figure 5] This is a block diagram showing an example of the configuration of an operating terminal. [Figure 6] This is a schematic diagram showing a specific example of an operating terminal. [Figure 7] It is a block diagram showing a configuration example of a management server. [Figure 8] It is a block diagram for explaining an example of processing of an information processing system. [Figure 9] It is a flowchart for explaining a first embodiment of monitoring support processing. [Figure 10] It is a diagram for explaining a method of generating a monitoring image. [Figure 11] It is a diagram for explaining a method of intervention by an operator. [Figure 12] It is a flowchart for explaining a second embodiment of monitoring support processing. [Figure 13] It is a diagram showing examples of a global monitoring image and a local monitoring image. [Figure 14] It is a diagram showing an example of a route along which a vehicle travels. [Figure 15] It is a diagram showing an example of an accident occurrence location. [Figure 16] It is a flowchart for explaining accident response processing. [Figure 17] It is a diagram showing an example of a monitoring image. [Figure 18] It is a diagram showing an example of an accident section. [Figure 19] It is a block diagram showing a second embodiment of an information processing system to which the present technology is applied. [Figure 20] It is a flowchart for explaining learning data generation processing. [Figure 21] It is a diagram showing an example of data without labels. [Figure 22] It is a diagram for explaining a method of assigning labels. [Figure 23] It is a diagram showing an example of learning data. [Figure 24] It is a diagram showing an example of the data configuration of learning data and tokens. [Figure 25] It is a diagram showing an example of the configuration of a computer.

Embodiments for Carrying Out the Invention

[0011] The following describes the configurations for implementing this technology. The explanation will proceed in the following order. 1. First Embodiment 2. Second Embodiment 3. Variant 4. Others

[0012] <<1. First Embodiment>> First, a first embodiment of this technology will be described with reference to Figures 1 to 18.

[0013] <Example Configuration of Information Processing System 1> Figure 1 shows an example of the configuration of an information processing system 1, which is a first embodiment of an information processing system 1 to which this technology is applied.

[0014] Information processing system 1 comprises systems 11-1 to 11-m, operation terminals 12-1 to 12-n, and a management server 13. Systems 11-1 to 11-m, operation terminals 12-1 to 12-n, and the management server 13 are interconnected via a network 21.

[0015] Hereafter, when it is not necessary to distinguish between systems 11-1 through 11-m individually, they will simply be referred to as system 11. Hereafter, when it is not necessary to distinguish between operation terminals 12-1 through 12-n individually, they will simply be referred to as operation terminal 12.

[0016] System 11 is comprised of, for example, a user support system that assists the user, or an autonomous mobile unit that moves autonomously.

[0017] User support systems consist of, for example, security systems, operation support systems, and behavioral support systems.

[0018] A security system is, for example, a system that a user wears or carries, which monitors the area around the security system (user) and protects the user from danger.

[0019] An operation support system is a system that assists a user in operating a device, for example, by being installed on the device used by the user, or by being worn or carried by the user. Devices that are the target of operation support include, for example, mobile objects that are moved by user operation such as vehicles, and machine tools that are operated by user operation.

[0020] A behavioral support system is, for example, a system worn by a user to assist the user's actions. Examples of behavioral support systems include power suits and prosthetic limbs that users wear on their arms, hands, legs, etc.

[0021] Autonomous mobile devices include, for example, vehicles, drones, and robots that are capable of moving autonomously. Furthermore, autonomous mobile devices may be either user-operated or user-unoperated.

[0022] The operation terminal 12 is an information processing device used by the operator. The operator uses the operation terminal 12 to monitor the surroundings of each system 11 and to intervene in the system 11 as needed.

[0023] The interventions performed by the operator may include at least one of the following: visual interventions, auditory interventions, physical interventions, or manipulative interventions.

[0024] Visual intervention is a process that intervenes in a user's field of vision and provides support to the user, for example, by presenting visual information (hereinafter referred to as "visual information") within the user's field of vision using System 11. Examples of the visual information presented include images, displayed messages, and the illumination or flashing of lights.

[0025] Auditory intervention is a process that intervenes in the user's hearing and provides support to the user, for example, by outputting auditory information (hereinafter referred to as auditory information) using system 11 to a user using system 11. Examples of auditory information that can be output include voice messages, warning sounds, sound effects, etc.

[0026] Physical intervention is a process that intervenes in the user's body and provides support to the user, for example, by using system 11 to make the user move or to stimulate the body.

[0027] Operational intervention is a process that intervenes in the operation of system 11, for example, by remotely operating system 11, and provides support to system 11 or the user operating system 11.

[0028] Furthermore, the operator can use the control terminal 12 to combine multiple types of interventions. For example, the operator can use the control terminal 12 to combine two or more of the following: visual interventions, auditory interventions, physical interventions, and manipulative interventions. By combining multiple types of interventions, for example, a more immersive interaction can be achieved.

[0029] The management server 13 acts as an intermediary between each system 11 and each operation terminal 12, providing support to operators, users, and the system 11. The management server 13 also intervenes in the system 11 on behalf of the operator as needed. Furthermore, the management server 13 provides map information to each system 11 and each operation terminal 12.

[0030] <Example configuration of user support system 51> Figure 2 shows an example configuration of a user support system 51, which is an example of system 11.

[0031] The user support system 51 includes an external sensor 61, an internal sensor 62, an operation input unit 63, a communication unit 64, a control unit 65, and an output unit 66.

[0032] The external environment sensor 61 includes sensors that sense the surrounding environment of the user assistance system 51 (user). For example, the external environment sensor 61 includes one or more of the following: a camera, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), ToF (Time Of Flight) sensor, millimeter-wave radar, ultrasonic sensor, distance sensor, etc. The external environment sensor 61 supplies sensor data obtained from each sensor (hereinafter referred to as external environment sensor data) to the control unit 65.

[0033] External sensor data includes captured images of the user's surroundings. These captured images may be either video or still images. For example, images captured using a fisheye camera or similar device, covering 360 degrees around the user assistance system 51, can be used as the captured images.

[0034] The internal environment sensor 62 includes sensors that sense the user support system 51 and the user's state. For example, the internal environment sensor 62 includes a GNSS receiver (Global Navigation Satellite System), an IMU (Inertial Measurement Unit), etc. The internal environment sensor 62 supplies sensor data obtained from each sensor (hereinafter referred to as internal environment sensor data) to the control unit 65.

[0035] The operation input unit 63 is equipped with various operating devices and is used for user operation. The operation input unit 63 supplies operation signals corresponding to user operations to the control unit 65.

[0036] The communication unit 64 is equipped with various communication devices and can communicate with other devices such as other systems 11, operating terminals 12, and management servers 13 via the network 21. The communication unit 64 supplies data received from other devices to the control unit 65 and obtains data to be transmitted to other devices from the control unit 65.

[0037] The control unit 65 includes, for example, a processor such as a CPU. The control unit 65 controls the user support system 51 and performs various processes. The control unit 65 includes a position estimation unit 71, a recognition unit 72, an information processing decision unit 73, and an output control unit 74.

[0038] The position estimation unit 71 estimates the position and orientation of the user support system 51 (user) based on external sensor data and internal sensor data.

[0039] The recognition unit 72 recognizes the surrounding conditions of the user support system 51, as well as the status of the user support system 51 and the user, based on external sensor data and internal sensor data. For example, the recognition unit 72 performs recognition processing of objects around the user support system 51 and recognizes the type, position, size, shape, movement, etc. of the objects.

[0040] The information processing decision unit 73 performs various information processing and makes judgments and controls regarding the operation and processing of the user support system 51 based on the operation signal from the operation input unit 63, the estimation result from the position estimation unit 71, and the recognition result from the recognition unit 72. Furthermore, for example, in response to intervention from an operator or the management server 13, the information processing decision unit 73 performs various information processing and makes judgments and controls regarding the operation and processing of the user support system 51, thereby realizing operational intervention.

[0041] The output control unit 74 controls the display unit 81, the audio output unit 82, and the drive unit 83 of the output unit 66. The output control unit 74 also generates monitoring information and transmits it to the operation terminal 12 and the management server 13 via the communication unit 64 and the network 21.

[0042] The monitoring information includes, for example, captured images of the area surrounding the user support system 51, the results of recognition of the surrounding conditions of the user support system 51, and the results of recognition of the status of the user support system 51 and the user.

[0043] The output unit 66 includes a display unit 81, an audio output unit 82, and a drive unit 83.

[0044] The display unit 81 is equipped with various display devices such as a display and a head-mounted display, and outputs visual information. Furthermore, visual intervention is realized, for example, when the display unit 81 outputs visual information in response to intervention from an operator or the management server 13.

[0045] The audio output unit 82 is equipped with various audio output devices such as headphones, earphones, and speakers, and outputs auditory information. Furthermore, auditory intervention is realized, for example, when the audio output unit 82 outputs auditory information in response to intervention from an operator or management server 13.

[0046] The drive unit 83 is composed of, for example, actuators that provide tactile sensations or mild electrical stimulation to the user's body that is not harmful to health. For example, the drive unit 83 is composed of devices that assist or restrain body movement by driving a power suit or exoskeleton worn by the user on their arms, hands, legs, etc. Furthermore, physical intervention is realized, for example, when the drive unit 83 moves the user's body or provides stimulation to the user's body in accordance with intervention from an operator or management server 13.

[0047] Figure 3 shows an example of how the user assistance system 51 is worn. In this example, the user assistance system 51 is shown as a ring-shaped wearable device worn on the user's head.

[0048] The user assistance system 51 may also be composed of other types of wearable devices. Furthermore, the user assistance system 51 may be composed of a combination of two or more devices. For example, the user assistance system 51 may be composed of a combination of a wearable device and a smartphone.

[0049] In the following, when any part of the user support system 51 communicates with other devices via the communication unit 64 and the network 21, the descriptions of the communication unit 64 and the network 21 will be omitted. For example, when the control unit 65 sends and receives data with the management server 13 via the communication unit 64 and the network 21, it will be described as the control unit 65 sending and receiving data with the management server 13.

[0050] <Example configuration of vehicle 101> Figure 4 shows an example configuration of vehicle 101, which is an example of system 11. Vehicle 101 is a vehicle capable of autonomous movement through automatic driving.

[0051] Vehicle 101 includes an external sensor 111, an internal sensor 112, a sensor information acquisition unit 113, a position estimation unit 114, a recognition unit 115, a monitoring information transmission unit 116, an antenna 117, a difference detection unit 118, a remote control receiving unit 119, an accident detection unit 120, a situation judgment unit 121, a route generation unit 122, a vehicle control unit 123, a map information receiving unit 124, a map update unit 125, and a map DB (database) 126.

[0052] The external sensor 111 includes various sensors used to recognize the surrounding conditions of the vehicle 101. For example, the external sensor 111 includes one or more of the following: a camera, radar, LiDAR, ultrasonic sensor, distance sensor, etc. The external sensor 111 supplies sensor data (hereinafter referred to as external sensor data) output from each sensor to the sensor information acquisition unit 113. The external sensor data includes captured images of the area around the vehicle 101.

[0053] The internal sensor 112 is equipped with various sensors used to recognize the state of the vehicle 101. For example, the internal sensor 112 includes a GNSS receiver, an IMU, a speed sensor, an accelerator sensor, a brake sensor, a wheel speed sensor, and the like. The internal sensor 62 supplies sensor data obtained from each sensor (hereinafter referred to as internal sensor data) to the sensor information acquisition unit 113.

[0054] The sensor information acquisition unit 113 supplies external sensor data and internal sensor data to the position estimation unit 114, the recognition unit 115, and the monitoring information transmission unit 116.

[0055] The position estimation unit 114 estimates the position and orientation of the vehicle 101 based on external sensor data and internal sensor data. The position estimation unit 114 supplies information indicating the estimation result of the position and orientation of the vehicle 101 to the recognition unit 115.

[0056] The recognition unit 115 recognizes the surrounding conditions of the vehicle 101 and the state of the vehicle 101 based on external sensor data, internal sensor data, and the estimated position and orientation of the vehicle 101. For example, the recognition unit 115 performs recognition processing of objects around the vehicle 101 and recognizes the type, position, size, shape, movement, etc. of the objects. The recognition unit 115 supplies information indicating the recognition results to the monitoring information transmission unit 116, the difference detection unit 118, and the situation determination unit 121.

[0057] The monitoring information transmission unit 116 transmits monitoring information to the operation terminal 12 and the management server 13 via the antenna 117 and the network 21.

[0058] The monitoring information includes, for example, captured images of the area around the vehicle 101, estimated results of the position and orientation of the vehicle 101, and recognition results of the conditions around the vehicle 101 and the state of the vehicle 101. The captured images may be either video or still images.

[0059] The difference detection unit 118 performs a process to detect the difference between the surrounding conditions of the vehicle 101 recognized by the recognition unit 115 and the map information stored in the map DB 126. The difference detection unit 118 supplies difference information indicating the detection result of the difference between the surrounding conditions of the vehicle 101 and the map information to the accident detection unit 120 and the map update unit 125.

[0060] The remote control receiving unit 119 receives remote control signals for remotely controlling the vehicle 101 from the operation terminal 12 or the management server 13 via the network 21 and the antenna 117. The remote control receiving unit 119 supplies the received remote control signals to the accident detection unit 120 and the vehicle control unit 123.

[0061] The accident detection unit 120 performs accident detection processing around the vehicle 101 based on differential information and remote control signals. Here, an accident is, for example, an external event that hinders the passage of the vehicle 101. Specifically, for example, disasters, accidents, construction, obstacles, road damage, etc., are considered to be accidents. The accident detection unit 120 supplies accident information indicating the accident detection result to the situation determination unit 121.

[0062] The situation determination unit 121 controls the driving method based on monitoring information and accident information. Here, the driving method refers to, for example, normal driving, low-speed driving, manual driving, and driving according to operator intervention. Normal driving and low-speed driving are methods in which the vehicle 101 drives autonomously through automatic driving. Manual driving is a method in which the vehicle is driven by the driver's operation. The situation determination unit 121 notifies the route generation unit 122 of the set driving method for the vehicle 101 and also supplies monitoring information and accident information to the route generation unit 122.

[0063] When the vehicle 101 is driving autonomously, the route generation unit 122 generates route information indicating the route the vehicle 101 will travel, based on monitoring information, accident information, and map information stored in the map DB 126. The route generation unit 122 supplies the route information to the vehicle control unit 123.

[0064] The vehicle control unit 123 controls the movement of the vehicle 101 according to route information, remote control signals, or operations by the driver.

[0065] The map information receiving unit 124 receives map information from the management server 13 via the network 21 and antenna 117. The map information receiving unit 124 supplies the received map information to the map update unit 125.

[0066] The map update unit 125 updates the map information stored in the map database 126 with map information received from an external source. Furthermore, the map update unit 125 corrects the map information stored in the map database 126 as needed based on the difference information.

[0067] In the following, when any part of the vehicle 101 communicates with other devices via the antenna 117 and network 21, the descriptions of the antenna 117 and network 21 will be omitted. For example, when the monitoring information transmission unit 116 transmits monitoring information to the management server 13 via the antenna 117 and network 21, it will be described as the monitoring information transmission unit 116 transmitting monitoring information to the management server 13.

[0068] <Example configuration of operating terminal 12> Figure 5 shows an example of the configuration of the operating terminal 12.

[0069] The operating terminal 12 includes a communication unit 151, a control unit 152, a display unit 153, an audio output unit 154, an operation input unit 155, an imaging unit 156, and an audio input unit 157.

[0070] The communication unit 151 is equipped with various communication devices and can communicate with other devices such as the system 11, other operating terminals 12, and the management server 13 via the network 21. The communication unit 151 supplies data received from other devices to the control unit 152 and obtains data to be transmitted to other devices from the control unit 152.

[0071] The control unit 152 includes, for example, a processor such as a CPU. The control unit 152 controls the operation terminal 12 and performs various processes. The control unit 152 includes an output control unit 161, an action recognition unit 162, and an intervention unit 163.

[0072] The output control unit 161 controls the output of visual information by the display unit 153 and the output of auditory information by the audio output unit 154.

[0073] The action recognition unit 162 recognizes the operator's actions (e.g., gestures) based on the operator's captured image captured by the imaging unit 156.

[0074] The intervention unit 163 generates intervention information for performing an intervention on the system 11 based on information input by the operator via the operation input unit 155 or the voice input unit 157, or the operator's gestures recognized by the action recognition unit 162. The intervention information includes, for example, visual information used for visual intervention, auditory signals used for auditory intervention, or remote control signals used for physical or manipulative intervention to remotely control the system 11. The intervention unit 163 transmits the intervention information to the system 11 or the management server 13 via the communication unit 151 and the network 21.

[0075] The display unit 153 includes various display devices such as a display or a head-mounted display, and outputs visual information.

[0076] The audio output unit 154 is equipped with various audio output devices such as headphones, earphones, and speakers, and outputs auditory information.

[0077] The operation input unit 155 is equipped with various operating devices and is used for operator operation. The operation input unit 155 supplies operation signals corresponding to the operator's operation to the control unit 152.

[0078] The imaging unit 156 includes, for example, a camera. The imaging unit 156, for example, images the operator and supplies the obtained image to the control unit 152.

[0079] The voice input unit 157 includes, for example, a microphone. The voice input unit 157 collects the operator's voice and other audio data, and supplies the obtained audio data to the control unit 152.

[0080] Figure 6 shows an example of how the operating terminal 12 is worn. In this example, the operating terminal 12 is shown to be a head-mounted display worn on the operator's head.

[0081] The operating terminal 12 may be composed of other types of wearable devices. Furthermore, the operating terminal 12 may be composed of information processing devices other than wearable devices, such as PCs (Personal Computers), smartphones, and tablet terminals. In addition, the operating terminal 12 may be composed of a combination of two or more devices. For example, the operating terminal 12 may be composed of a combination of a wearable device and a smartphone. For example, the operating terminal 12 may be composed of a combination of multiple displays and a computer.

[0082] In the following, when each part of the operating terminal 12 communicates with other devices via the communication unit 151 and the network 21, the descriptions of the communication unit 151 and the network 21 will be omitted. For example, when the control unit 152 sends and receives data with the management server 13 via the communication unit 151 and the network 21, it will be described as the control unit 152 sending and receiving data with the management server 13.

[0083] <Example configuration of management server 13> Figure 7 shows an example configuration of the management server 13.

[0084] The management server 13 includes a communication unit 201 and a control unit 202.

[0085] The communication unit 201 is equipped with various communication devices. The communication unit 201 can communicate with the system 11 and other devices such as the operating terminal 12 via the network 21. The communication unit 201 supplies data received from other devices to the control unit 202 and obtains data to be transmitted to other devices from the control unit 202.

[0086] The control unit 202 is equipped with a processor such as a CPU and performs control of the management server 13 and various processing. The control unit 202 includes a recognition unit 211, an image processing unit 212, an accident section setting unit 213, an intermediary unit 214, an intervention unit 215, a learning unit 216, and a map information provision unit 217.

[0087] The recognition unit 211 recognizes the surrounding conditions of system 11 based on monitoring information received from system 11.

[0088] The image processing unit 212 performs various image processing operations on the captured images acquired from the system 11. For example, the image processing unit 212 generates monitoring images that the operator uses to monitor the environment around the system 11, based on the captured images acquired from each system 11.

[0089] The accident section setting unit 213 sets an accident section based on intervention information transmitted from the operation terminal 12 when an operator intervenes in the system 11, and monitoring information transmitted from the system 11 when an operator intervenes. An accident section is, for example, the section in which an operator intervenes, including the location where the accident occurred.

[0090] The intermediary unit 214 mediates monitoring of the system 11 by the operation terminal 12 (operator) and intervention processing of the system 11 by the operation terminal 12 (operator). For example, the intermediary unit 214 transmits monitoring images generated by the image processing unit 212 to the operation terminal 12 via the communication unit 201 and the network 21. For example, the intermediary unit 214 transmits intervention information received from the operation terminal 12 to the system 11 to be intervened in via the communication unit 201 and the network 21. At this time, the intermediary unit 214 processes the intervention information as necessary.

[0091] The intervention unit 215 performs intervention processing on system 11 on behalf of the operator, or together with the operator. For example, the intervention unit 215 generates intervention information for performing an intervention on system 11 and transmits it to the target system 11.

[0092] The learning unit 216 learns the operator's operation of the system 11 based on the monitoring information acquired from the system 11 and the remote control signals included in the intervention information acquired from the operation terminal 12. For example, the learning unit 216 learns the operator's operation of a vehicle 101, which is a type of system 11.

[0093] The map information provision unit 217 provides map information to each system 11 and each operating terminal 12.

[0094] <Processing by Information Processing System 1> Next, the processing of the information processing system 1 will be explained with reference to Figures 8 to 18.

[0095] <Operation support processing> First, we will explain the operation support processes performed by the information processing system 1, referring to the flowchart in Figure 8.

[0096] The following describes an example in which a user operates the user support system 51 and an operator intervenes as needed.

[0097] In step S1, the user support system 51 senses the user's state and operations, as well as the surrounding environment. Specifically, the external sensor 61 senses the environment around the user support system 51 (user) and supplies the external sensor data to the control unit 65. The internal sensor 62 senses the state of the user support system 51 and the user and supplies the internal sensor data to the control unit 65. The operation input unit 63 supplies operation signals corresponding to user operations to the control unit 65 in response to user operations.

[0098] The position estimation unit 71 estimates the position and orientation of the user support system 51 (user) based on external sensor data and internal sensor data. The recognition unit 72 recognizes the surrounding conditions of the user support system 51, as well as the state of the user support system 51 and the user, based on external sensor data and internal sensor data.

[0099] The output control unit 74 generates monitoring information and transmits it to the management server 13. The monitoring information includes, for example, captured images of the area around the user support system 51, the recognition results of the conditions around the user support system 51, and the recognition results of the status of the user support system 51 and the user.

[0100] In response, the control unit 202 of the management server 13 receives monitoring information from the system 11. The image processing unit 212 generates a monitoring image based on the captured image included in the monitoring information. The intermediary unit 214 transmits the monitoring image to the operator's terminal 12.

[0101] In response, the control unit 152 of the operating terminal 12 receives monitoring images from the management server 13. The display unit 153 displays the monitoring images under the control of the output control unit 161. The operator monitors the situation around the user while viewing the monitoring images displayed on the display unit 153.

[0102] In step S2, the user support system 51 estimates the degree of operator support needed. For example, the recognition unit 72 estimates the degree of risk, which indicates the probability that the user will encounter danger, based on the result of the processing in step S1. The information processing decision unit 73 estimates the degree of operator support needed based on the estimated degree of risk. For example, the degree of support needed increases as the degree of risk increases and decreases as the degree of risk decreases.

[0103] In step S3, the information processing determination unit 73 determines whether or not operator support is required. For example, if the degree of support required estimated in step S2 is less than a predetermined threshold, the information processing determination unit 73 determines that operator support is not required, and the process returns to step S1.

[0104] Subsequently, the processes in steps S1 to S3 are repeatedly executed until it is determined in step S3 that operator support is required.

[0105] On the other hand, in step S3, if the support need estimated in step S2 is above a predetermined threshold, the information processing determination unit 73 determines that operator support is needed, and the process proceeds to step S4.

[0106] In step S4, the management server 13 determines whether or not an operator is available to assist.

[0107] For example, the information processing decision unit 73 of the user support system 51 requests operator support from the management server 13.

[0108] In response, the mediation unit 214 of the management server 13 receives a support request from the user support system 51 and inquires with the operator's terminal 12 whether or not the user can be supported.

[0109] In response, the control unit 152 of the operating terminal 12 receives an inquiry from the management server 13 regarding the availability of user support. The display unit 153, under the control of the output control unit 161, displays a message inquiring about the availability of user support.

[0110] In response, the operator determines whether or not they can provide support to the user and inputs the determination result to the operation terminal 12 via the operation input unit 155. The intervention unit 163 of the operation terminal 12 notifies the management server 13 of the operator's determination result regarding whether or not they can provide support to the user.

[0111] In response, the mediation unit 214 of the management server 13 receives notification of the operator's determination of whether or not they can support the user. If the mediation unit 214 determines, based on the operator's determination, that the operator is not able to provide support, the process proceeds to step S5.

[0112] In step S5, the user support system 51 autonomously stops operating. Specifically, the mediation unit 214 of the management server 13 notifies the user support system 51 that the operator is unavailable.

[0113] In response, the control unit 65 of the user support system 51 receives notification that the operator is unavailable. The user support system 51 autonomously stops operating under the control of the information processing decision unit 73. This ensures the user's safety locally.

[0114] Subsequently, the process returns to step S4, where steps S4 and S5 are repeatedly executed until it is determined that the operator is available to handle the situation.

[0115] On the other hand, in step S4, if the mediation unit 214 of the management server 13 determines, based on the operator's determination result, that the operator is able to handle the situation, the process proceeds to step S6.

[0116] In step S6, the management server 13 obtains the operator's decision result. For example, the operator looks at the monitoring image and decides on an intervention method to support the user, and inputs information about the intervention method using the operation input unit 155 or the like. The intervention unit 163 generates intervention information to execute an intervention on the user support system 51 based on the information input by the operator. The intervention unit 163 transmits the intervention information to the management server 13.

[0117] In response, the control unit 202 of the management server 13 receives intervention information from the operation terminal 12.

[0118] In step S7, the user support system 51 performs semi-automatic control according to the user's abilities and status. Specifically, the intervention unit 215 of the management server 13 transmits intervention information received from the operation terminal 12 to the user support system 51.

[0119] In response, the control unit 65 of the user support system 51 receives intervention information from the management server 13. For example, the information processing decision unit 73 determines the proportion of automatic control by operator intervention based on the intervention information, according to the user's abilities and status. The output control unit 74 controls the output unit 66 based on the determined proportion of automatic control. This enables operator intervention.

[0120] After that, the process returns to step S1, and the processes from step S1 onward are executed.

[0121] As described above, the user can be assisted through operator intervention.

[0122] <First embodiment of monitoring support processing> For example, the more systems 11 an operator monitors, the greater the operator's workload. As a result, for example, in step S4 of Figure 8, it becomes more likely that the system will determine that the operator is unable to respond, causing the user support system 51 to stop working or the operator to be unable to support the user.

[0123] In response to this, as explained below, the management server 13 assists in monitoring the operators and reduces the operator's workload.

[0124] Here, with reference to the flowchart in Figure 9, a first embodiment of the monitoring support process performed by the management server 13 will be described.

[0125] This process starts, for example, when the management server 13 is powered on, and ends when the management server 13 is powered off.

[0126] The following explanation uses the example of assisting one operator who monitors the surrounding environment of multiple systems 11 (users).

[0127] In step S31, the management server 13 acquires monitoring information from each system 11. That is, the control unit 202 of the management server 13 receives monitoring information transmitted from each system 11 that the operator is monitoring.

[0128] The monitoring information includes, for example, captured images of the area around system 11, the results of recognition of the surrounding environment of system 11, and the results of recognition of the state of system 11. Furthermore, if system 11 is being used by a user (for example, if system 11 is not a robot or similar entity not operated by a user), the monitoring information also includes the results of recognition of the user's state.

[0129] In step S32, the recognition unit 211 extracts areas of concern from the captured images of each system 11. For example, the recognition unit 211 extracts areas of concern from the captured images included in the monitoring information, based on the monitoring information received from each system 11.

[0130] Here, the area requiring attention is an area where it is presumed that the user or system 11 needs to pay attention. For example, the area requiring attention includes areas containing hazardous materials and areas containing objects that the user or system 11 needs to check.

[0131] Hazardous materials are, for example, objects that may endanger the user or system 11, and conversely, objects that may endanger the user or system 11. For example, hazardous materials include objects that the user or system 11 may collide with or come into contact with, such as surrounding vehicles, pedestrians, and obstacles. For example, hazardous materials include objects that the user or system 11 may fall into, such as puddles or holes in construction sites.

[0132] Objects that the user or system 11 needs to verify include, for example, traffic lights, road signs, intersections, railway crossings, etc.

[0133] It should be noted that areas of concern are not always extracted from all captured images, and there may be captured images from which no areas of concern are extracted.

[0134] In step S33, the image processing unit 212 generates a monitoring image by combining each of the areas of concern. Specifically, the image processing unit 212 extracts images of areas of concern (hereinafter referred to as "area of ​​concern images") from each captured image and generates a monitoring image by combining the extracted areas of concern images. At this time, the image processing unit 212 places each area of ​​concern image in the monitoring image at a position corresponding to its position in the original captured image. As a result, each area of ​​concern image is placed in the monitoring image while maintaining its position as seen from each system 11, and an overhead image is generated that provides an overview of each area of ​​concern at once.

[0135] For example, the image processing unit 212 may generate a two-dimensional or three-dimensional model of an object present in the area of ​​concern instead of an image extracted from the captured image, and use this as the image of the area of ​​concern.

[0136] In step S34, the intermediary unit 214 transmits the monitoring image to the operator. That is, the intermediary unit 214 transmits the monitoring image to the operating terminal 12 used by the operator.

[0137] In response, the operating terminal 12 receives the monitoring image and displays it on the display unit 153. The operator monitors the area around each system 11 while viewing the monitoring image displayed on the display unit 153.

[0138] Here, with reference to Figure 10, an example of how to generate monitoring images will be described. Figure 10 shows an example where operator OP is monitoring the surroundings of users U1 to U3 (the system 11 they are using).

[0139] For example, if a tree 301 is located to the right and in front of user U1, user U1's system 11 captures an image that includes the tree 301. Then, from user U1's captured image, the region containing the tree 301 is extracted as a region requiring attention.

[0140] For example, if a traffic light 302 is located in the center in front of user U2, user U2's system 11 acquires an image including the traffic light 302. Then, from user U2's image, the area including the traffic light 302 is extracted as a region requiring attention.

[0141] For example, if vehicle 303 is located to the left and in front of user U3, user U3's system 11 acquires an image including vehicle 303. Then, from user U3's image, the area including vehicle 303 is extracted as a region requiring attention.

[0142] Then, a monitoring image 311 is generated, which includes images of the tree 301, the traffic light 302, and the vehicle 303. The tree 301, the traffic light 302, and the vehicle 303 are positioned in the monitoring image 311 at locations corresponding to their positions in the original captured image.

[0143] In this way, the tree 301, traffic light 302, and vehicle 303, which are actually located in different places, are placed within a single monitoring image 311. Furthermore, the tree 301, traffic light 302, and vehicle 303 are placed in the same positions within the monitoring image as they appear from each user's (system 11's) perspective.

[0144] This allows the operator (OP) to simultaneously check the areas of concern around each user (system 11) by viewing the monitoring image. Furthermore, since areas other than the areas of concern are not displayed in the monitoring image, the visibility of the areas of concern is improved, making it easier for the operator (OP) to recognize them. In addition, the operator (OP) can easily recognize the relative position of the areas of concern with respect to each user (system 11). In other words, the operator (OP) can easily recognize the position of the areas of concern from the perspective of each user (system 11).

[0145] In step S35, the management server 13 determines whether or not an area requiring attention has been designated by the operator.

[0146] For example, if the operator determines that there is a region of concern within the monitoring image that requires intervention to prevent the user or system 11 from danger, they designate that region of concern (hereinafter referred to as the region of interest).

[0147] The method for specifying the area of ​​interest is not particularly limited. For example, the operator may specify the area of ​​interest using the operation input unit 155, gestures (e.g., pointing), or voice.

[0148] The intervention unit 163 of the operation terminal 12 transmits information indicating the area of ​​interest specified by the operator (hereinafter referred to as "area of ​​interest information") to the management server 13.

[0149] In response, the control unit 202 of the management server 13 receives information about the area of ​​interest from the operation terminal 12. The intermediary unit 214 then determines that the area of ​​interest has been designated by the operator, and the process proceeds to step S36.

[0150] In step S36, the intermediary unit 214 identifies the system 11 to be intervened with. Specifically, based on the region of interest information, the intermediary unit 214 identifies the captured image from which the region of interest specified by the operator is extracted. The intermediary unit 214 also identifies the system 11 that captured the identified image. This identifies the system 11 to be intervened with (hereinafter referred to as the system to be intervened with).

[0151] In step S37, the management server 13 mediates intervention in the identified system 11.

[0152] For example, after specifying the area of ​​interest, the operator inputs the necessary information for intervention into the target system via the control terminal 12. The operator's intervention method may be any of the visual, auditory, physical, or manipulative interventions described above. Furthermore, two or more intervention methods may be combined.

[0153] Furthermore, the method of inputting information necessary for intervention is not particularly limited. For example, the operator may input information necessary for intervention into the operation terminal 12 using the operation input unit 155, gestures, or voice.

[0154] The intervention unit 163 of the operation terminal 12 generates intervention information based on the information entered by the operator and transmits it to the management server 13.

[0155] In response, the mediation unit 214 of the management server 13 receives intervention information from the operation terminal 12. The mediation unit 214 then transmits the received intervention information to the system to be intervened.

[0156] In response, the system under intervention receives intervention information and, based on that information, executes processing in accordance with the operator's intervention.

[0157] For example, if the system to be intervened is the user support system 51, the output unit 66, under the control of the output control unit 74, executes processing according to the intervention information, such as visual intervention, auditory intervention, physical intervention, or manipulative intervention.

[0158] For example, if the system to be intervened is vehicle 101, the vehicle control unit 123 controls the operation of vehicle 101 based on the intervention information, thereby executing processing according to the operational intervention.

[0159] For example, as schematically shown in Figure 11, if operator OP selects vehicle 303 in the monitoring image 311 and inputs the voice message "There is a car on the left" into the operation terminal 12, the operation terminal 12 sends intervention information including the voice message to the management server 13.

[0160] In response, the mediation unit 214 of the management server 13 receives intervention information from the operation terminal 12 and transmits it to the user U3's system 11.

[0161] In response, user U3's system 11 outputs a voice message, "There is a car on your left," based on the intervention information.

[0162] In this way, the operator's auditory intervention on user U3's system 11 is realized.

[0163] After that, the process returns to step S31, and the processes from step S31 onward are executed.

[0164] On the other hand, in step S35, if the intermediary unit 214 has not received any area of ​​interest information from the operation terminal 12, it determines that no area of ​​concern has been designated by the operator, and the process returns to step S31.

[0165] Subsequently, the processes from step S31 onward are executed.

[0166] As described above, the operator can efficiently monitor the surroundings of multiple systems 11. Furthermore, the operator can quickly detect hazards around each system 11 and intervene in the system 11 easily and quickly. This allows the operator to quickly and reliably remove users or systems 11 from hazards, improving the reliability of operator intervention.

[0167] <Second embodiment of monitoring support processing> Next, a second embodiment of the monitoring support processing performed by the management server 13 will be described with reference to the flowchart in Figure 12.

[0168] In step S61, monitoring information is acquired from each system 11, similar to the process in step S31 in Figure 9.

[0169] In step S62, similar to the process in step S32 in Figure 9, areas requiring attention are extracted from the captured images of each system 11.

[0170] In step S63, the recognition unit 211 estimates the risk level of each location. For example, the recognition unit 211 estimates the risk level of each area requiring attention that was extracted in the processing of step S62.

[0171] Here, the method for estimating the degree of risk is not particularly limited. For example, the recognition unit 211 estimates the probability and degree of risk that the user or system 11 will be endangered in the attention area, and conversely, the probability and degree of risk that the user or system 11 will be endangered in the attention area. Based on the estimated results, the recognition unit 211 then estimates the degree of risk in the attention area.

[0172] Furthermore, for example, the recognition unit 211 estimates the risk level of each location. For example, the recognition unit 211 groups together areas of concern that exist within a predetermined range into a single location. Then, for example, the recognition unit 211 calculates the risk level of each location by taking the average of the risk levels of the areas of concern that exist within each location.

[0173] In step S64, the intermediary unit 214 determines whether or not to present the global monitoring image to the operator. For example, if the conditions for presenting the global monitoring image to the operator are met, the intermediary unit 214 determines to present the global monitoring image to the operator, and the process proceeds to step S65.

[0174] Conditions for presenting a broad surveillance image to the operator include, for example, when the operator requests the presentation of a broad surveillance image, or when there are no locations where the risk level exceeds a predetermined threshold.

[0175] In step S65, the image processing unit 212 generates a global surveillance image.

[0176] The process then proceeds to step S67.

[0177] On the other hand, in step S64, for example, if the conditions for presenting a local monitoring image to the operator are met, the image processing unit 212 determines to present the local monitoring image to the operator, and the process proceeds to step S66.

[0178] Conditions for presenting localized surveillance images to the operator include, for example, when the operator instructs the presentation of localized surveillance images, or when there are locations where the risk level exceeds a predetermined threshold.

[0179] Situations where the level of danger exceeds a predetermined threshold include, for example, situations where obstacles are scattered on the road or pedestrian routes are disrupted due to disasters or accidents. This situation can be recognized, for example, by estimating the degree of non-stationary status using pattern matching or machine learning based on data from cameras and distance sensors.

[0180] Furthermore, a situation where the level of danger exceeds a predetermined threshold is, for example, when someone attempts to enter a level crossing despite the barrier being down. This situation is recognized based on a pre-set scenario, using image processing and machine learning based on data from cameras and distance sensors.

[0181] In step S66, the image processing unit 212 generates a local monitoring image.

[0182] The process then proceeds to step S67.

[0183] In step S67, the monitoring image is transmitted to the operator, similar to the process in step S34 in Figure 9. This presents the operator with either a global or local monitoring image.

[0184] Here, with reference to Figure 13, examples of global and local monitoring images presented to the operator OP are described. The right side of Figure 13 shows an example of a global monitoring image, and the left side shows an example of a local monitoring image.

[0185] The overall monitoring image is an overview image that shows the distribution and risk level of each location, including the area of ​​concern, on a map. In this example, locations A through C are shown on the map as locations that include the area of ​​concern. It is also shown that the risk level of location A is 0.89, the risk level of location B is 0.54, and the risk level of location C is 0.81.

[0186] Furthermore, the way the circles indicating the location of each point are displayed changes depending on the level of danger. For example, the higher the level of danger, the larger the circle becomes or the darker its color. Conversely, the lower the level of danger, the smaller the circle becomes or the lighter its color.

[0187] For localized surveillance images, for example, a composite overhead image is used, in which areas of concern near the highest-risk locations are extracted. Localized surveillance images are generated, for example, by the same method as in the example in Figure 10 described above. Here, an example of a surveillance image is shown in which areas of concern are extracted from images captured by each system 11 located near point A and then composited.

[0188] In this example, the risk level of location A is shown within the monitoring image. Furthermore, areas of particular high risk are highlighted and enclosed in a rectangular frame (hereinafter referred to as a bounding box). In this example, train 351, vehicle 352, and motorbike 353 are enclosed in bounding boxes.

[0189] The bounding box's display changes based on factors such as the level of danger of the area of ​​concern within the bounding box. For example, the higher the danger level, the thicker the bounding box's border becomes or the darker the border's color. Conversely, the lower the danger level, the thinner the bounding box's border becomes or the lighter the border's color. Also, for example, if the area inside the bounding box is semi-transparent, the higher the danger level, the darker the color inside the bounding box becomes, and the lower the danger level, the lighter the color inside the bounding box becomes.

[0190] For example, the audio output unit 154 may output a sound (e.g., a warning sound) corresponding to an object in each attention area under the control of the output control unit 161. In this case, for example, the volume and pitch of the output sound may change based on the level of danger in each attention area.

[0191] After that, the process returns to step S61, and the processes from step S61 onward are executed.

[0192] Furthermore, for example, if a localized monitoring image is presented to the operator, the same processing as in steps S35 to S37 of Figure 9 may be performed. That is, by the operator specifying a region of concern within the localized monitoring image, intervention may be performed on the system 11 that captured the image containing the specified region of concern.

[0193] In this way, the operator can monitor the surrounding conditions of each system 11 both globally and locally.

[0194] For example, both a global monitoring image and a local monitoring image may be presented to the operator simultaneously. Furthermore, the operator can select either monitoring image, and the selected image may be enlarged and displayed.

[0195] <Accident Response Procedure> For example, as shown in Figure 14, when autonomous buses, vehicles 101-1 to 101-4, are patrolling a designated route 401, it is conceivable that this technology could be applied to allow an operator to monitor the surroundings of vehicles 101-1 to 101-4 in order to ensure safety.

[0196] In the following, when it is not necessary to distinguish between vehicles 101-1 through 101-4 individually, they will simply be referred to as vehicle 101.

[0197] This diagram shows an example where four vehicles 101 patrol Route 401, but there is no particular limit to the number of vehicles 101. However, due to the increased visual load on the operator, in reality, the number of vehicles 101 that one operator can monitor is limited to a few.

[0198] Furthermore, as shown in Figure 15, if an accident occurs at point P1 on Route 401, it is conceivable that the operator may need to intervene in the operation of vehicle 101 to avoid the accident.

[0199] In this case, the operator would need to intervene in the operation of each vehicle 101 each time it travels through the section where the accident occurred. This would increase the operator's workload, and the waiting time for operator intervention would become the bottleneck, potentially causing delays in the operation of vehicle 101.

[0200] In contrast, the accident response process performed by the management server 13 will be explained with reference to the flowchart in Figure 16.

[0201] This process starts, for example, when the management server 13 is powered on and ends when it is powered off.

[0202] During this process, monitoring information is transmitted from each vehicle 101 to the management server 13.

[0203] In response, the image processing unit 212 of the management server 13 generates a monitoring image based on the captured images included in the monitoring information of each vehicle 101. The image processing unit 212 may, for example, use the captured images of each vehicle 101 as the monitoring image as they are, or it may generate a monitoring image by the method described above, referring to Figure 9 or Figure 12. The intermediary unit 214 transmits the monitoring image to the operation terminal 12.

[0204] The display unit 153 of the operating terminal 12 displays a monitoring image under the control of the output control unit 161. The operator monitors the area around each vehicle 101 while viewing the monitoring image.

[0205] In step S101, the recognition unit 211 determines whether or not an accident has occurred. This process is repeated until it is determined that an accident has occurred. If an accident has been determined to have occurred, the process proceeds to step S102.

[0206] Accident detection may be performed by the management server 13, the vehicle 101, or the operator.

[0207] For example, the recognition unit 211 of the management server 13 detects an accident based on monitoring information acquired from the vehicle 101.

[0208] For example, if vehicle 101 detects an accident, monitoring information including the accident detection result is sent from vehicle 101 to management server 13. In response, the recognition unit 211 of management server 13 recognizes the occurrence of an accident based on the monitoring information received from vehicle 101.

[0209] For example, if an operator detects an accident, they use the operation terminal 12 to notify the management server 13 of the occurrence of the accident. In response, the recognition unit 211 of the management server 13 recognizes the occurrence of the accident based on the notification from the operation terminal 12.

[0210] In step S102, the mediation unit 214 determines whether or not the operator has intervened in the operation.

[0211] For example, the operator determines whether or not operational intervention is necessary by looking at the monitoring image displayed on the display unit 153 of the operation terminal 12.

[0212] Figure 17 shows an example of a monitoring image displayed to the operator. In this example, an obstacle 411 is located in front of vehicle 101. For example, the operator can view this monitoring image and determine whether or not operational intervention is necessary for vehicle 101 to avoid the obstacle 411.

[0213] If the operator determines that operational intervention is necessary, they input the information required for the intervention into the operation terminal 12.

[0214] The input method required for the intervention is not particularly limited. For example, the operator may use the operation input unit 155, gestures, or voice to input the information required for the intervention into the operation terminal 12.

[0215] In response, the intervention unit 163 of the operating terminal 12 generates intervention information based on the information input by the operator. The intervention information includes, for example, remote control signals for remotely controlling the vehicle 101 and information indicating the operator's line of sight. The intervention unit 163 transmits the intervention information to the management server 13.

[0216] In response, when the mediation unit 214 of the management server 13 receives intervention information from the operation terminal 12, it determines that the operator has made an operational intervention, and the process proceeds to step S104.

[0217] In step S103, the management server 13 mediates the operational intervention. Specifically, the mediation unit 214 transmits the intervention information received from the operation terminal 12 to the vehicle 101 that is the target of the operational intervention.

[0218] In response, the vehicle control unit 123 of the vehicle 101 that is the target of the operational intervention receives intervention information from the management server 13. Based on the received intervention information, the vehicle control unit 123 controls the operation of the vehicle 101 and executes processing in accordance with the operational intervention.

[0219] In step S104, the management server 13 collects information related to the operational intervention. Specifically, the learning unit 216 collects the operator's operation history during the operational intervention based on the remote control signals included in the intervention information. The operation history includes, for example, the timing and amount of operation of the steering wheel (steering angle), accelerator, and brake of the vehicle 101, as well as the section in which the operational intervention occurred. Furthermore, the learning unit 216 collects visual information regarding objects etc. that the operator saw during the operational intervention based on the operator's gaze direction included in the intervention information and the monitoring image presented to the operator during the operational intervention. In addition, the learning unit 216 collects sensor information at the time of the operational intervention based on the monitoring information received from the vehicle 101.

[0220] In step S105, the accident section setting unit 213 sets an accident section. For example, based on the information collected by the learning unit 216 in the process of step S104, the accident section setting unit 213 sets the section in which the operator intervened as an accident section.

[0221] For example, as shown in Figure 18, in Route 401, section 401A, which includes the point P1 where the accident occurred, is designated as the accident section.

[0222] In step S106, the learning unit 216 learns the operator's actions. For example, the learning unit 216 uses the operator's actions in the accident section as an expert and learns the operator's actions in the accident section by utilizing the operator's visual information in the accident section. The learning unit 216 obtains parameters for a learning model that mimics the operator's actions in the accident section and generates a learning model using the obtained parameters.

[0223] Any learning method can be used to learn the operator's actions. For example, inverse reinforcement learning combined with simulation, inverse reinforcement learning using GAIL (Generative Adversarial Imitation Learning), and imitation learning using expert actions can be employed. By performing retraining (fine-tuning) using inverse reinforcement learning, a learning model can be generated that not only simply imitates the operator's actions but also can avoid moving objects such as vehicles and pedestrians.

[0224] In step S107, the intervention unit 215 performs an operational intervention in the accident section based on the learning results. Specifically, the intervention unit 215 generates intervention information, including a remote control signal, using the learning model, and transmits it to the vehicle 101 traveling through the accident section.

[0225] In response, the vehicle control unit 123 of the vehicle 101 receives intervention information from the management server 13 and controls the operation of the vehicle 101 based on the received intervention information, thereby executing processing according to the operational intervention.

[0226] As a result, the management server 13 can perform the same operational interventions on the vehicles 101 traveling through the accident section without the operator's intervention. Consequently, the workload on the operators is reduced, while each vehicle 101 is able to avoid accidents.

[0227] In step S108, the management server 13 determines whether or not to cancel the accident section. For example, the recognition unit 211 monitors changes in the status of the accident section based on monitoring information transmitted from the vehicle 101. If the recognition unit 211 determines that there has been a significant change in the status of the accident section, it notifies the operator's terminal 12 of the change in the accident section.

[0228] In response, the output control unit 161 of the operation terminal 12 receives notification of a change in the situation in the accident section and controls the display unit 153 or the audio output unit 154 to notify the operator of the change in the accident section.

[0229] In response, the operator checks the monitoring image displayed on the display unit 153 and determines whether or not operational intervention is necessary in the accident section. The operator inputs the determination result into the operation terminal 12.

[0230] The method of inputting the judgment result is not particularly limited. For example, the operator may input the judgment result to the operation terminal 12 using the operation input unit 155, gestures, or voice.

[0231] The intervention unit 163 of the operation terminal 12 notifies the management server 13 of the result of the determination of whether or not operational intervention is necessary in the accident section.

[0232] In response, the accident section setting unit 213 of the management server 13 determines that if the operator determines that operational intervention is necessary in the accident section, it will not cancel the accident section, and the process will return to step S107.

[0233] Furthermore, if there is no significant change in the condition of the accident section, the accident section setting unit 213 determines that it will not cancel the accident section, and the process returns to step S107.

[0234] Subsequently, in step S108, the processes in steps S107 to S108 are repeatedly executed until it is determined that the accident section should be released. As a result, the management server 13 intervenes in the operation of the vehicle 101 traveling through the accident section until the accident section is released.

[0235] On the other hand, in step S108, if the accident section setting unit 213 determines that the operator does not need to intervene in the accident section, it determines to cancel the accident section, and the process proceeds to step S109.

[0236] In step S109, the accident zone setting unit 213 cancels the accident zone. Following the cancellation of the accident zone, the intervention unit 215 of the management server 13 stops intervening in the operation of the vehicle 101.

[0237] After that, the process returns to step S101, and the processes from step S101 onward are executed.

[0238] As described above, the management server 13 intervenes in the operation of each vehicle 101 on behalf of the operator, thereby reducing the operator's workload.

[0239] Furthermore, since the management server 13 performs operational interventions using a learning model generated by executing a learning process based on the operation history during operator interventions, each vehicle 101 can safely avoid accidents.

[0240] Furthermore, if a similar accident occurs in the future, the management server 13 will be able to intervene using a trained model in the accident section, including the location of the accident, enabling each vehicle 101 to safely avoid the accident.

[0241] <<2. Second Embodiment>> Next, a second embodiment of the present technology will be described with reference to Figures 19 to 24.

[0242] As described above, the recognition unit 211 of the management server 13 performs a process to extract areas requiring attention from the captured image. In this process, the recognition unit 211, for example, recognizes hazardous materials in the captured image and extracts the area containing the hazardous material as an area requiring attention.

[0243] In this case, the recognition unit 211 needs to determine, for example, whether the situation or object in the captured image is dangerous or not. For this determination process, it is possible to use, for example, a learned model obtained through machine learning.

[0244] Furthermore, running machine learning with more training data improves the accuracy of the learning model. However, collecting a large amount of high-quality training data requires accurately labeling that large amount of data.

[0245] In contrast, a second embodiment of this technology enables the rapid and accurate assignment of labels to training data.

[0246] <Example configuration of information processing system 501> Figure 19 shows an example configuration of an information processing system 501, which is a second embodiment of an information processing system to which this technology is applied.

[0247] The information processing system 501 includes a data transmission terminal 511, an unlabeled data server 512, a learning data generation server 513, client terminals 514-1 to 514-n, and a learning data server 515.

[0248] Hereafter, when there is no need to distinguish between client terminals 514-1 through 514-n individually, they will simply be referred to as client terminal 514.

[0249] The data transmission terminal 511 selects unlabeled data to be labeled and uploads the selected unlabeled data to the unlabeled data server 512. Unlabeled data is data that has not been assigned a label.

[0250] The training data generation server 513 retrieves unlabeled data from the unlabeled data server 512 and sends it to each client terminal 514.

[0251] Each client terminal 514 presents unlabeled data to the evaluator. Here, the evaluator is the user who performs the task of assigning labels.

[0252] Each client terminal 514 retrieves the labels assigned by the evaluator and assigns them to the unlabeled data to generate labeled data. Each client terminal 514 then sends the labeled data to the training data generation server 513.

[0253] The training data generation server 513 determines the correct labels based on the labels assigned by each evaluator. The training data generation server 513 generates training data by assigning correct labels to unlabeled data. The training data generation server 513 stores the training data in the training data server 515.

[0254] Then, data users perform machine learning using the training data stored on the training data server 515.

[0255] <Training data generation process> Next, referring to the flowchart in Figure 20, we will explain the details of the learning data generation process performed by the information processing system 501.

[0256] In step S201, the data transmission terminal 511 selects the data to be labeled. That is, the data transmission terminal 511 selects one of the unlabeled data that has not yet been labeled as the data to be labeled.

[0257] Figure 21 shows an example of unlabeled data. In this example, the unlabeled data includes images and sensor information. The images may be either video or still images. The sensor information is, for example, information detected by a predetermined sensor under the conditions shown in the image. For example, the sensor information may include information indicating the speed of a vehicle in the image.

[0258] In step S202, the data transmission terminal 511 uploads the selected data to the unlabeled data server 512.

[0259] In step S203, the training data generation server 513 distributes the data to be labeled to each client terminal 514. Specifically, the training data generation server 513 retrieves the data uploaded from the data transmission terminal 511 from the unlabeled data server 512 and sends it to each client terminal 514.

[0260] In step S204, each client terminal 514 assigns a label to the distributed data.

[0261] For example, as shown in Figure 22, the client terminal 514 presents the evaluator with a situation indicated by images and sensor information contained in the acquired unlabeled data. The client terminal 514 also asks the evaluator whether the presented situation is dangerous by displaying a message such as "Is this dangerous?" or by outputting an audio message.

[0262] In response, the evaluator determines whether the presented situation is dangerous or not, and inputs a label indicating the result of that determination into the client terminal 514. For example, the evaluator inputs a label with the value either "Dangerous" or "Safe".

[0263] The client terminal 514 generates labeled data by attaching labels entered by the evaluator to the distributed data.

[0264] In step S205, the training data generation server 513 collects labeled data from each client terminal 514.

[0265] In step S206, the training data generation server 513 determines the correct label based on the labels assigned to the collected data. For example, the training data generation server 513 determines the correct label by majority vote from among the labels assigned to the collected labeled data. That is, the training data generation server 513 determines the label with the highest number of votes among the labels assigned to the collected labeled data as the correct label.

[0266] The learning data generation server 513 generates learning data by assigning correct labels to the data to be labeled.

[0267] For example, as shown in FIG. 23, learning data is generated by assigning a correct label with a value of "dangerous" to the unlabeled data in FIG. 21.

[0268] In step S207, the learning data generation server 513 distributes tokens to the evaluators who have been assigned correct labels. The token is, for example, data that can be exchanged for rewards. Note that the types of rewards that can be exchanged with tokens are not particularly limited.

[0269] Specifically, the learning data generation server 513 sends a token to the client terminal 514 which is the source of the labeled data to which the correct label has been assigned. Thereby, tokens are distributed to the evaluators who have been assigned correct labels.

[0270] The right side of FIG. 24 shows an example of the data structure of the token. The token includes a data ID and a token ID.

[0271] The data ID is an ID for identifying the data to which the correct label has been assigned.

[0272] The token ID is an ID for identifying the distributed token.

[0273] In step S208, the learning data generation server 513 associates the learning data with the distributed tokens. For example, the learning data generation server 513 associates the learning data with the distributed tokens by adding information about the distributed tokens to the learning data.

[0274] The left side of Figure 24 shows an example of the data structure of the training data after linking information about the distributed tokens. The training data includes data, labels, data IDs, and a list of token IDs.

[0275] The data ID corresponds to the data ID included in the distributed token.

[0276] The token ID list contains a pair of token ID and user ID for each evaluator to whom a token was distributed. The token ID corresponds to the token ID contained in the distributed token. The user ID is an ID used to identify the evaluator to whom the token was distributed.

[0277] This makes it possible to identify the evaluators to whom tokens were distributed by assigning correct labels to each piece of training data.

[0278] In step S209, the training data generation server 513 saves the training data to the training data server 515.

[0279] By repeating the above process, a large amount of appropriately labeled training data can be collected quickly.

[0280] In other words, the correct label is determined by majority vote from among the labels assigned by multiple evaluators. Furthermore, tokens are distributed to evaluators who assign the correct label, thus incentivizing each evaluator to assign appropriate labels. This improves the accuracy of the labels and reduces the assignment of inappropriate labels.

[0281] Furthermore, because an unspecified number of evaluators can participate in the labeling process, labels for each data point can be collected quickly.

[0282] Furthermore, since the correct label is determined by majority vote and the process of verifying the content of the correct label is omitted, the correct label can be assigned to each data point more quickly and at a lower cost.

[0283] Furthermore, it is possible to calculate the probability of assigning the correct label to each evaluator (hereinafter referred to as the accuracy rate) based on the token ID list of each training data. This allows for the determination of the skill level of each evaluator, and for example, rewards can be varied based on the evaluator's skill level. For instance, evaluators with a higher accuracy rate can receive higher rewards.

[0284] Alternatively, for example, evaluators who assign the correct label may be directly rewarded with money or other means instead of tokens.

[0285] Furthermore, for example, multiple data sets could be bundled together and distributed to evaluators, who could then assign labels to each data set.

[0286] Furthermore, data users may use the training data stored in the training data server 515 individually, or they may use a set of training data that combines multiple training data sets. The training data sets are classified, for example, by data type and purpose of use.

[0287] Furthermore, data users may either pay a fee to the data administrator who manages the training data for each individual training data set, or they may pay a fee for the entire set of training data. In either case, the token ID list allows the evaluators who assigned correct labels to each training data set to be identified, making it possible to return a portion of the fees paid by data users to the evaluators who assigned the correct labels.

[0288] Furthermore, data managers may request labeling from groups such as companies or organizations, rather than individuals. In this case, the group would be evaluated and rewarded based, for example, on the average accuracy rate of the evaluators within the group.

[0289] <<3. Variant Example>> The following describes some modifications of the embodiments of the present technology described above.

[0290] <Variations regarding the division of labor> In the information processing system 1, it is possible to change the division of each process as needed.

[0291] For example, the operation terminal 12 or the management server 13 may execute all or part of the recognition process of the situation around the system 11 and the states of the system 11 and the user.

[0292] For example, the operation terminal 12 may execute the process of the management server 13 and directly intervene in each system 11. In this case, for example, the generation process of the monitoring image described above with reference to FIGS. 9 and 12 is executed by the operation terminal 12.

[0293] For example, the management server 13 may execute the learning data generation process described above with reference to FIG. 19 and the like. Also, the management server 13 may execute the learning process using the learning data generated by the learning data generation process.

[0294] <Other variations> The system 11 of the information processing system 1 in FIG. 1 can be installed at an arbitrary location such as a road for monitoring. In this case, for example, the situation around the system 11 is monitored based on the monitoring information obtained by the system 11. Since the system 11 is not particularly used or moved by the user, basically, no operator intervention is required.

[0295] The learning data generation process described above with reference to FIG. 19 and the like can be applied to the entire process of attaching labels to learning data regardless of the learning method, learning purpose, type of learning data, etc. <000097​​​​​​The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers that can perform various functions by installing various programs.

[0297] Figure 25 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program.

[0298] In computer 1000, the CPU (Central Processing Unit) 1001, ROM (Read Only Memory) 1002, and RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0299] An input / output interface 1005 is further connected to the bus 1004. An input / output interface 1005 is connected to an input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010.

[0300] The input section 1006 consists of input switches, buttons, a microphone, an image sensor, etc. The output section 1007 consists of a display, a speaker, etc. The storage section 1008 consists of a hard disk or non-volatile memory, etc. The communication section 1009 consists of a network interface, etc. The drive 1010 drives removable media 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0301] In the computer 1000 configured as described above, the CPU 1001 loads, for example, a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004, and executes it, thereby performing the series of processes described above.

[0302] The program executed by computer 1000 (CPU 1001) can be provided by recording it on removable media 1011, such as a packaged media. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0303] In computer 1000, programs can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable media 1011 into the drive 1010. Alternatively, programs can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Furthermore, programs can be pre-installed in the ROM 1002 or the storage unit 1008.

[0304] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0305] Furthermore, in this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device in which multiple modules are housed in one enclosure, are both considered systems.

[0306] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0307] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0308] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0309] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0310] <Examples of configuration combinations> This technology can also be configured as follows:

[0311] (1) A recognition unit that extracts areas requiring attention around each of the systems based on multiple captured images taken at different locations by multiple systems, An image processing unit generates a first overhead image based on the captured image including the aforementioned area requiring attention. Information processing device. (2) The image processing unit generates the first overhead image by combining multiple images of areas requiring attention, each representing one of the aforementioned areas requiring attention. The information processing device described in (1) above. (3) The image processing unit places each of the attention-sensitive area images in the first overhead image at a position corresponding to its position in the original captured image. The information processing device described in (2) above. (4) The system that has captured the image including the area of ​​concern designated by the operator within the first overhead image is provided with an intermediary unit to mediate intervention by the operator. The information processing apparatus described in (3) above is further provided. (5) The operator's intervention may include at least one of the following: visual intervention, auditory intervention, physical intervention, or manipulative intervention. The information processing device described in (4) above. (6) The aforementioned image of the area requiring attention is an image extracted from the captured image. The information processing device described in any of (2) to (5) above. (7) The aforementioned image of the area requiring attention is a two-dimensional or three-dimensional model of an object located within that area. The information processing device described in any of (2) to (5) above. (8) The recognition unit estimates the degree of danger for each of the aforementioned areas requiring attention, The image processing unit changes the display mode of the image of the area requiring attention based on the degree of danger of the area requiring attention. The information processing apparatus described in any of (2) to (7) above. (9) The image processing unit generates a second overhead image showing the distribution of locations including the area requiring attention on a map. The information processing device described in any of (2) to (8) above. (10) The recognition unit estimates the degree of risk at each of the aforementioned locations. Based on the risk level of each of the aforementioned locations, either the first overhead view image or the second overhead view image is selected and transmitted to the operating terminal used by the operator. The information processing device described in (9) above. (11) The image processing unit generates the first overhead image by synthesizing the caution area image, which shows the caution area included in the points where the risk level is above a predetermined threshold. The information processing device described in (10) above. (12) The image processing unit generates the first overhead image, which shows the distribution of locations including the area requiring attention on a map. The information processing device described in (1) above. (13) The aforementioned area requiring attention includes at least one of the following: hazardous materials and objects that the system or the user of the system needs to check. The information processing device described in any of (1) to (12) above. (14) An accident section setting unit sets a first accident section including the location where the first accident occurred, based on the section in which the operator intervened in the vehicle, which is the system, to avoid the first accident; A learning unit that generates a learning model by learning the operator's actions on the vehicle in the first accident section, An intervention unit that uses the learning model to intervene in the operation of other vehicles in the first accident section, An information processing apparatus according to any one of (1) to (13) above, comprising: (15) The accident section setting unit, when the operator determines that no operational intervention is necessary in the first accident section, cancels the first accident section. The intervention unit stops intervening in the operation of the other vehicle when the first accident section is released. The information processing device described in (14) above. (16) If a second accident similar to the first accident occurs, the intervention unit uses the learning model to perform operational intervention on the vehicle in the second accident section, which includes the location where the second accident occurred. The information processing device described in (14) or (15) above. (17) A learning data generation unit generates learning data by assigning correct labels determined based on labels assigned to the data by multiple evaluators to the data used for training the learning model used by the recognition unit. The information processing apparatus further comprises any of the above (1) to (16). (18) The learning data generation unit provides a reward to the evaluator who, among the multiple evaluators, assigned the correct label. The information processing device described in (17) above. (19) Information processing device, Based on multiple captured images taken at different locations by multiple systems, a region requiring attention is extracted around each of the systems. Based on the captured image including the aforementioned area requiring attention, an overhead view image is generated. Information processing methods. (20) Based on multiple captured images taken at different locations by multiple systems, a region requiring attention is extracted around each of the systems. Based on the captured image including the aforementioned area requiring attention, an overhead view image is generated. A program that causes a computer to perform a process.

[0312] Furthermore, the effects described herein are merely illustrative and not limiting; other effects may also occur. [Explanation of symbols]

[0313] 1 Information processing system, 11-1 to 11-m system, 12-1 to 12-m, 13 Management server, 51 User support system, 61 External sensor, 62 Internal sensor, 65 Control unit, 66 Output unit, 101 Vehicle, 111 External sensor, 113 Internal sensor, 115 Recognition unit, 116 Monitoring information transmission unit, 120 Accident detection unit, 121 Situation judgment unit, 122 Route generation unit, 123 Vehicle control unit, 152 Control unit, 153 Display unit, 154 Voice output unit, 155 Operation input unit, 156 Imaging unit, 157 Voice input unit, 161 Output control unit, 162 Action recognition unit, 163 Intervention unit, 202 Control unit, 211 Recognition unit, 212 Image processing unit, 213 Accident interval setting unit, 214 Mediation unit, 215 Intervention unit, 216 Learning unit, 501 Information processing system, 513 Learning data generation server, 514-1 to 514-n Client terminals, 515 Learning data server

Claims

1. A recognition unit that extracts areas requiring attention around each of the systems based on multiple captured images taken at different locations by multiple systems, An image processing unit generates a first monitoring image by synthesizing a plurality of caution region images, each representing a plurality of caution region, based on the captured image including the aforementioned caution region, and arranging each of the caution region images at a position corresponding to its position in the original captured image. Information processing device.

2. The system that has captured the image containing the area of ​​concern designated by the operator within the first monitoring image is provided with an intermediary unit to mediate intervention by the operator. The information processing apparatus according to claim 1, further comprising:

3. The operator's intervention may include at least one of the following: visual intervention, auditory intervention, physical intervention, or manipulative intervention. The information processing apparatus according to claim 2.

4. The aforementioned image of the area requiring attention is an image extracted from the captured image. The information processing apparatus according to claim 1.

5. The aforementioned image of the area requiring attention is a two-dimensional or three-dimensional model of an object present in the area requiring attention. The information processing apparatus according to claim 1.

6. The recognition unit estimates the degree of danger for each of the aforementioned areas requiring attention, The image processing unit changes the display mode of the image of the area requiring attention based on the degree of danger of the area requiring attention. The information processing apparatus according to claim 1.

7. The image processing unit generates a second monitoring image that shows the distribution of locations including the area of ​​concern on a map. The information processing apparatus according to claim 1.

8. The recognition unit estimates the degree of risk at each of the aforementioned locations. Based on the risk level of each of the aforementioned locations, either the first monitoring image or the second monitoring image is selected and transmitted to the operating terminal used by the operator. The information processing apparatus according to claim 7.

9. The image processing unit generates the first monitoring image by synthesizing the caution area image, which shows the caution area included in the points where the risk level is above a predetermined threshold. The information processing apparatus according to claim 8.

10. The aforementioned area requiring attention includes at least one of the following: hazardous materials and objects that the system or the user of the system needs to check. The information processing apparatus according to claim 1.

11. An accident section setting unit sets a first accident section including the location where the first accident occurred, based on the section in which the operator intervened in the vehicle, which is the system, to avoid the first accident; A learning unit that generates a learning model by learning the operator's actions on the vehicle in the first accident section, An intervention unit that uses the learning model to perform operational intervention on other vehicles in the first accident section, The information processing apparatus according to claim 1, comprising:

12. The accident section setting unit, when the operator determines that no operational intervention is necessary in the first accident section, cancels the first accident section. The intervention unit stops intervening in the operation of the other vehicle when the first accident section is released. The information processing apparatus according to claim 11.

13. When a second accident similar to the first accident occurs, the intervention unit uses the learning model to perform operational intervention on the vehicle in the second accident section, which includes the location where the second accident occurred. The information processing apparatus according to claim 11.

14. A learning data generation unit generates learning data by assigning correct labels determined based on labels assigned to the data by multiple evaluators to the data used for training the learning model used by the recognition unit. The information processing apparatus according to claim 1, further comprising:

15. The learning data generation unit provides a reward to the evaluator who, among the multiple evaluators, assigned the correct label. The information processing apparatus according to claim 14.

16. Information processing device, Based on multiple captured images taken at different locations by multiple systems, the system extracts areas requiring attention around each of the systems, and Based on the captured image including the aforementioned areas of concern, multiple images representing multiple areas of concern are synthesized, and a monitoring image is generated in which each of the aforementioned areas of concern is placed at a position corresponding to its position in the original captured image. Information processing methods including

17. Based on multiple captured images taken at different locations by multiple systems, the system extracts areas requiring attention around each of the systems, and Based on the captured image including the aforementioned areas of concern, multiple images representing multiple areas of concern are synthesized, and a monitoring image is generated in which each of the aforementioned areas of concern is placed at a position corresponding to its position in the original captured image. A program that causes a computer to perform a process that includes [a specific action].

Images