Autonomous driving system, autonomous driving method, moving body, and autonomous driving program

Abstract

To enable safer autonomous driving.SOLUTION: A target moving body 120 determines whether monitoring is necessary by referencing self position data and map data, and transmits a monitoring request along with travel status data. A surrounding situation monitoring device 130 receives the monitoring request, monitors the surrounding situation of the target moving body, and transmits monitoring data. An autonomous driving control device 400 receives the travel status data and the monitoring data, determines travel obstacles for the target moving body based on the travel status data and the monitoring data, and transmits control data to the target moving body.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a system for realizing safe driving through automated driving. [Background technology]

[0002] There is a shortage of drivers for delivery trucks and taxis, and this problem can be solved by realizing autonomous driving. To realize autonomous driving, it is necessary to be able to check for obstacles in the blind spots of autonomous vehicles.

[0003] Patent Document 1 discloses an information provision system for compensating for blind spots of an autonomous vehicle to prevent collision with a moving object such as an oncoming vehicle. In this information providing system, roadside devices installed near roads detect moving objects. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-50627 Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 does not mention the timing at which the roadside device starts detecting a moving object, and it is assumed that the roadside device is always detecting a moving object. As a result, detection of moving objects is performed even during times when there is no risk of collision with moving objects, which requires communication and processing of a huge amount of information, including unnecessary detection information, and increases the load on the information provision system.

[0006] The present disclosure aims to enable safer autonomous driving by monitoring the surroundings of an autonomously driving vehicle using sensors installed independently of the autonomously driving vehicle when the autonomously driving vehicle passes through an area where there is a blind spot from the autonomously driving vehicle. [Means for solving the problem]

[0007] The autonomous driving system of the present disclosure includes a target moving body that is a moving body that travels autonomously, a surrounding situation monitoring device, and an autonomous driving control device, The target moving body is a group of mobile sensors; an on-board device that determines whether monitoring by the surrounding situation monitoring device is necessary by referring to self-location data obtained by the mobile body sensor group and indicating the position of the target mobile body, and map data indicating a monitoring area set as an area where the surrounding situation of the target mobile body should be monitored by the surrounding situation monitoring device when the target mobile body passes, and when it determines that monitoring by the surrounding situation monitoring device is necessary, transmits a monitoring request to the surrounding situation monitoring device and transmits driving situation data obtained by the mobile body sensor group and indicating the driving situation of the target mobile body to the automatic driving control device; Equipped with The surrounding situation monitoring device is A group of monitoring sensors; a monitoring control device that receives the monitoring request, causes the monitoring sensors to monitor the surrounding conditions of the target moving object, and transmits monitoring data indicating the surrounding conditions of the target moving object to the autonomous driving control device; Equipped with The autonomous driving control device receives the driving situation data and the monitoring data, determines a driving obstacle for the target moving body based on the driving situation data and the monitoring data, and transmits control data notifying the target moving body of the driving obstacle, The vehicle-mounted device receives the control data and controls the travel of the target moving object based on the control data and surrounding situation data obtained by the group of moving object sensors and indicating the surrounding situation of the target moving object. [Effects of the Invention]

[0008] According to the present disclosure, when an autonomously driving vehicle passes through an area where there is a blind spot from the autonomously driving vehicle, it is possible to monitor the surroundings of the autonomously driving vehicle using sensors installed independently from the autonomously driving vehicle, thereby achieving safer autonomous driving. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a configuration diagram of an autonomous driving system 100 according to a first embodiment. [Figure 2] FIG. 2 is a configuration diagram of an in-vehicle device 200 according to the first embodiment. [Figure 3] FIG. 2 is a configuration diagram of a monitoring and control device 300 according to the first embodiment. [Figure 4] FIG. 1 is a configuration diagram of an automatic driving control device 400 according to a first embodiment. [Figure 5] 4 is a flowchart of the vehicle-mounted device 200 according to the first embodiment. [Figure 6] 3 is a flowchart of the monitoring control device 300 according to the first embodiment. [Figure 7] 4 is a flowchart of the automatic driving control device 400 according to the first embodiment. [Figure 8] FIG. 1 is a configuration diagram of an autonomous driving system 100 according to a second embodiment. [Figure 9] 10 is a flowchart of the vehicle-mounted device 200 according to the second embodiment. [Figure 10] 10 is a flowchart of the monitoring control device 300 according to the second embodiment. [Figure 11] FIG. 10 is a configuration diagram of an autonomous driving system 100 according to a third embodiment. [Figure 12] 10 is a flowchart of the monitoring control device 300 according to the third embodiment. [Figure 13] 10 is a flowchart of an automatic driving control device 400 according to a fourth embodiment. [Figure 14] 10 is a flowchart of the vehicle-mounted device 200 according to the fourth embodiment. [Figure 15] FIG. 10 is a configuration diagram of an automatic driving system 100 according to a fifth embodiment. [Figure 16] FIG. 10 is a configuration diagram of an automatic driving control device 400 according to a fifth embodiment. [Figure 17] 10 is a flowchart of the automatic driving control device 400 in the fifth embodiment. [Figure 18] FIG. 20 is a diagram showing a part of the functional configuration of an autonomous driving control device 400 according to a sixth embodiment. [Figure 19] FIG. 20 is a diagram showing an example of a video stream AI processing pipeline 439 according to the sixth embodiment. [Figure 20] FIG. 20 is a diagram showing an example of a video stream AI processing pipeline 439 according to the sixth embodiment. [Figure 21] FIG. 20 is a diagram showing an example of a video stream AI processing pipeline 439 according to the sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] In the embodiments and drawings, the same or corresponding elements are denoted by the same reference numerals. The description of elements denoted by the same reference numerals as those already described will be omitted or simplified as appropriate. Arrows in the drawings primarily indicate the flow of data or the flow of processing.

[0011] Embodiment 1 The autonomous driving system 100 will be described with reference to FIGS. 1 to 7. FIG.

[0012] ***Configuration Description*** The configuration of the autonomous driving system 100 will be described with reference to FIG. The autonomous driving system 100 includes a target moving object 120, a surrounding situation monitoring device 130, an autonomous driving control device 400, and a map database 110.

[0013] The target moving body 120 is a moving body that travels on the ground by automatic driving. For example, the target moving object 120 is an autonomous car, bus, or taxi.

[0014] The target moving object 120 includes an in-vehicle device 200 and a moving object sensor group 290 . The mobile body sensor group 290 is one or more types of sensors mounted on the target mobile body 120 . For example, the mobile sensor group 290 includes a depth camera 291, a lidar 292, and a positioning sensor 293. The positioning sensor 293 is a sensor that measures the position of the target moving object 120. For example, the positioning sensor 293 performs positioning using a satellite positioning system. An example of the satellite positioning system is the Global Positioning System (GPS). The mobile body sensor group 290 is used to grasp the traveling conditions of the target mobile body 120 and the conditions around the target mobile body 120 . The driving conditions include position, speed, and acceleration. The surroundings include the positions of objects present in the surroundings.

[0015] The configuration of the vehicle-mounted device 200 will be described with reference to FIG. The vehicle-mounted device 200 is a computer including hardware such as a processor 201, a memory 202, an auxiliary storage device 203, a communication device 204, and an input / output interface 205. These pieces of hardware are connected to one another via signal lines.

[0016] The processor 201 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 201 is a CPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit.

[0017] The memory 202 is a volatile or non-volatile storage device. The memory 202 is also called a primary storage device or a main memory. For example, the memory 202 is a RAM. Data stored in the memory 202 is saved in the secondary storage device 203 as needed. RAM is an abbreviation for Random Access Memory.

[0018] The auxiliary storage device 203 is a non-volatile storage device. For example, the auxiliary storage device 203 is a ROM, a HDD, a flash memory, or a combination thereof. Data stored in the auxiliary storage device 203 is loaded into the memory 202 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0019] The auxiliary storage device 203 is a receiver and a transmitter. For example, the auxiliary storage device 203 is a communication chip or a NIC. The communication of the vehicle-mounted device 200 is performed using a communication device 204. NIC is an abbreviation for Network Interface Card.

[0020] The input / output interface 205 is a port to which an input device and an output device are connected. For example, a mobile sensor group 290 is connected to the input / output interface 205.

[0021] The vehicle-mounted device 200 includes elements such as a monitoring request unit 211, a control data acquisition unit 212, and a driving control unit 213. These elements are realized by software.

[0022] The auxiliary storage device 203 stores an automatic driving program for causing the computer to function as a monitoring request unit 211, a control data acquisition unit 212, and a driving control unit 213. The automatic driving program is loaded into the memory 202 and executed by the processor 201. The auxiliary storage device 203 also stores an OS. At least a part of the OS is loaded into the memory 202 and executed by the processor 201. The processor 201 executes the automatic driving program while executing the OS. OS is an abbreviation for Operating System.

[0023] Data of the automatic driving program (input data, output data, etc.) is stored in the storage unit 220. The memory 202 functions as the storage unit 220. However, a storage device such as the auxiliary storage device 203, a register in the processor 201, or a cache memory in the processor 201 may function as the storage unit 220 instead of or together with the memory 202.

[0024] The autonomous driving program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

[0025] Returning to Figure 1, we continue the explanation. The surrounding situation monitoring device 130 is a device that monitors the surrounding situation of the target moving object 120 . For example, the surrounding situation monitoring device 130 is a drone or a roadside unit. A drone is an air vehicle that flies autonomously in the sky. A roadside unit is a device that is installed on a road in a monitored area. The monitored area will be described later.

[0026] The surrounding situation monitoring device 130 includes a monitoring control device 300 and a group of monitoring sensors 390 . The monitoring sensor group 390 is one or more types of sensors mounted on the surroundings monitoring device 130 . For example, the monitoring sensor group 390 includes a depth camera 391 and an RGB camera 392 . The monitoring sensors 390 are used to monitor the surroundings of the target moving object 120 .

[0027] The configuration of the monitoring and control device 300 will be described with reference to FIG. The monitoring and control device 300 is a computer that includes hardware such as a processor 301, a memory 302, an auxiliary storage device 303, a communication device 304, and an input / output interface 305. These pieces of hardware are connected to one another via signal lines.

[0028] The processor 301 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 301 is a CPU. The memory 302 is a volatile or non-volatile storage device. The memory 302 is also called a primary storage device or a main memory. For example, the memory 302 is a RAM. Data stored in the memory 302 is saved in the secondary storage device 303 as needed. The auxiliary storage device 303 is a non-volatile storage device. The auxiliary storage device 303 is also called storage. For example, the auxiliary storage device 303 is a ROM, a HDD, a flash memory, or a combination thereof. Data stored in the auxiliary storage device 303 is loaded into the memory 302 as needed. The communication device 304 is a receiver and a transmitter. For example, the communication device 304 is a communication chip or a NIC. The communication of the monitoring and control device 300 is performed using the communication device 304. The input / output interface 305 is a port to which an input device and an output device are connected.

[0029] The monitoring control device 300 comprises elements such as a request receiving unit 311, a monitoring control unit 312, a monitoring data generating unit 313, and a monitoring data providing unit 314. These elements are realized by software.

[0030] The auxiliary storage device 303 stores a monitoring program for causing the computer to function as a request receiving unit 311, a monitoring control unit 312, a monitoring data generating unit 313, and a monitoring data providing unit 314. The monitoring program is loaded into the memory 302 and executed by the processor 301. The auxiliary storage device 303 also stores an OS. At least a part of the OS is loaded into the memory 302 and executed by the processor 301. The processor 301 executes the monitoring program while running the OS.

[0031] The input and output data of the monitoring program is stored in the storage unit 320 . The memory 302 functions as the storage unit 320. However, a storage device such as the auxiliary storage device 303, a register in the processor 301, or a cache memory in the processor 301 may function as the storage unit 320 instead of or together with the memory 302.

[0032] The monitoring program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

[0033] Returning to Figure 1, the explanation will continue. The automatic driving control device 400 is a device that manages and controls the automatic driving of the target moving body 120.

[0034] The configuration of the automatic driving control device 400 will be described with reference to FIG. The autonomous driving control device 400 is a computer that includes hardware such as a processor 401, a memory 402, an auxiliary storage device 403, a communication device 404, and an input / output interface 405. These pieces of hardware are connected to each other via signal lines.

[0035] The processor 401 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 401 is a CPU. The memory 402 is a volatile or non-volatile storage device. The memory 402 is also called a primary storage device or a main memory. For example, the memory 402 is a RAM. Data stored in the memory 402 is saved in the secondary storage device 403 as needed. The auxiliary storage device 403 is a non-volatile storage device. The auxiliary storage device 403 is also called storage. For example, the auxiliary storage device 403 is a ROM, a HDD, a flash memory, or a combination thereof. Data stored in the auxiliary storage device 403 is loaded into the memory 402 as needed. The communication device 404 is a receiver and a transmitter. For example, the communication device 404 is a communication chip or a NIC. Communication of the autonomous driving control device 400 is performed using the communication device 404. The input / output interface 405 is a port to which an input device and an output device are connected.

[0036] The autonomous driving control device 400 includes elements such as a data reception unit 411, a driving obstacle determination unit 412, a control data generation unit 413, and a control data provision unit 414. These elements are realized by software.

[0037] The auxiliary storage device 403 stores a control program for causing the computer to function as a data receiving unit 411, a driving obstacle determination unit 412, and a control data generation unit 413. The control program is loaded into the memory 402 and executed by the processor 401. The auxiliary storage device 403 also stores an OS. At least a part of the OS is loaded into the memory 402 and executed by the processor 401. The processor 401 executes the control program while running the OS.

[0038] Input and output data of the control program are stored in the storage unit 420 . The memory 402 functions as the storage unit 420. However, a storage unit such as the auxiliary storage unit 403, a register in the processor 401, or a cache memory in the processor 401 may function as the storage unit 420 instead of the memory 402 or together with the memory 402.

[0039] The control program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

[0040] Returning to Figure 1, the explanation will continue. The map database 110 is a device that manages map data 111 . The map data 111 will be described later.

[0041] ***Explanation of Operation*** The operation procedures of the automatic driving system 100 correspond to an automatic driving method. The operation procedures of the automatic driving system 100 also correspond to processing procedures by the automatic driving system program. The automatic driving system program includes an automatic driving program, a monitoring program, and a control program.

[0042] The operation of the vehicle-mounted device 200 will be described. First, the premise of the operation of the vehicle-mounted device 200 will be described. The mobile body sensor group 290 periodically observes the traveling conditions of the target mobile body 120 and the conditions around the target mobile body 120. For example, a depth camera 291 measures the surroundings of the target mobile body 120, a lidar 292 measures the distance to each point around the target mobile body 120, and a positioning sensor 293 performs satellite positioning. The mobile sensor group 290 provides driving condition data and surrounding condition data. The traveling condition data is data that indicates the traveling condition of the target moving object 120. The traveling condition data indicates the position, speed, acceleration, etc. of the target moving object 120. The position of the target moving object 120 is referred to as the self-position. Data that indicates the self-position is referred to as self-position data. The surrounding situation data is data that indicates the surrounding situation of the target moving object 120. The surrounding situation data indicates various objects and the like observed by the moving object sensor group 290 mounted on the target moving object 120. The driving control unit 213 controls the driving of the target moving object 120 based on the driving plan data and the surrounding situation data. The driving plan data indicates the planned driving route of the target moving object 120 .

[0043] The procedure of the operation of the vehicle-mounted device 200 will be described with reference to FIG. Steps S121 to S125 are repeatedly executed.

[0044] In step S121, the monitoring request unit 211 refers to the self-location data and the map data 111 and determines whether monitoring by the surrounding situation monitoring device 130 is necessary. The map data 111 is obtained by communication with the map database 110 .

[0045] The map data 111 indicates one or more surveillance areas. The monitoring area is set as an area where the surrounding conditions of the target moving object 120 should be monitored by the surrounding conditions monitoring device 130 when the target moving object 120 passes through. Specifically, the monitoring area is a place where a blind spot from the target moving object 120 occurs, such as an area before and after a curve.

[0046] Whether or not monitoring by the surroundings monitoring device 130 is necessary is determined as follows. The monitoring request unit 211 determines whether the target moving object 120 has entered the monitoring area. For example, the monitoring request unit 211 maps the position of the target moving object 120 on the map shown in the map data 111. Then, the monitoring request unit 211 determines whether the position of the target moving object 120 has been mapped within the monitoring area. When the target moving object 120 enters the monitoring area, the monitoring request unit 211 determines that monitoring by the surrounding situation monitoring device 130 is necessary.

[0047] If it is determined that monitoring by the surroundings monitoring device 130 is necessary, the process proceeds to step S122.

[0048] In step S122, the monitoring request unit 211 transmits a monitoring request to the surrounding situation monitoring device . The monitoring request is data for requesting the surrounding situation monitoring device 130 to monitor the surrounding situation of the target moving object 120 . The monitoring request indicates the identifier of the target mobile object 120, the location of the target mobile object 120, the range of the monitoring area, and the like.

[0049] In step S123, the control data acquisition unit 212 transmits the driving plan data and the driving situation data to the automatic driving control device 400.

[0050] Steps S122 and S123 may be performed in reverse order.

[0051] In step S124, the control data acquisition unit 212 receives the control data transmitted from the autonomous driving control device 400. The control data is data for notifying the target moving body 120 of a road obstruction. For example, the control data indicates the positions of objects (obstacles) that may hinder the travel of the target moving body 120.

[0052] In step S125, the driving control unit 213 controls the traveling of the target moving object 120 based on the control data and the surrounding situation data.

[0053] The travel of the target moving object 120 is controlled as follows. First, the driving control unit 213 changes the driving plan. For example, the driving control unit 213 changes the planned driving route to a route that can avoid the obstacles indicated in the control data and the objects indicated in the surrounding situation data. Next, the driving control unit 213 determines control (driving control) for the driving device of the target moving body 120 so that the target moving body 120 can travel according to the changed travel plan. Examples of the driving device are a steering device (handle) and a motor (engine). Then, the driving control unit 213 controls the driving device of the target moving object 120 in accordance with the determined driving control.

[0054] The procedure of the operation of the monitoring and control device 300 will be described with reference to FIG. Steps S131 to S134 are executed every time a monitoring request is transmitted from the target moving object 120.

[0055] In step S131, the request receiving unit 311 receives a monitoring request transmitted from the target moving object 120.

[0056] In step S132 , the monitoring control unit 312 causes the monitoring sensors 390 to monitor the surrounding conditions of the target moving object 120 .

[0057] The surroundings of the target moving object 120 are monitored as follows. The monitoring control unit 312 outputs an operation command to the monitoring sensor group 390 . The monitoring sensors 390 start operating upon receiving an operation command. For example, the depth camera 391 and the RGB camera 392 each capture images of the surroundings of the target moving object 120.

[0058] The monitoring sensor group 390 monitors the surrounding conditions of the target moving object 120 to obtain sensor data.

[0059] In step S133, the monitoring data generation unit 313 generates monitoring data using the sensor data obtained from the monitoring sensor group 390. The monitoring data is data that indicates the surrounding conditions of the target moving object 120. The monitoring data indicates the positions of various objects detected by the monitoring sensors 390, etc.

[0060] The monitoring data is generated as follows: First, the monitoring data generation unit 313 executes object detection using sensor data at each time, and detects each object present around the target moving object 120. Next, the monitoring data generation unit 313 calculates the speed of each object based on the position of each object at each time, and predicts the movement path of each object. The monitoring data generator 313 then generates data indicating the position, speed, and predicted path of each detected object. The generated data is monitoring data.

[0061] In step S134, the monitoring data providing unit 314 transmits the monitoring data to the autonomous driving control device 400.

[0062] The procedure of the operation of the automatic driving control device 400 will be described with reference to FIG. Steps S141 to S145 are executed every time the target moving object 120 transmits travel plan data and travel situation data and the surrounding situation monitoring device 130 transmits monitoring data.

[0063] In step S141, the data receiving unit 411 receives the driving plan data and driving situation data transmitted from the target moving object 120.

[0064] In step S142, the data receiving unit 411 receives the monitoring data transmitted from the surrounding situation monitoring device 130.

[0065] Steps S141 and S142 may be performed in reverse order.

[0066] In step S143, the traveling obstacle determination unit 412 determines whether or not there is a traveling obstacle for the target moving object 120 based on the traveling condition data and the monitoring data.

[0067] The presence or absence of a roadblock for the target moving object 120 is determined as follows. The traveling obstacle determination unit 412 determines whether there is an obstacle facing the target moving body 120 within a proximity range from the position of the target moving body 120. The proximity range is a range that has the position of the target moving body 120 as its base point, and the size of the proximity range is determined in advance. The position of the target moving body 120 and the position of each object are managed using a position database. The position database is managed by the autonomous driving control device 400. The traveling obstacle determination unit 412 maps the position of the target moving body 120 and the position of each object on a map shown in the position database. Then, the traveling obstacle determination unit 412 makes a determination by referring to the map on which the position of the target moving body 120 and the position of each object are mapped. If there is an obstacle facing the target moving object 120 within the proximity range from the position of the target moving object 120, the traveling obstacle determination unit 412 determines that there is an obstacle to the traveling of the target moving object 120.

[0068] If it is determined that there is an obstacle to the travel of the target moving object 120, the process proceeds to step S144. If it is determined that there is no obstacle to the travel of the target moving object 120, the process ends.

[0069] In step S144, the control data generation unit 413 generates control data for notifying the target moving body 120 of a travel obstacle.

[0070] In step S145 , the control data providing unit 414 transmits the control data to the target moving object 120 .

[0071] ***Effects of the First Embodiment*** According to the first embodiment, when the target moving object 120 passes through a monitoring area where a blind spot exists from the target moving object 120, the surroundings of the target moving object 120 can be monitored using the monitoring sensor group 390 provided independently from the target moving object 120. This makes it possible to realize safer automated driving.

[0072] ***Supplement to the first embodiment*** The autonomous driving system 100 includes a target moving object 120, a surrounding situation monitoring device 130, an autonomous driving control device 400, and a map database 110. The target moving object 120 is a moving object capable of autonomous driving. The target moving object 120 has sensors for detecting the surrounding environment, such as an optical sensor and a LIDAR, and a communication function. The surrounding situation monitoring device 130 has a sensor such as a camera that exists independently of the target moving object 120, and a communication function. The autonomous driving control device 400 has communication and calculation functions, and notifies the target moving body 120 of the control data necessary for planning a driving plan. The map database 110 has a storage function and a communication function. The map database 110 stores map data 111 used to determine the need to check the surrounding conditions. The map database 110 can communicate with the target moving object 120, the surrounding conditions monitoring device 130, and the autonomous driving control device 400. The target moving object 120 transmits information such as its own position, speed, and acceleration to the autonomous driving control device 400. The target moving object 120 also maps its own position on the map shown in the map data 111 and determines whether monitoring of the surrounding situation is necessary. If monitoring of the surrounding situation is necessary, the target moving object 120 transmits a monitoring request to the surrounding situation monitoring device 130. The surrounding situation monitoring device 130 may be fixed to the side of the road, or may be an aircraft launched from the target moving object 120. The surrounding situation monitoring device 130 acquires information such as the position, speed, and acceleration of obstacles that affect the travel plan of the target moving object 120 from the monitoring sensor group 390. The surrounding situation monitoring device 130 then transmits the information such as the position, speed, and acceleration of the obstacles to the autonomous driving control device 400. The surrounding situation monitoring device 130 also maps the positions of the obstacles on a map shown in the map data 111. The autonomous driving control device 400 determines a driving obstacle for the target moving body 120 based on information such as the position (including future position) of the target moving body 120 and information such as the position (including future position) of the obstacle. Then, the autonomous driving control device 400 generates control data based on the determination result of the driving obstacle and transmits the control data to the target moving body 120.

[0073] Embodiment 2 The embodiment in which the surrounding situation monitoring device 130 is a drone 131 will be described with reference to Figs. 8 to 10, focusing mainly on the differences from the first embodiment.

[0074] ***Configuration Description*** The configuration of the automatic driving system 100 will be described with reference to FIG. The autonomous driving system 100 is equipped with a drone 131 as a surrounding situation monitoring device 130. Drone 131 is an autonomous flying vehicle that flies in the sky. The target moving object 120 includes a drone port 121 in addition to the elements described in the first embodiment. The drone port 121 is a facility where the processor 301 is stored and where the processor 301 takes off and lands. The drone port 121 has a charging port for charging the drone 131.

[0075] ***Explanation of Operation*** The procedure of the operation of the vehicle-mounted device 200 will be described with reference to FIG. Steps S221 to S225 correspond to steps S121 to S125 in the first embodiment.

[0076] In step S221, the monitoring request unit 211 refers to the self-position data and the map data 111 and determines whether monitoring by the drone 131 is necessary.

[0077] If it is determined that monitoring by the drone 131 is necessary, the process proceeds to step S222.

[0078] In step S222, the monitoring request unit 211 transmits a monitoring request to the drone 131.

[0079] The monitoring request includes self-location data and monitoring area data. The monitoring area data indicates the monitoring area through which the target mobile object 120 passes.

[0080] In step S223, the control data acquisition unit 212 transmits the driving plan data and driving situation data to the automatic driving control device 400.

[0081] In step S224, the control data acquisition unit 212 receives the control data transmitted from the autonomous driving control device 400.

[0082] In step S225, the driving control unit 213 controls the traveling of the target moving object 120 based on the control data and the surrounding situation data.

[0083] The procedure of the operation of the monitoring and control device 300 will be described with reference to FIG. Steps S231 to S234 correspond to steps S131 to S134 in the first embodiment.

[0084] In step S231, the request receiving unit 311 receives a monitoring request transmitted from the target moving object 120.

[0085] The monitoring request includes self-location data and monitoring area data.

[0086] In step S232, the monitoring control unit 312 causes the monitoring sensors 390 to monitor the surrounding conditions of the target moving object 120.

[0087] The monitoring control unit 312 causes the drone 131 to fly over the monitoring area in order to have the monitoring sensor group 390 monitor the surrounding conditions of the target moving object 120.

[0088] Drone 131 flies over the surveillance area as follows: First, the monitoring control unit 312 plans a flight path for the drone 131 based on the self-position data and the monitoring area data. The planned flight route is a route from taking off from the drone port 121 of the target moving body 120, flying over the monitored area, and landing at the drone port 121 of the target moving body 120. The monitoring control unit 312 then causes the drone 131 to fly along the planned flight path. To this end, the monitoring control unit 312 controls the driving device of the drone 131.

[0089] In step S233, the monitoring data generation unit 313 generates monitoring data using the sensor data obtained from the monitoring sensor group 390.

[0090] In step S234, the monitoring data providing unit 314 transmits the monitoring data to the autonomous driving control device 400.

[0091] ***Effects of the Second Embodiment*** According to the second embodiment, it becomes possible to monitor from the sky a monitoring area that is a blind spot from the target moving object 120 traveling on the ground.

[0092] ***Supplement to Embodiment 2*** The ambient situation monitoring device 130 is an autonomous flying vehicle. Upon receiving a monitoring request, the autonomously driven aircraft calculates a flight plan based on the information on the monitoring area that triggered the monitoring request, and then begins flight. The autonomously driven aircraft determines the position, speed, and acceleration of obstacles from data obtained from its onboard sensors, and transmits the position, speed, and acceleration of the obstacles to the autonomous driving control device 400. The target moving object 120 has a port where autonomously driven aircraft can take off and land. The target moving object 120 acquires location information of areas where blind spots occur from map data 111, and launches the autonomously driven aircraft when it enters an area where a blind spot occurs. The autonomously driven aircraft then monitors the situation in the blind spot area to determine the position, speed, and acceleration of any obstacles, and transmits the position, speed, and acceleration of the obstacles to the autonomous driving control device 400.

[0093] Embodiment 3 The embodiment in which the position of an obstacle is detected using a color sensor will be described below, focusing mainly on the differences from the first embodiment, with reference to FIGS. 11 and 12. FIG.

[0094] ***Configuration Description*** The configuration of the automatic driving system 100 will be described with reference to FIG. In the surroundings monitoring device 130 , the monitoring sensor group 390 includes a color sensor 393 . The color sensor 393 is an RGB camera or an RGB-D camera. R stands for red, G stands for green, B stands for blue. D stands for depth.

[0095] ***Explanation of Operation*** The procedure of the operation of the monitoring and control device 300 will be described with reference to FIG. Steps S331 to S334 correspond to steps S131 to S134 in the first embodiment.

[0096] In step S331, the request receiving unit 311 receives a monitoring request transmitted from the target moving object 120.

[0097] In step S332, the monitoring control unit 312 causes the monitoring sensors 390 to monitor the surrounding conditions of the target moving object 120.

[0098] In step S333, the monitoring data generation unit 313 generates monitoring data using the sensor data obtained from the monitoring sensor group 390.

[0099] The monitoring data indicates the position of each object present around the target moving object 120, etc.

[0100] The position of each object is estimated as follows: The sensor data obtained by the color sensor 393 is referred to as color sensor data. First, the monitoring data generation unit 313 executes object detection using color sensor data to detect objects present around the target moving object 120 and the plane where the objects are located. The monitoring data generation unit 313 then estimates the position of the object on the detected plane.

[0101] In step S334, the monitoring data providing unit 314 transmits the monitoring data to the autonomous driving control device 400.

[0102] ***Effects of the Third Embodiment*** According to the third embodiment, it is possible to more accurately detect the position of each object that may become an obstacle.

[0103] ***Supplement to embodiment 3*** If the connected sensor is an RGB sensor, the surroundings monitoring device 130 detects a horizontal plane and estimates the coordinates of the detected obstacle relative to the detected horizontal plane. However, the estimation may also be performed by the autonomous driving control device 400. When the sensor that exists independently of the target moving body 120 is a camera that obtains RGB information, the surrounding situation monitoring device 130 performs object detection, detects a plane based on the RGB information, and estimates the position of the object relative to the detected plane. When the sensor that exists independently of the target moving body 120 is a camera that obtains RGB-D information, the surrounding situation monitoring device 130 performs object detection, detects a plane based on the RGB-D information, and estimates the position of the object relative to the detected plane.

[0104] The third embodiment may be implemented in combination with the second embodiment. That is, the surrounding situation monitoring device 130 may be a drone 131, and the target moving object 120 may include a drone port 121.

[0105] Embodiment 4 The manner in which the target moving object 120 deals with a road obstacle depending on the type of the obstacle will be described below with reference to Figs. 13 and 14, mainly in terms of differences from the first embodiment.

[0106] ***Configuration Description** The configuration of the automatic driving system 100 is the same as that in the first embodiment. However, the map data 111 indicates one or more escape zones.

[0107] ***Explanation of Operation*** The procedure of the operation of the automatic driving control device 400 will be described with reference to FIG. Steps S441 to S445 correspond to steps S141 to S145 in the first embodiment.

[0108] In step S441, the data receiving unit 411 receives the driving plan data and driving situation data transmitted from the target moving object 120.

[0109] In step S442, the data receiving unit 411 receives the monitoring data transmitted from the surrounding situation monitoring device .

[0110] In step S443, the traveling obstacle determination unit 412 determines whether or not there is a traveling obstacle for the target moving object 120 based on the traveling condition data and the monitoring data.

[0111] If it is determined that there is a traveling obstacle for the target moving object 120, the traveling obstacle determination unit 412 determines the type of obstacle based on the monitoring data. Specifically, the traveling obstacle determination unit 412 determines whether the obstacle is a moving object or a non-moving object. Examples of moving objects are vehicles and people. An example of a non-moving object is falling rocks. The traveling obstacle determination unit 412 also determines whether there are multiple obstacles. An example of multiple obstacles is multiple pedestrians. Furthermore, the traveling obstacle determination unit 412 determines the type of road (target road) on which the target moving object 120 and obstacles exist, based on the position database. Specifically, the traveling obstacle determination unit 412 determines whether the target road is a narrow road with a width of only one lane.

[0112] If it is determined that there is an obstacle to the travel of the target moving object 120, the process proceeds to step S444. If it is determined that there is no obstacle to the travel of the target moving object 120, the process ends.

[0113] In step S444, the control data generation unit 413 generates control data for notifying the target moving body 120 of a travel obstacle.

[0114] The control data indicates the type of roadway obstruction, etc.

[0115] The type of driving impairment is determined as follows: If the target road is a narrow road and the obstacle is a moving object, the control data generation unit 413 determines that the type of driving obstacle is "impossible to pass each other." "Impossible to pass each other" means that the target moving body 120 and the obstacle cannot pass each other while traveling. If the target road is a narrow road and the obstacle is an immovable object, the control data generation unit 413 determines the type of driving obstacle as "impassable." "Impassable" means that the target moving body 120 cannot pass through the obstacle by avoiding it. If there are multiple obstacles, the control data generation unit 413 transmits control data indicating "slow down required" as control information. "slow down required" means that slow down is required to avoid the obstacles.

[0116] In step S445 , the control data providing unit 414 transmits the control data to the target moving object 120 .

[0117] The procedure of the operation of the vehicle-mounted device 200 will be described with reference to FIG. Steps S421 to S425 correspond to steps S121 to S125 in the first embodiment.

[0118] In step S421, the monitoring request unit 211 refers to the self-position data and the map data 111 and determines whether monitoring by the surrounding situation monitoring device 130 is necessary.

[0119] If it is determined that monitoring by the surroundings monitoring device 130 is necessary, the process proceeds to step S422.

[0120] In step S422, the monitoring request unit 211 transmits a monitoring request to the surrounding situation monitoring device .

[0121] In step S423, the control data acquisition unit 212 transmits the driving plan data and driving situation data to the autonomous driving control device 400.

[0122] In step S424, the control data acquisition unit 212 receives the control data transmitted from the autonomous driving control device 400.

[0123] The control data indicates the type of obstacle to travel of the target moving body 120, etc.

[0124] In step S425, the driving control unit 213 controls the traveling of the target moving object 120 based on the control data and the surrounding situation data.

[0125] The travel of the target moving object 120 is controlled according to the type of travel obstacle as follows. If the type of driving obstacle is “no passing”, the driving control unit 213 causes the target moving object 120 to retreat to the nearest escape zone indicated in the map data 111. If the type of driving obstacle is “impassable”, the driving control unit 213 causes the target moving object 120 to make a U-turn using the nearest escape zone indicated in the map data 111. If the type of road obstacle is "requires slow driving", the driving control unit 213 causes the target moving object 120 to drive slowly.

[0126] ***Effects of the Fourth Embodiment*** The fourth embodiment makes it possible to realize safe automatic driving depending on the type of driving obstacle.

[0127] ***Supplement to the fourth embodiment*** The autonomous driving control system 400 obtains information on the position, speed, and acceleration from the target moving body 120, and obtains information on the position, speed, and acceleration of an obstacle from the surroundings monitoring device 130. The autonomous driving control system 400 records future position information calculated based on the position information of the target moving body 120 and the obstacle, together with the predicted time, in a database that has version information and space-time axis information. Every time the position information and future predicted positions of the target moving body 120 and obstacles change, the autonomous driving control device 400 updates the version and updates the position information of the target moving body 120 and obstacles. If the position of the target moving body 120 is in a close relationship with the position of the obstacle at the time of updating, the autonomous driving control device 400 determines that there is a driving obstacle, and the target moving body 120 modifies the driving plan. If the type of driving obstacle is determined to be "passing is not permitted", the target moving body 120 acquires the position information of the escape zone from the map data 111 and changes the driving plan to take refuge in the escape zone. If the type of driving obstacle is determined to be "impassable," the target moving body 120 acquires the position information of the escape zone from the map data 111 and changes the driving plan to make a U-turn using the escape zone. If the type of road obstacle is determined to be "required to drive slowly," the target moving body 120 changes its driving plan to drive slowly.

[0128] The fourth embodiment may be implemented in combination with at least one of the second and third embodiments. That is, the surrounding situation monitoring device 130 may be a drone 131, and the target moving object 120 may include a drone port 121. The monitoring control device 300 may also use a color sensor 393 to detect the position of an obstacle.

[0129] Embodiment 5 The following describes a mode of notifying an obstacle of a travel obstacle of the target moving body 120, mainly in terms of differences from the first embodiment, with reference to FIGS. 15 to 17. FIG.

[0130] ***Configuration Description** The configuration of the automatic driving system 100 will be described with reference to FIG. The autonomous driving system 100 includes one or more moving objects 101 and one or more mobile terminals 102. The moving object 101 is a moving object similar to the target moving object 120 . The mobile terminal 102 is carried by a pedestrian. For example, the mobile terminal 102 is a smartphone. An application program for the autonomous driving system 100 is executed on the smartphone. Information on each moving body 101 and information on each mobile terminal 102 is registered in advance in the automatic driving control device 400.

[0131] The configuration of the automatic driving control device 400 will be described with reference to FIG. The autonomous driving control device 400 further includes an element called a driving obstacle notification unit 415. The automatic driving control program further causes the computer to function as a driving obstacle notification unit 415.

[0132] ***Explanation of Operation*** The following describes the operation of the autonomous driving control system 400. First, the premise for the operation of the autonomous driving control system 400 will be described. Each moving body 101 periodically transmits its own position data to the automatic driving control device 400. The own position data indicates the position of the moving body 101. Each mobile terminal 102 periodically transmits its own location data to the autonomous driving control system 400. The own location data indicates the locations of the pedestrian and the mobile terminal 102.

[0133] The procedure of the operation of the automatic driving control device 400 will be described with reference to FIG. Steps S542 to S546 correspond to steps S141 to S145 in the first embodiment.

[0134] In step S541, the data receiving unit 411 receives the self-location data from each moving object 101. Furthermore, the data receiving unit 411 receives the self-location data from each mobile terminal 102 .

[0135] In step S542, the data receiving unit 411 receives the driving plan data and driving situation data transmitted from the target moving object 120.

[0136] In step S543, the data receiving unit 411 receives the monitoring data transmitted from the surrounding situation monitoring device .

[0137] In step S544, the traveling obstacle determination unit 412 determines whether or not there is a traveling obstacle for the target moving object 120 based on the traveling condition data and the monitoring data.

[0138] In step S545, the control data generation unit 413 generates control data for notifying the target moving body 120 of a travel obstacle.

[0139] In step S546 , the control data providing unit 414 transmits the control data to the target moving object 120 .

[0140] In step S547, the traveling obstacle notification unit 415 determines whether the obstacle is one of the moving bodies 101 based on the self-position data of each of the one or more moving bodies 101. If it is determined that the obstacle is one of the moving bodies 101, the traveling obstacle notification unit 415 transmits notification data to the moving body 101 that is the obstacle. The notification data is data for notifying a travel obstacle of the target moving body 120. For example, the travel obstacle notification unit 415 generates control data for the moving body 101, which is an obstacle, as the notification data.

[0141] Furthermore, the driving obstacle notification unit 415 determines whether the obstacle is a pedestrian carrying one of the mobile terminals 102, based on the self-position data of each of the one or more mobile terminals 102. When it is determined that the obstacle is a pedestrian carrying one of the mobile terminals 102, the driving obstacle notification unit 415 transmits notification data to the mobile terminal 102 of the pedestrian who is the obstacle.

[0142] ***Effects of the Fifth Embodiment*** According to the fifth embodiment, moving objects and pedestrians that are obstacles to the target moving object 120 are notified of a road obstacle for the target moving object 120. This makes it possible to realize safer automated driving.

[0143] ***Supplement to embodiment 5*** If it is determined that there is a driving obstacle and that the obstacle is a human-driven moving body or a pedestrian, the autonomous driving control device 400 sends a notification to the smartphone carried by the human. Also, if the obstacle is an autonomously drivable moving body, the autonomous driving control device 400 sends information to instruct the human to change the driving plan. Obstacles can be classified as objects that can be expected to move or objects that cannot be expected to move. For example, potential obstacles include human-driven vehicles, autonomous vehicles (moving objects capable of autonomous driving), non-moving objects (parked vehicles, falling rocks, etc.), and other objects (pedestrians, cyclists, vulnerable road users, etc.).

[0144] The fifth embodiment may be implemented in combination with at least one of the second to fourth embodiments. That is, the surrounding situation monitoring device 130 may be a drone 131, and the target moving object 120 may include a drone port 121. The monitoring control device 300 may also detect the position of an obstacle using a color sensor 393. The target moving object 120 may also deal with a traveling obstacle depending on the type of the traveling obstacle.

[0145] Embodiment 6 The following describes an embodiment in which a video stream AI processing pipeline system is applied, focusing on the differences from embodiment 1, with reference to Figures 18 to 21. AI is an abbreviation for artificial intelligence.

[0146] ***Configuration Description*** The configuration of the automatic driving system 100 will be described with reference to FIG. The autonomous driving system 100 includes multiple surrounding condition monitoring devices 130. In Fig. 18, the autonomous driving system 100 includes four surrounding condition monitoring devices 130: one drone and three roadside units. The number of surrounding condition monitoring devices 130 is not limited to four. The monitoring sensor group 390 of each surroundings monitoring device 130 includes a video image sensor.

[0147] The autonomous driving control system 400 further includes elements such as a switcher unit 431, an object detection unit 432, a position estimation unit 433, an annotation unit 434, and a learning unit 435. These elements are realized by software. The autonomous driving control system 400 also includes a database 438. The database 438 has space and time axis information. The autonomous driving control device 400 has multiple types of object detection models. The object detection models are trained models generated by machine learning or the like in order to detect objects shown in images. In FIG. 18, the autonomous driving control device 400 has a general-purpose object detection model, a general-purpose road detection model, and a general-purpose human detection model. The road detection model is an object detection model for detecting roads. The human detection model is an object detection model for detecting people. The object detection models used are not limited to these models.

[0148] ***Explanation of Operation*** The operation of the multiple surrounding situation monitoring devices 130 and the automatic driving control device 400 will be described with reference to FIG.

[0149] The monitoring control device 300 of each surrounding condition monitoring device 130 transmits monitoring data, including a video stream obtained by a video image sensor, to the autonomous driving control device 400 (S133, S134 in FIG. 6).

[0150] In step S142 (see FIG. 7), the autonomous driving control system 400 receives monitoring data transmitted from each surrounding situation monitoring device 130. The monitoring data includes a video stream. In step S143 (see FIG. 7), the autonomous driving control device 400 uses the object detection model to detect obstacles to the travel of the target moving object 120, and determines whether there is a travel obstacle to the target moving object 120 using the obstacle detection result.

[0151] Obstacles are detected as follows: The input destination of the video stream from each surrounding condition monitoring device 130 is determined in advance in association with the surrounding condition monitoring device 130 that is the transmission source. The switcher unit 431 sends each video stream to an input-destination object detection model associated with the sending-side surroundings monitoring device 130. For example, the switcher unit 431 sends video stream A from roadside unit A to object detection and annotation, sends video stream B from roadside unit B to object detection, road detection, and annotation, and sends video stream C from roadside unit C to human detection and annotation. The object detection model used at the input destination associated with the surrounding situation monitoring device 130 that is the sender is called a corresponding model. The object detection unit 432 performs object detection for each video stream using a corresponding model and inputting the video stream. Position estimation unit 433 estimates the position (coordinate values) of the detected object based on the result of object detection, and records detection information of the detected object in database 438. The detection information indicates the type, position, etc. of the detected object. The user uses the autonomous driving control device 400 to refer to the detection information for each object recorded in the database 438, selects the object detection model they want to update, and specifies the selected object detection model to the autonomous driving control device 400. The selected object detection model is called the selected model. The switcher unit 431 sends the video stream input to the selection model to the annotation unit 434 . The annotation unit 434 uses a selection model to input a video stream and corrects the detection information of detected objects using annotations. The detection information is corrected as follows: The annotation unit 434 displays the video stream on a display. The user refers to the video stream, annotates the objects shown in the video stream, and inputs the annotation information to the autonomous driving control device 400. The annotation information indicates the position and label of the object. The label identifies the type of object. The annotation unit 434 receives the annotation information and corrects the detection information based on the annotation information. The learning unit 435 updates the selection model by learning using the video stream and the corrected detection information as learning data. For example, the parameters of the selection model are updated. As a result, a new version of the object detection model is generated. When a video stream is input to any type of object detection model, the object detection unit 432 receives the video stream as input and performs object detection using the object detection model for each version of the same type of object detection model. Then, detection information for each version of the object detection model is recorded in the database 438. The user uses the autonomous driving control device 400 to refer to each piece of detection information for each object recorded in the database 438. The user determines an object detection model to be used to determine a driving obstacle from multiple versions of object detection models. The user then specifies the determined object detection model to the autonomous driving control device 400. The determined object detection model is referred to as a determined model. Thereafter, the object detection unit 432 receives the video stream as input and performs object detection using a decision model to detect obstacles to the travel of the target moving object 120. The detection result includes prediction information of the obstacle's future behavior. The obstacle's future behavior is predicted based on the time series of the obstacle's position. The decision model may be changed as needed. Two or more decision models may be used. In this case, an obstacle is identified based on two or more detection results corresponding to the two or more decision models. In this case, a weighted average or other method is used.

[0152] The input destination of the above video stream is determined by the settings of the video stream AI processing pipeline 439. The video stream AI processing pipeline 439 is set for each surrounding situation monitoring device 130 and indicates the flow of the video stream. 19 to 21 show an example of a video stream AI processing pipeline 439. The "x" in "Vx" indicates the version number. 19, video stream A from roadside device A is sent to a general-purpose object detection model (V0), sent to annotation unit 434, where annotation information is added, and sent to learning unit 435 as part of learning data. Then, an object detection model (V1) is generated by updating the general-purpose object detection model (V0). Furthermore, detection information obtained by the general-purpose object detection model (V0) and position estimation is recorded in database 438. Thereafter, video stream A is also sent to object detection model (V1), sent to annotation unit 434, where annotation information is added. Then, the annotation information of video stream A is recorded in database 438. 20, video stream B from roadside device B is sent to an object detection model (V2) generated by updating a general-purpose object detection model (V0), and then sent to annotation unit 434, where annotation information is added. Then, the movement of an obstacle is estimated based on the annotation information of video stream B, and obstacle detection information including information on the estimated movement is recorded in database 438. In FIG. 21, a video stream D from a drone D is sent to a general-purpose road detection model (V0) and a road detection model (V1) generated by updating the general-purpose road detection model (V0).

[0153] ***Effects of the Sixth Embodiment*** According to the sixth embodiment, it becomes possible to more accurately determine whether there is a road obstruction using a video stream.

[0154] ***Supplement to Embodiment 6*** The autonomous driving control device 400 processes information from the video image sensors of multiple monitoring control devices 300. To this end, the autonomous driving control device 400 has a video switcher. The video switcher has the function of switching the flow of information from the video image sensors between image processing functions such as object detection, road detection, and lane detection, and annotation functions for learning.

[0155] The video switcher acquires moving images obtained from a plurality of monitor control devices 300 via real-time communication streams, and distributes each stream to the necessary processing functions. The autonomous driving control device 400 has the function of setting up a video switcher and an AI processing pipeline.

[0156] The video switcher settings include the following functions: a function to perform various detections and classifications using an AI model, a function to convert the detection results from the AI ​​model into video, a function to perform annotations for learning, a function to save video images, a function to adjust the image of the video images, etc. The video switcher settings are defined by describing a pipeline in the processing order of these functions.

[0157] The video switcher-based video stream AI processing pipeline system can be used for a variety of purposes, not just autonomous driving. For example, the video stream AI processing pipeline system can be used for the following purposes: (1) Detecting the number of vehicles entering and leaving the parking lot (2) Detecting the number of cars parked in a parking lot (3) Detection of people entering and leaving the facility (4) Detecting the number of pedestrians (up and down) on the road (5) Age and gender detection for passersby (6) Estimation of speed for pedestrians (7) Detection of stopping positions in relation to facilities (8) Detection and classification of facility work (9) Detection of human behavior

[0158] The sixth embodiment may be implemented in combination with at least one of the second to fifth embodiments.

[0159] ***Supplementary explanation of implementation form*** The autonomous driving system 100 includes a target moving object 120, a surrounding situation monitoring device 130, an autonomous driving control device 400, and a map database 110. The target moving object 120 is a moving object capable of autonomous driving. The target moving object 120 has sensors for detecting the surrounding environment, such as an optical sensor and a LIDAR, and a communication function. The surrounding situation monitoring device 130 has a sensor such as a camera that exists independently of the target moving object 120, and a communication function. The automatic driving control device 400 has a computer that can communicate with the target moving body 120 and the surrounding situation monitoring device 130. The map database 110 stores map data 111 that indicates surveillance areas and escape zones. The target moving body 120 estimates its own position using a computer, detects surrounding objects that may become obstacles using sensors, and estimates the position information of the obstacles through calculation. The surrounding situation monitoring device 130 uses sensors to detect the target moving body 120 and surrounding objects that may become obstacles, and estimates the position information of the obstacles by calculation. The surroundings monitoring device 130 supplements missing information about blind spots of the target moving object 120's sensors. The autonomous driving control device 400 generates information necessary to change the driving plan. The target moving object 120 determines the intersection of the predicted path of the object and the planned path of the target moving object 120, and recalculates and updates the driving plan, including the speed and path, so as to avoid hitting the object. The updated driving plan is converted into control information and used to control the driving of the target moving object 120.

[0160] The autonomous driving system 100 includes a target moving body 120, a drone 131, an autonomous driving control device 400, and a map database 110. The target moving object 120 is a moving object capable of autonomous driving, and has sensors for detecting the surrounding environment, such as optical sensors and LIDAR, and a communication function. The drone 131 is an aircraft with a flight function, and has a sensor such as a camera that exists independently of the target moving object 120, and a communication function. The autonomous driving control device 400 has a communication function and a calculation function. The autonomous driving control device 400 notifies the target moving object 120 of the control data required for creating a driving plan. The map database 110 has storage and communication functions. The map database 110 stores map data 111 used to determine the need to check the surrounding conditions. The map data 111 indicates a monitoring area and an escape zone. The map database 110 can communicate with the target moving object 120, the surrounding conditions monitoring device 130, and the autonomous driving control device 400. The drone 131 has a function of creating a flight plan for a round trip to a position where the blind spot can be seen when the approach of the target moving body 120 to an area with a large blind spot is detected, and a function of controlling the drone 131 based on the flight plan.

[0161] Each embodiment is an example of a preferred embodiment and is not intended to limit the technical scope of the present disclosure. Each embodiment may be implemented in part or in combination with other embodiments. Procedures described using flowcharts, etc. may be modified as appropriate.

[0162] Each element of the vehicle-mounted device 200, the monitoring and control device 300, and the automatic driving control device 400 may be realized by software, hardware, firmware, or a combination of these. The "part" of each element of the in-vehicle device 200, the monitoring control device 300, and the automatic driving control device 400 may be read as "processing," "step," "circuit," or "processing circuit."

[0163] Various aspects of the present disclosure are described below as appendices. (Appendix 1) An autonomous driving system including a target moving body that is a moving body that travels autonomously, a surrounding situation monitoring device, and an autonomous driving control device, The target moving body is a group of mobile sensors; an on-board device that determines whether monitoring by the surrounding situation monitoring device is necessary by referring to self-location data obtained by the mobile body sensor group and indicating the position of the target mobile body, and map data indicating a monitoring area set as an area where the surrounding situation of the target mobile body should be monitored by the surrounding situation monitoring device when the target mobile body passes, and when it determines that monitoring by the surrounding situation monitoring device is necessary, transmits a monitoring request to the surrounding situation monitoring device and transmits driving situation data obtained by the mobile body sensor group and indicating the driving situation of the target mobile body to the automatic driving control device; Equipped with The surrounding situation monitoring device is A group of monitoring sensors; a monitoring control device that receives the monitoring request, causes the monitoring sensors to monitor the surrounding conditions of the target moving object, and transmits monitoring data indicating the surrounding conditions of the target moving object to the autonomous driving control device; Equipped with The autonomous driving control device receives the driving situation data and the monitoring data, determines a driving obstacle for the target moving body based on the driving situation data and the monitoring data, and transmits control data notifying the target moving body of the driving obstacle, The vehicle-mounted device receives the control data and controls the travel of the target moving object based on the control data and surrounding situation data obtained by the moving object sensor group and indicating the surrounding situation of the target moving object. Autonomous driving system.

[0164] (Appendix 2) The surrounding situation monitoring device is an autonomous flying vehicle. 1. An automated driving system as described in Appendix 1.

[0165] (Appendix 3) The target moving body is provided with a drone port where the flying object takes off and lands. An automated driving system as described in Appendix 2.

[0166] (Appendix 4) the vehicle-mounted device transmits the monitoring request together with the self-location data and monitoring area data indicating the monitoring area; The monitoring control device receives the monitoring request, plans a flight path based on the self-location data and the monitoring area data from taking off from the drone port to flying over the monitoring area and landing at the drone port, and causes the aircraft to fly along the flight path.

[0167] (Appendix 5) The surrounding situation monitoring device is a roadside device installed in the monitoring area. 1. An automated driving system as described in Appendix 1.

[0168] (Appendix 6) the monitoring sensor group includes a color sensor; The monitoring and control device includes: Detecting objects present around the target moving object and a plane where the objects are located based on color sensor data obtained by the color sensor; Estimating the position of the object in the sensed plane 10. The automated driving system of claim 1.

[0169] (Appendix 7) The color sensor is an RGB camera or an RGB-D camera. 1. An automated driving system as described in Appendix 6.

[0170] (Appendix 8) The autonomous driving control device determines that the driving obstacle exists when an obstacle facing the target moving object exists within a proximity range from the position of the target moving object. 10. The automated driving system of claim 1.

[0171] (Appendix 9) When the obstacle is a moving object, the autonomous driving control device transmits the control data indicating that passing is not permitted as the type of the driving obstacle to the target moving body, When the type of the driving obstacle is the impossibility of passing each other, the vehicle-mounted device causes the target moving object to retreat to an escape zone indicated in the map data. 1. An automated driving system as described in Appendix 8.

[0172] (Appendix 10) When the obstacle is a stationary object, the autonomous driving control device transmits the control data indicating impassability as the type of the driving obstacle to the target moving body, When the type of the driving obstacle is the impassable road, the vehicle-mounted device makes the target moving object make a U-turn by using an escape zone shown in the map data. 1. An automated driving system as set forth in Appendix 8 or Appendix 9.

[0173] (Appendix 11) When a plurality of obstacles are present, the autonomous driving control device transmits the control data indicating a need to slow down as the type of the driving obstacle to the target moving body, The vehicle-mounted device causes the target moving object to slow down when the type of the driving obstacle requires the target moving object to slow down. 11. The automated driving system of any one of Supplementary Notes 8 to 10.

[0174] (Appendix 12) The autonomous driving control device receives self-position data from each of the one or more mobile bodies, determines whether the obstacle is one of the one or more mobile bodies based on the self-position data of each of the one or more mobile bodies, and notifies the mobile body that is the obstacle of the driving obstacle when it determines that the obstacle is one of the one or more mobile bodies. 12. The automated driving system of any one of Supplementary Note 8 to Supplementary Note 11.

[0175] (Appendix 13) The autonomous driving control device receives self-location data from each of one or more mobile devices carried by one or more pedestrians, determines whether the obstacle is one of the one or more pedestrians based on the self-location data of each of the one or more mobile devices, and if it determines that the obstacle is one of the one or more pedestrians, notifies the mobile device of the pedestrian who is the obstacle of the driving obstacle. 13. The automated driving system of any one of Supplementary Notes 8 to 12.

[0176] (Appendix 14) The autonomous driving system includes a plurality of the surrounding situation monitoring devices, the group of monitoring sensors of each of the surrounding situation monitoring devices includes a moving image sensor; The monitoring control device of each of the surrounding situation monitoring devices transmits the monitoring data, including the video stream obtained by the video image sensor, to the autonomous driving control device; The automatic driving control device For each of the video streams, an object detection model corresponding to the surrounding situation monitoring device that is a transmission source is used to input the video stream and perform object detection, and detection information of the detected object is recorded in a database; using a selection model, which is an object detection model selected from the plurality of types of object detection models, to input the video stream and correct detection information of the detected object using annotations, learning the video stream and the corrected detection information to update the selection model and generate a new version of the object detection model; The video stream is input and the object detection is performed using a determined model, which is an object detection model determined from the plurality of types of object detection models for each version, to detect an obstacle to the running of the target moving object. 14. The automated driving system of any one of Supplementary Notes 1 to 13.

[0177] (Appendix 15) An autonomous driving method using an autonomous driving system including a target moving body that is a moving body traveling by autonomous driving, a surrounding situation monitoring device, and an autonomous driving control device, The target moving body is determining whether or not monitoring by the surrounding situation monitoring device is necessary by referring to self-location data indicating the position of the target moving body and map data indicating a monitoring area set as an area where the surrounding situation of the target moving body should be monitored by the surrounding situation monitoring device when the target moving body passes, and when determining that monitoring by the surrounding situation monitoring device is necessary, transmitting a monitoring request to the surrounding situation monitoring device and transmitting driving situation data indicating the driving situation of the target moving body to the automatic driving control device; The surrounding situation monitoring device is receiving the monitoring request, monitoring the surrounding situation of the target moving object, and transmitting monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device; The automatic driving control device receiving the driving condition data and the monitoring data, determining a driving obstacle for the target moving body based on the driving condition data and the monitoring data, and transmitting control data notifying the target moving body of the driving obstacle; The target moving body is The control data is received, and the target moving body travels in an automatic driving mode based on the control data and surrounding situation data indicating the surrounding situation of the target moving body. Autonomous driving method.

[0178] (Appendix 16) It is a vehicle that runs autonomously, a group of mobile sensors; an on-board unit that determines whether monitoring by the surrounding situation monitoring device is necessary by referring to self-location data obtained by the mobile body sensor group and indicating the position of the mobile body, and map data indicating a monitoring area set as an area where the surrounding situation of the mobile body should be monitored by a surrounding situation monitoring device when the mobile body passes, and when it determines that monitoring by the surrounding situation monitoring device is necessary, sends a monitoring request to the surrounding situation monitoring device and sends driving situation data obtained by the mobile body sensor group and indicating the driving situation of the mobile body to an automatic driving control device, receives control data from the automatic driving control device that notifies of a driving obstacle for the mobile body determined based on the driving situation data and the monitoring data obtained by the surrounding situation monitoring device and indicating the surrounding situation of the mobile body, and controls the driving of the mobile body based on the control data and the surrounding situation data obtained by the mobile body sensor group and indicating the surrounding situation of the mobile body; A mobile body comprising:

[0179] (Appendix 17) the surrounding situation monitoring device is an autonomous flying vehicle, The mobile body includes a drone port where the flying object takes off and lands. 17. The mobile object according to claim 16.

[0180] (Appendix 18) A group of monitoring sensors; a monitoring control device that receives a monitoring request from a target moving body when the target moving body travels autonomously through a monitoring area, causes the monitoring sensor group to monitor the surrounding conditions of the target moving body to obtain monitoring data indicating the surrounding conditions of the target moving body, and transmits the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle for the target moving body determined based on the monitoring data and driving condition data indicating the driving condition of the target moving body; An ambient situation monitoring device comprising:

[0181] (Appendix 19) the target moving object includes a drone port; The surrounding situation monitoring device is an autonomous flying vehicle that takes off and lands using the drone port. 19. The ambient conditions monitoring device of claim 18.

[0182] (Appendix 20) The monitoring control device receives the monitoring request including self-location data indicating the position of the target moving object and monitoring area data indicating the monitoring area, and plans a flight path from taking off from the drone port to flying over the monitoring area and landing at the drone port based on the self-location data and the monitoring area data, and causes the flying object to fly along the flight path. 20. The ambient conditions monitoring device of claim 19.

[0183] (Appendix 21) a roadside unit installed in the monitoring area; 19. The ambient conditions monitoring device of claim 18.

[0184] (Appendix 22) the monitoring sensor group includes a color sensor; The monitoring and control device includes: Detecting objects present around the target moving object and a plane where the objects are located based on color sensor data obtained by the color sensor; Estimating the position of the object in the sensed plane 22. The surroundings monitoring device according to any one of Supplementary Note 18 to Supplementary Note 21.

[0185] (Appendix 23) The color sensor is an RGB camera or an RGB-D camera. 23. The ambient conditions monitoring device of claim 22.

[0186] (Appendix 24) receiving, from a target moving object traveling by autonomous driving, driving status data indicating a driving status of the target moving object; receiving monitoring data indicating the surrounding conditions of the target moving object from a surrounding conditions monitoring device that receives a monitoring request from the target moving object when the target moving object passes through a monitoring area and monitors the surrounding conditions of the target moving object; determining a travel obstacle for the target moving object based on the travel condition data and the monitoring data; Transmitting control data notifying the travel obstacle to the target moving body. Autonomous driving control device.

[0187] (Appendix 25) When an obstacle facing the target moving object is present within a proximity range from the position of the target moving object, it is determined that the target moving object has a travel obstacle. 2. An automated driving control device as set forth in Appendix 24.

[0188] (Appendix 26) When the obstacle is a moving object, the control data indicating that passing is not permitted as the type of the travel obstacle is transmitted to the target moving object, thereby causing the target moving object to retreat to an escape zone. 2. An automated driving control device as described in Appendix 25.

[0189] (Appendix 27) If the obstacle is a stationary object, the control data indicating impassability as the type of the travel obstacle is transmitted to the target moving object, thereby making the target moving object turn around using an escape zone. An automated driving control device as set forth in Appendix 25 or Appendix 26.

[0190] (Appendix 28) When a plurality of obstacles are present, the control data indicating the need for slowing down as the type of obstacle to travel is transmitted to the target moving object, thereby causing the target moving object to slow down. 28. An automatic driving control device according to any one of appendices 25 to 27.

[0191] (Appendix 29) receiving self-location data from each of one or more mobile bodies, determining whether the obstacle is one of the one or more mobile bodies based on the self-location data of each of the one or more mobile bodies, and notifying the mobile body that is the obstacle of the driving obstacle when it is determined that the obstacle is one of the one or more mobile bodies; 29. An automatic driving control device according to any one of appendices 25 to 28.

[0192] (Appendix 30) receiving self-location data from each of one or more portable devices carried by one or more pedestrians, determining whether the obstacle is one of the one or more pedestrians based on the self-location data of each of the one or more portable devices, and notifying the portable device of the pedestrian who is the obstacle of the driving obstacle when determining that the obstacle is one of the one or more pedestrians; 29. An automatic driving control device according to any one of appendices 25 to 29.

[0193] (Appendix 31) a process of determining whether or not monitoring by the surrounding situation monitoring device is necessary by referring to self-location data indicating the location of the mobile body traveling in an autonomous driving mode and map data indicating a monitoring area set as an area where the surrounding situation of the mobile body should be monitored by the surrounding situation monitoring device when the mobile body passes; a process of transmitting a monitoring request to the surrounding situation monitoring device and transmitting driving situation data indicating the driving situation of the moving body to an automatic driving control device when it is determined that monitoring by the surrounding situation monitoring device is necessary; A process of receiving control data from the autonomous driving control device that notifies of a driving obstacle of the moving body determined based on the driving situation data and monitoring data indicating the surrounding situation of the moving body obtained by the surrounding situation monitoring device; a process of controlling the travel of the moving object based on the control data and surrounding situation data indicating the surrounding situation of the moving object; An autonomous driving program that allows a computer to execute the above.

[0194] (Appendix 32) A process of receiving a monitoring request from a target moving object traveling in an autonomous driving manner when the target moving object passes through a monitoring area; a process of causing a group of monitoring sensors to monitor the surroundings of the target moving object and obtaining monitoring data indicating the surroundings of the target moving object; A process of transmitting the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle of the target moving body determined based on the monitoring data and driving situation data indicating the driving situation of the target moving body; A monitoring program that causes a computer to execute the following.

[0195] (Appendix 33) A process of receiving driving status data indicating a driving status of a target moving body from the target moving body that is driving autonomously; a process of receiving monitoring data indicating the surrounding situation of the target moving object from a surrounding situation monitoring device that receives a monitoring request from the target moving object when the target moving object passes through a monitoring area and monitors the surrounding situation of the target moving object; A process of determining a driving obstacle of the target moving object based on the driving condition data and the monitoring data; a process of transmitting control data notifying the target moving body of the road obstruction to the target moving body; A control program that causes a computer to execute the above.

[0196] (Appendix 34) An autonomous driving system including a target moving body that is a moving body that travels on the ground by autonomous driving, an air vehicle that flies in the sky by autonomous driving, and an autonomous driving control device, The target moving body is a group of mobile sensors; an in-vehicle device that transmits driving situation data obtained by the group of moving body sensors and indicating the driving situation of the target moving body to the automatic driving control device; Equipped with The flying vehicle is A group of monitoring sensors; a monitoring control device that causes the monitoring sensors to monitor the surrounding conditions of the target moving object from the sky and transmits monitoring data indicating the surrounding conditions of the target moving object to the autonomous driving control device; Equipped with the autonomous driving control device receives the driving situation data and the monitoring data, determines a driving obstacle for the target moving body based on the driving situation data and the monitoring data, and transmits control data notifying the target moving body of the driving obstacle; The vehicle-mounted device receives the control data and controls the travel of the target moving object based on the control data and surrounding situation data obtained by the moving object sensor group and indicating the surrounding situation of the target moving object. Autonomous driving system.

[0197] (Appendix 35) The target moving body is provided with a drone port where the flying object takes off and lands. 34. The automated driving system described in Appendix 34.

[0198] (Appendix 36) the vehicle-mounted device transmits a monitoring request including monitoring area data indicating a monitoring area and self-location data obtained by the mobile body sensor group and indicating a position of the target mobile body; The monitoring control device receives the monitoring request, plans a flight path based on the self-location data and the monitored area data from taking off from the drone port to flying over the monitored area and landing at the drone port, and flies the aircraft along the flight path.

[0199] (Appendix 37) An autonomous driving method using an autonomous driving system including a target moving body that is a moving body that travels on the ground by autonomous driving, an air vehicle that flies in the sky by autonomous driving, and an autonomous driving control device, The target moving body is Transmitting driving situation data indicating the driving situation of the target moving body to the automatic driving control device; The flying vehicle is monitoring the surrounding conditions of the target moving object from the sky, and transmitting monitoring data indicating the surrounding conditions of the target moving object to the autonomous driving control device; The automatic driving control device receiving the driving condition data and the monitoring data, determining a driving obstacle for the target moving body based on the driving condition data and the monitoring data, and transmitting control data notifying the target moving body of the driving obstacle; The target moving body is The control data is received, and the target moving body travels in an automatic driving mode based on the control data and surrounding situation data indicating the surrounding situation of the target moving body. Autonomous driving method.

[0200] (Appendix 38) It is a vehicle that travels on the ground autonomously, a group of mobile sensors; an on-board device that transmits driving situation data obtained by the group of mobile body sensors and indicating the driving situation of the mobile body to an autonomous driving control device, receives control data from the autonomous driving control device that notifies of a driving obstacle to the mobile body, the control data being determined based on the driving situation data and monitoring data obtained by an aircraft flying overhead in an autonomous driving manner and indicating the situation around the mobile body, and controls the driving of the mobile body based on the control data and surrounding situation data obtained by the group of mobile body sensors and indicating the situation around the mobile body; A mobile body comprising:

[0201] (Appendix 39) A drone port is provided for the aircraft to take off and land on. 39. The mobile object according to claim 38.

[0202] (Appendix 40) It is an autonomous flying vehicle that flies through the sky. A group of monitoring sensors; a monitoring control device that causes the monitoring sensors to monitor from the sky the surrounding conditions of a target moving object traveling autonomously on the ground, obtains monitoring data indicating the surrounding conditions of the target moving object, and transmits the monitoring data to an autonomous driving control device that notifies the target moving object of a traveling obstacle to the target moving object determined based on the monitoring data and traveling situation data indicating the traveling situation of the target moving object; An aircraft equipped with:

[0203] (Appendix 41) the target moving object includes a drone port; The aircraft takes off and lands using the drone port. 41. The air vehicle described in Appendix 40.

[0204] (Appendix 42) The monitoring control device receives a monitoring request including monitoring area data indicating a monitoring area and self-location data indicating the position of the target moving object, and plans a flight path from taking off from the drone port to flying over the monitoring area and landing at the drone port based on the self-location data and the monitoring area data, and causes the flying object to fly along the flight path. 41. The air vehicle described in Appendix 41.

[0205] (Appendix 43) receiving, from a target moving object traveling on the ground by autonomous driving, driving status data indicating a driving status of the target moving object; receiving monitoring data indicating a surrounding situation of the target moving object from a surrounding situation monitoring device that flies above the target moving object in an automatic driving manner and monitors a surrounding situation of the target moving object; determining a travel obstacle for the target moving object based on the travel condition data and the monitoring data; Transmitting control data notifying the travel obstacle to the target moving body. Autonomous driving control device.

[0206] (Appendix 44) A process of transmitting driving status data indicating the driving status of a mobile object traveling on the ground by autonomous driving to an autonomous driving control device; A process of receiving control data from the autonomous driving control device that notifies of a driving obstacle to the moving body, the control data being determined based on the driving situation data and monitoring data indicating the surrounding situation of the moving body obtained by an aircraft flying overhead in an autonomous driving manner; a process of controlling the travel of the moving object based on the control data and surrounding situation data indicating the surrounding situation of the moving object; An autonomous driving program that allows a computer to execute the above.

[0207] (Appendix 45) A process of having a group of monitoring sensors mounted on an autonomous flying vehicle fly in the sky monitor the surroundings of a target moving object traveling autonomously on the ground, thereby obtaining monitoring data indicating the surroundings of the target moving object; A process of transmitting the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle of the target moving body determined based on the monitoring data and driving situation data indicating the driving situation of the target moving body; A monitoring program that causes a computer to execute the following.

[0208] (Appendix 46) A process of receiving driving status data indicating a driving status of a target moving object from the target moving object that is driving on the ground by autonomous driving; receiving monitoring data indicating the surrounding situation of the target moving object from a surrounding situation monitoring device that flies above the target moving object in an autonomous driving manner and monitors the surrounding situation of the target moving object; A process of determining a driving obstacle of the target moving object based on the driving condition data and the monitoring data; a process of transmitting control data notifying the target moving body of the road obstruction to the target moving body; A control program that causes a computer to execute the above. [Explanation of symbols]

[0209] 100 Autonomous driving system, 101 Mobile body, 102 Mobile terminal, 110 Map database, 111 Map data, 120 Target mobile body, 121 Drone port, 130 Surrounding situation monitoring device, 131 Drone, 200 On-board device, 201 Processor, 202 Memory, 203 Auxiliary storage device, 204 Communication device, 205 Input / output interface, 211 Monitoring request unit, 212 Control data acquisition unit, 213 Driving control unit, 220 Memory unit, 290 Mobile body sensor group, 291 Depth camera, 292 Lidar, 293 Positioning sensor, 300 Monitoring control device, 301 Processor, 302 Memory, 303 Auxiliary storage device, 304 Communication device, 305 Input / output interface, 311 Request reception unit, 312 Monitoring control unit, 313 Monitoring data generation unit, 314 Monitoring data provision unit, 320 Memory unit, 390 monitoring sensor group, 391 depth camera, 392 RGB camera, 393 color sensor, 400 autonomous driving control device, 401 processor, 402 memory, 403 auxiliary storage device, 404 communication device, 405 input / output interface, 411 data reception unit, 412 driving obstacle determination unit, 413 control data generation unit, 414 control data provision unit, 415 driving obstacle notification unit, 420 memory unit, 431 switcher unit, 432 object detection unit, 433 position estimation unit, 434 annotation unit, 435 learning unit, 438 database, 439 video stream AI processing pipeline.

Claims

1. An autonomous driving system including a target moving body that is a moving body that travels autonomously, a surrounding situation monitoring device, and an autonomous driving control device, The target moving body is a group of mobile sensors; an on-board device that determines whether monitoring by the surrounding situation monitoring device is necessary by referring to self-location data obtained by the mobile body sensor group and indicating the position of the target mobile body, and map data indicating a monitoring area set as an area where the surrounding situation of the target mobile body should be monitored by the surrounding situation monitoring device when the target mobile body passes, and when it determines that monitoring by the surrounding situation monitoring device is necessary, transmits a monitoring request to the surrounding situation monitoring device and transmits driving situation data obtained by the mobile body sensor group and indicating the driving situation of the target mobile body to the automatic driving control device; Equipped with The surrounding situation monitoring device A group of monitoring sensors; a monitoring control device that receives the monitoring request, causes the monitoring sensors to monitor the surrounding conditions of the target moving object, and transmits monitoring data indicating the surrounding conditions of the target moving object to the autonomous driving control device; Equipped with the autonomous driving control device receives the driving situation data and the monitoring data, determines a driving obstacle for the target moving body based on the driving situation data and the monitoring data, and transmits control data notifying the target moving body of the driving obstacle; The vehicle-mounted device receives the control data and controls the travel of the target moving object based on the control data and surrounding situation data obtained by the moving object sensor group and indicating the surrounding situation of the target moving object. Autonomous driving system.

2. The surrounding situation monitoring device is an autonomous flying vehicle. The automated driving system according to claim 1 .

3. The target moving body is provided with a drone port where the flying object takes off and lands. The automated driving system according to claim 2.

4. the vehicle-mounted device transmits the monitoring request together with the self-location data and monitoring area data indicating the monitoring area; The autonomous driving system described in claim 3, wherein the monitoring control device receives the monitoring request, plans a flight path from taking off from the drone port based on the self-location data and the monitoring area data, flying over the monitoring area, and landing at the drone port, and causes the aircraft to fly along the flight path.

5. The surrounding situation monitoring device is a roadside device installed in the monitoring area. The automated driving system according to claim 1 .

6. the monitoring sensor group includes a color sensor; The monitoring and control device includes: Detecting objects present around the target moving object and a plane where the objects are located based on color sensor data obtained by the color sensor; Estimating the position of the object in the sensed plane The automated driving system according to any one of claims 1 to 5.

7. The color sensor is an RGB camera or an RGB-D camera. The automated driving system according to claim 6.

8. The autonomous driving control device determines that the driving obstacle exists when an obstacle facing the target moving object exists within a proximity range from the position of the target moving object. The automated driving system according to claim 1 .

9. When the obstacle is a moving object, the autonomous driving control device transmits the control data indicating that passing is not permitted as the type of the driving obstacle to the target moving body, When the type of the driving obstacle is the impossibility of passing each other, the vehicle-mounted device causes the target moving object to retreat to an escape zone indicated in the map data. The automated driving system according to claim 8.

10. When the obstacle is a stationary object, the autonomous driving control device transmits the control data indicating impassability as the type of the driving obstacle to the target moving body, When the type of the driving obstacle is the impassable road, the vehicle-mounted device makes the target moving object turn around by utilizing an escape zone indicated in the map data. The automated driving system according to claim 8 or 9.

11. When a plurality of obstacles are present, the autonomous driving control device transmits the control data indicating a need to slow down as the type of the driving obstacle to the target moving body, The vehicle-mounted device causes the target moving object to slow down when the type of the driving obstacle requires the target moving object to slow down. The automated driving system according to claim 8 or 9.

12. The autonomous driving control device receives self-position data from each of the one or more mobile bodies, determines whether the obstacle is one of the one or more mobile bodies based on the self-position data of each of the one or more mobile bodies, and notifies the mobile body that is the obstacle of the driving obstacle when it determines that the obstacle is one of the one or more mobile bodies. The automated driving system according to claim 8 or 9.

13. The autonomous driving control device receives self-location data from each of one or more mobile devices carried by one or more pedestrians, determines whether the obstacle is one of the one or more pedestrians based on the self-location data of each of the one or more mobile devices, and if it determines that the obstacle is one of the one or more pedestrians, notifies the mobile device of the pedestrian who is the obstacle of the driving obstacle. The automated driving system according to claim 8 or 9.

14. The autonomous driving system includes a plurality of the surrounding situation monitoring devices, the group of monitoring sensors of each of the surrounding situation monitoring devices includes a moving image sensor; The monitoring control device of each of the surrounding situation monitoring devices transmits the monitoring data, including the video stream obtained by the video image sensor, to the autonomous driving control device; The automatic driving control device For each of the video streams, an object detection model corresponding to the surrounding situation monitoring device that is a transmission source is used to input the video stream and perform object detection, and detection information of the detected object is recorded in a database; using a selection model, which is an object detection model selected from the plurality of types of object detection models, to input the video stream and correct detection information of the detected object using annotations, learning the video stream and the corrected detection information to update the selection model and generate a new version of the object detection model; The video stream is input and the object detection is performed using a determined model, which is an object detection model determined from the plurality of types of object detection models for each version, to detect an obstacle to the running of the target moving object. The automated driving system according to claim 1 .

15. An autonomous driving method using an autonomous driving system including a target moving body that is a moving body traveling by autonomous driving, a surrounding situation monitoring device, and an autonomous driving control device, The target moving body is determining whether or not monitoring by the surrounding situation monitoring device is necessary by referring to self-location data indicating the position of the target moving body and map data indicating a monitoring area set as an area where the surrounding situation of the target moving body should be monitored by the surrounding situation monitoring device when the target moving body passes, and when determining that monitoring by the surrounding situation monitoring device is necessary, transmitting a monitoring request to the surrounding situation monitoring device and transmitting driving situation data indicating the driving situation of the target moving body to the automatic driving control device; The surrounding situation monitoring device receiving the monitoring request, monitoring the surrounding situation of the target moving object, and transmitting monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device; The automatic driving control device receiving the driving condition data and the monitoring data, determining a driving obstacle for the target moving body based on the driving condition data and the monitoring data, and transmitting control data notifying the target moving body of the driving obstacle; The target moving body is The control data is received, and the target moving body travels in an automatic driving mode based on the control data and surrounding situation data indicating the surrounding situation of the target moving body. Autonomous driving method.

16. It is a vehicle that runs autonomously, a group of mobile sensors; an on-board unit that determines whether monitoring by the surrounding situation monitoring device is necessary by referring to self-location data obtained by the mobile body sensor group and indicating the position of the mobile body, and map data indicating a monitoring area set as an area where the surrounding situation of the mobile body should be monitored by a surrounding situation monitoring device when the mobile body passes, and when it determines that monitoring by the surrounding situation monitoring device is necessary, sends a monitoring request to the surrounding situation monitoring device and sends driving situation data obtained by the mobile body sensor group and indicating the driving situation of the mobile body to an automatic driving control device, receives control data from the automatic driving control device that notifies of a driving obstacle for the mobile body determined based on the driving situation data and the monitoring data obtained by the surrounding situation monitoring device and indicating the surrounding situation of the mobile body, and controls the driving of the mobile body based on the control data and the surrounding situation data obtained by the mobile body sensor group and indicating the surrounding situation of the mobile body; A mobile body comprising:

17. the surrounding situation monitoring device is an autonomous flying vehicle, The mobile body includes a drone port where the flying object takes off and lands. The moving body according to claim 16.

18. A group of monitoring sensors; a monitoring control device that receives a monitoring request from a target moving body when the target moving body travels autonomously through a monitoring area, causes the monitoring sensor group to monitor the surrounding conditions of the target moving body to obtain monitoring data indicating the surrounding conditions of the target moving body, and transmits the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle for the target moving body determined based on the monitoring data and driving condition data indicating the driving condition of the target moving body; An ambient situation monitoring device comprising:

19. the target moving object includes a drone port; The surrounding situation monitoring device is an autonomous flying vehicle that takes off and lands using the drone port. The surroundings monitoring device according to claim 18.

20. The monitoring control device receives the monitoring request including self-location data indicating the position of the target moving object and monitoring area data indicating the monitoring area, and plans a flight path from taking off from the drone port to flying over the monitoring area and landing at the drone port based on the self-location data and the monitoring area data, and causes the flying object to fly along the flight path.

20. The surroundings monitoring device according to claim 19.

21. a roadside unit installed in the monitoring area; The surroundings monitoring device according to claim 18.

22. the monitoring sensor group includes a color sensor; The monitoring and control device includes: Detecting objects present around the target moving object and a plane where the objects are located based on color sensor data obtained by the color sensor; Estimating the position of the object in the sensed plane The surrounding situation monitoring device according to any one of claims 18 to 21.

23. The color sensor is an RGB camera or an RGB-D camera.

23. The ambient conditions monitoring device according to claim 22.

24. receiving, from a target moving object traveling by autonomous driving, driving status data indicating a driving status of the target moving object; receiving monitoring data indicating the surrounding conditions of the target moving object from a surrounding conditions monitoring device that receives a monitoring request from the target moving object when the target moving object passes through a monitoring area and monitors the surrounding conditions of the target moving object; determining a travel obstacle for the target moving object based on the travel condition data and the monitoring data; Transmitting control data notifying the travel obstacle to the target moving body. Autonomous driving control device.

25. When an obstacle facing the target moving object is present within a proximity range from the position of the target moving object, it is determined that the target moving object has a travel obstacle. The automatic driving control device according to claim 24.

26. When the obstacle is a moving object, the control data indicating that passing is not permitted as the type of the travel obstacle is transmitted to the target moving object, thereby causing the target moving object to retreat to an escape zone. The automatic driving control device according to claim 25.

27. If the obstacle is a stationary object, the control data indicating that the obstacle is impassable is transmitted to the target moving object, thereby making the target moving object turn around by utilizing an escape zone. The automatic driving control device according to claim 25 or 26.

28. When a plurality of obstacles are present, the control data indicating the need for slowing down as the type of obstacle to travel is transmitted to the target moving object, thereby causing the target moving object to slow down. The automatic driving control device according to claim 25 or 26.

29. receiving self-location data from each of one or more mobile bodies, determining whether the obstacle is one of the one or more mobile bodies based on the self-location data of each of the one or more mobile bodies, and notifying the mobile body that is the obstacle of the driving obstacle when it is determined that the obstacle is one of the one or more mobile bodies; The automatic driving control device according to claim 25 or 26.

30. receiving self-location data from each of one or more portable devices carried by one or more pedestrians, determining whether the obstacle is one of the one or more pedestrians based on the self-location data of each of the one or more portable devices, and notifying the portable device of the pedestrian who is the obstacle of the driving obstacle when determining that the obstacle is one of the one or more pedestrians; The automatic driving control device according to claim 25 or 26.

31. a process of determining whether or not monitoring by the surrounding situation monitoring device is necessary by referring to self-location data indicating the position of the mobile body traveling by automatic driving and map data indicating a monitoring area set as an area where the surrounding situation of the mobile body should be monitored by the surrounding situation monitoring device when the mobile body passes; a process of transmitting a monitoring request to the surrounding situation monitoring device and transmitting driving situation data indicating the driving situation of the moving body to an automatic driving control device when it is determined that monitoring by the surrounding situation monitoring device is necessary; A process of receiving control data from the autonomous driving control device that notifies of a driving obstacle of the moving body determined based on the driving situation data and monitoring data indicating the surrounding situation of the moving body obtained by the surrounding situation monitoring device; a process of controlling the travel of the moving object based on the control data and surrounding situation data indicating the surrounding situation of the moving object; An autonomous driving program that allows a computer to execute the above.

32. A process of receiving a monitoring request from a target moving object traveling in an autonomous driving manner when the target moving object passes through a monitoring area; a process of causing a group of monitoring sensors to monitor the surroundings of the target moving object and obtaining monitoring data indicating the surroundings of the target moving object; A process of transmitting the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle of the target moving body determined based on the monitoring data and driving situation data indicating the driving situation of the target moving body; A monitoring program that causes a computer to execute the following.

33. A process of receiving driving status data indicating a driving status of a target moving body from the target moving body that is driving autonomously; a process of receiving monitoring data indicating the surrounding situation of the target moving object from a surrounding situation monitoring device that receives a monitoring request from the target moving object when the target moving object passes through a monitoring area and monitors the surrounding situation of the target moving object; A process of determining a driving obstacle of the target moving object based on the driving condition data and the monitoring data; a process of transmitting control data notifying the target moving body of the road obstruction to the target moving body; A control program that causes a computer to execute the above.

34. An autonomous driving system including a target moving body that is a moving body that travels on the ground by autonomous driving, an air vehicle that flies in the sky by autonomous driving, and an autonomous driving control device, The target moving body is a group of mobile sensors; an in-vehicle device that transmits driving situation data obtained by the group of moving body sensors and indicating the driving situation of the target moving body to the automatic driving control device; Equipped with The flying vehicle is A group of monitoring sensors; a monitoring control device that causes the monitoring sensors to monitor the surrounding conditions of the target moving object from the sky and transmits monitoring data indicating the surrounding conditions of the target moving object to the autonomous driving control device; Equipped with the autonomous driving control device receives the driving situation data and the monitoring data, determines a driving obstacle for the target moving body based on the driving situation data and the monitoring data, and transmits control data notifying the target moving body of the driving obstacle; The vehicle-mounted device receives the control data and controls the travel of the target moving object based on the control data and surrounding situation data obtained by the moving object sensor group and indicating the surrounding situation of the target moving object. Autonomous driving system.

35. The target moving body is provided with a drone port where the flying object takes off and lands.

35. The automated driving system of claim 34.

36. the vehicle-mounted device transmits a monitoring request including monitoring area data indicating a monitoring area and self-location data obtained by the mobile body sensor group and indicating a position of the target mobile body; The autonomous driving system described in claim 35, wherein the monitoring control device receives the monitoring request, plans a flight path based on the self-location data and the monitoring area data from taking off from the drone port to flying over the monitoring area and landing at the drone port, and causes the aircraft to fly along the flight path.

37. An autonomous driving method using an autonomous driving system including a target moving body that is a moving body that travels on the ground by autonomous driving, an air vehicle that flies in the sky by autonomous driving, and an autonomous driving control device, The target moving body is Transmitting driving situation data indicating the driving situation of the target moving body to the automatic driving control device; The flying vehicle is monitoring the surrounding conditions of the target moving object from the sky, and transmitting monitoring data indicating the surrounding conditions of the target moving object to the autonomous driving control device; The automatic driving control device receiving the driving condition data and the monitoring data, determining a driving obstacle for the target moving body based on the driving condition data and the monitoring data, and transmitting control data notifying the target moving body of the driving obstacle; The target moving body is The control data is received, and the target moving body travels in an automatic driving mode based on the control data and surrounding situation data indicating the surrounding situation of the target moving body. Autonomous driving method.

38. It is a vehicle that travels on the ground autonomously, a group of mobile sensors; an on-board device that transmits driving situation data obtained by the group of mobile body sensors and indicating the driving situation of the mobile body to an autonomous driving control device, receives control data from the autonomous driving control device that notifies of a driving obstacle to the mobile body, the control data being determined based on the driving situation data and monitoring data obtained by an aircraft flying overhead in an autonomous driving manner and indicating the situation around the mobile body, and controls the driving of the mobile body based on the control data and surrounding situation data obtained by the group of mobile body sensors and indicating the situation around the mobile body; A mobile body comprising:

39. A drone port is provided for the aircraft to take off and land on.

39. The mobile body according to claim 38.

40. It is an autonomous flying vehicle that flies through the sky. A group of monitoring sensors; a monitoring control device that causes the monitoring sensors to monitor from the sky the surrounding conditions of a target moving object traveling autonomously on the ground, obtains monitoring data indicating the surrounding conditions of the target moving object, and transmits the monitoring data to an autonomous driving control device that notifies the target moving object of a traveling obstacle to the target moving object determined based on the monitoring data and traveling situation data indicating the traveling situation of the target moving object; An aircraft equipped with:

41. the target moving object includes a drone port; The aircraft takes off and lands using the drone port. The air vehicle of claim 40.

42. The monitoring control device receives a monitoring request including monitoring area data indicating a monitoring area and self-location data indicating the position of the target moving object, and plans a flight path from taking off from the drone port to flying over the monitoring area and landing at the drone port based on the self-location data and the monitoring area data, and causes the flying object to fly along the flight path.

42. The air vehicle of claim 41.

43. receiving, from a target moving object traveling on the ground by autonomous driving, driving status data indicating a driving status of the target moving object; receiving monitoring data indicating a surrounding situation of the target moving object from a surrounding situation monitoring device that flies above the target moving object in an automatic driving manner and monitors a surrounding situation of the target moving object; determining a travel obstacle for the target moving object based on the travel condition data and the monitoring data; Transmitting control data notifying the travel obstacle to the target moving body. Autonomous driving control device.

44. A process of transmitting driving status data indicating the driving status of a mobile object traveling on the ground by autonomous driving to an autonomous driving control device; A process of receiving control data from the autonomous driving control device that notifies of a driving obstacle to the moving body, the control data being determined based on the driving situation data and monitoring data indicating the surrounding situation of the moving body obtained by an aircraft flying overhead in an autonomous driving manner; a process of controlling the travel of the moving object based on the control data and surrounding situation data indicating the surrounding situation of the moving object; An autonomous driving program that allows a computer to execute the above.

45. A process of having a group of monitoring sensors mounted on an autonomous flying vehicle fly in the sky monitor the surroundings of a target moving object traveling autonomously on the ground, thereby obtaining monitoring data indicating the surroundings of the target moving object; A process of transmitting the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle of the target moving body determined based on the monitoring data and driving situation data indicating the driving situation of the target moving body; A monitoring program that causes a computer to execute the following.

46. A process of receiving driving status data indicating a driving status of a target moving object from the target moving object that is driving on the ground by autonomous driving; receiving monitoring data indicating the surrounding situation of the target moving object from a surrounding situation monitoring device that flies above the target moving object in an autonomous driving manner and monitors the surrounding situation of the target moving object; A process of determining a driving obstacle of the target moving object based on the driving condition data and the monitoring data; a process of transmitting control data notifying the target moving body of the road obstruction to the target moving body; A control program that causes a computer to execute the above.

Images