AUTONOMOUS DRIVING SYSTEM, AUTONOMOUS DRIVING METHOD, MOBILE BODY, AND AUTONOMOUS DRIVING PROGRAM

Abstract

Making safer autonomous driving possible. [Solution] A target moving body 120 refers to its own position data and map data to determine whether monitoring is required, and transmits a monitoring request along with driving condition data. A surrounding condition monitoring device 130 receives the monitoring request, monitors the surrounding conditions of the target moving body, and transmits monitoring data. An automatic driving control device 400 receives the driving condition data and the monitoring data, determines a driving obstacle for the target moving body based on the driving condition data and the monitoring data, and transmits control data to the target moving body.

Description

[Technical field]

[0001] The present disclosure relates to a system for achieving safe autonomous driving. [Background technology]

[0002] There is a shortage of drivers for delivery trucks and taxis, etc., which is a problem that can be resolved with the realization of autonomous driving. To realize autonomous driving, it is necessary to be able to check for obstacles in the blind spots of autonomous vehicles.

[0003] Patent Document 1 discloses an information provision system for compensating for blind spots of an autonomous vehicle to prevent collision with a moving object such as an oncoming vehicle. In this information providing system, roadside devices installed near roads detect moving objects. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-50627 A Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 does not mention the timing at which the roadside equipment starts detecting moving objects, and it is assumed that the roadside equipment is constantly detecting moving objects. As a result, moving objects are detected even during times when there is no risk of collision with the moving object, which requires communication and processing of a huge amount of information including unnecessary detection information, placing a high load on the information provision system.

[0006] The present disclosure aims to achieve safer autonomous driving by monitoring the surroundings of an autonomously driving vehicle using sensors installed independently of the autonomously driving vehicle when the autonomously driving vehicle passes through an area where a blind spot exists from the autonomously driving vehicle. [Means for solving the problem]

[0007] The autonomous driving system of the present disclosure includes a target moving body that is an autonomous driving moving body, a surrounding situation monitoring device, and an autonomous driving control device. The target moving body is A group of mobile sensors; an on-board device that determines whether or not monitoring by the surrounding situation monitoring device is necessary by referring to self-location data obtained by the mobile body sensor group and indicating the position of the target mobile body, and map data indicating a monitoring area set as an area in which the surrounding situation of the target mobile body should be monitored by the surrounding situation monitoring device when the target mobile body passes, and when it determines that monitoring by the surrounding situation monitoring device is necessary, transmits a monitoring request to the surrounding situation monitoring device and transmits driving situation data obtained by the mobile body sensor group and indicating the driving situation of the target mobile body to the automatic driving control device; Equipped with The surrounding situation monitoring device is A monitoring sensor group; a monitoring control device that receives the monitoring request, causes the monitoring sensor group to monitor the surrounding situation of the target moving object, and transmits monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device; Equipped with The autonomous driving control device receives the driving status data and the monitoring data, determines a driving obstacle of the target moving body based on the driving status data and the monitoring data, and transmits control data notifying the target moving body of the driving obstacle, The vehicle-mounted device receives the control data and controls the travel of the target moving object based on the control data and surrounding condition data obtained by the group of moving object sensors and indicating the surrounding condition of the target moving object. Effect of the Invention

[0008] According to the present disclosure, when an autonomously driving vehicle passes through an area where there is a blind spot from the autonomously driving vehicle, it is possible to realize safer autonomous driving by monitoring the surroundings of the autonomously driving vehicle using sensors that are installed independently of the autonomously driving vehicle. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a configuration diagram of an autonomous driving system 100 according to a first embodiment. [Diagram 2] FIG. 2 is a configuration diagram of an in-vehicle device 200 according to the first embodiment. [Diagram 3] FIG. 2 is a configuration diagram of a monitoring and control device 300 according to the first embodiment. [Figure 4] FIG. 2 is a configuration diagram of an automatic driving control device 400 according to the first embodiment. [Diagram 5] 4 is a flowchart of the vehicle-mounted device 200 according to the first embodiment. [Figure 6] 4 is a flowchart of the monitoring control device 300 in the first embodiment. [Figure 7] 4 is a flowchart of the autonomous driving control device 400 in the first embodiment. [Figure 8] FIG. 1 is a configuration diagram of an autonomous driving system 100 according to a second embodiment. [Figure 9] 10 is a flowchart of the vehicle-mounted device 200 according to the second embodiment. [Figure 10] 10 is a flowchart of the monitoring control device 300 in the second embodiment. [Figure 11] FIG. 11 is a configuration diagram of an autonomous driving system 100 according to a third embodiment. [Figure 12] 11 is a flowchart of the monitoring control device 300 in the third embodiment. [Figure 13] 11 is a flowchart of the automatic driving control device 400 in the fourth embodiment. [Figure 14] 13 is a flowchart of the vehicle-mounted device 200 according to the fourth embodiment. [Figure 15] FIG. 13 is a configuration diagram of an autonomous driving system 100 according to a fifth embodiment. [Figure 16] FIG. 13 is a configuration diagram of an automatic driving control device 400 according to a fifth embodiment. [Figure 17] 13 is a flowchart of the automatic driving control device 400 in the fifth embodiment. [Figure 18] FIG. 23 is a diagram showing a part of the functional configuration of an automatic driving control device 400 in a sixth embodiment. [Figure 19] FIG. 23 shows an example of a moving image stream AI processing pipeline 439 in the sixth embodiment. [Figure 20] FIG. 23 shows an example of a moving image stream AI processing pipeline 439 in the sixth embodiment. [Figure 21] FIG. 23 shows an example of a moving image stream AI processing pipeline 439 in the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In the embodiments and drawings, the same or corresponding elements are denoted by the same reference numerals. Descriptions of elements denoted by the same reference numerals as those of the elements described above will be omitted or simplified as appropriate. Arrows in the drawings mainly indicate data flow or processing flow.

[0011] Embodiment 1 The autonomous driving system 100 will be described with reference to FIGS. 1 to 7. FIG.

[0012] ***Configuration Description*** The configuration of an autonomous driving system 100 will be described with reference to FIG. The autonomous driving system 100 includes a target moving object 120, a surrounding situation monitoring device 130, an autonomous driving control device 400, and a map database 110.

[0013] The target moving body 120 is a moving body that travels on the ground by autonomous driving. For example, the target moving object 120 is an autonomous car, bus, or taxi.

[0014] The target moving object 120 includes an in-vehicle device 200 and a moving object sensor group 290 . The moving body sensor group 290 is one or more types of sensors mounted on the target moving body 120 . For example, the mobile sensor group 290 includes a depth camera 291, a lidar 292, and a positioning sensor 293. The positioning sensor 293 is a sensor that measures the position of the target moving object 120. For example, the positioning sensor 293 performs positioning using a satellite positioning system. An example of the satellite positioning system is the Global Positioning System (GPS). The moving object sensor group 290 is used to grasp the traveling condition of the target moving object 120 and the surrounding condition of the target moving object 120 . The driving conditions include position, speed, and acceleration. The surroundings include the positions of objects present in the surroundings.

[0015] The configuration of the vehicle-mounted device 200 will be described with reference to FIG. The vehicle-mounted device 200 is a computer including hardware such as a processor 201, a memory 202, an auxiliary storage device 203, a communication device 204, and an input / output interface 205. These pieces of hardware are connected to each other via signal lines.

[0016] The processor 201 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 201 is a CPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit.

[0017] The memory 202 is a volatile or non-volatile storage device. The memory 202 is also called a primary storage device or a main memory. For example, the memory 202 is a RAM. Data stored in the memory 202 is saved in the secondary storage device 203 as necessary. RAM is an abbreviation for Random Access Memory.

[0018] The auxiliary storage device 203 is a non-volatile storage device. For example, the auxiliary storage device 203 is a ROM, a HDD, a flash memory, or a combination of these. Data stored in the auxiliary storage device 203 is loaded into the memory 202 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0019] The auxiliary storage device 203 is a receiver and a transmitter. For example, the auxiliary storage device 203 is a communication chip or a NIC. The communication of the vehicle-mounted device 200 is performed using a communication device 204. NIC is an abbreviation for Network Interface Card.

[0020] The input / output interface 205 is a port to which an input device and an output device are connected. For example, a mobile sensor group 290 is connected to the input / output interface 205.

[0021] The vehicle-mounted device 200 includes components such as a monitoring request unit 211, a control data acquisition unit 212, and a driving control unit 213. These components are realized by software.

[0022] The auxiliary storage device 203 stores an automatic driving program for causing the computer to function as a monitoring request unit 211, a control data acquisition unit 212, and a driving control unit 213. The automatic driving program is loaded into the memory 202 and executed by the processor 201. The auxiliary storage device 203 further stores an OS. At least a portion of the OS is loaded into the memory 202 and executed by the processor 201. The processor 201 executes an automatic operation program while executing the OS. OS is an abbreviation for Operating System.

[0023] Data of the automatic driving program (input data, output data, etc.) is stored in the memory unit 220. The memory 202 functions as the storage unit 220. However, a storage device such as the auxiliary storage device 203, a register in the processor 201, or a cache memory in the processor 201 may function as the storage unit 220 instead of the memory 202 or together with the memory 202.

[0024] The autonomous driving program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

[0025] Returning to Figure 1, we will continue with the explanation. The surrounding situation monitoring device 130 is a device that monitors the surrounding situation of the target moving object 120 . For example, the surrounding situation monitoring device 130 is a drone or a roadside unit. A drone is an aircraft that flies in the sky by autonomous driving. A roadside unit is a device installed on a road in a monitoring area. The monitoring area will be described later.

[0026] The surrounding condition monitoring device 130 includes a monitoring control device 300 and a group of monitoring sensors 390 . The monitoring sensor group 390 is one or more types of sensors mounted on the surroundings monitoring device 130 . For example, the monitoring sensor group 390 includes a depth camera 391 and an RGB camera 392 . The monitoring sensors 390 are used to monitor the surroundings of the target moving object 120 .

[0027] The configuration of the monitoring and controlling device 300 will be described with reference to FIG. The monitoring and control device 300 is a computer that includes hardware such as a processor 301, a memory 302, an auxiliary storage device 303, a communication device 304, and an input / output interface 305. These pieces of hardware are connected to each other via signal lines.

[0028] The processor 301 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 301 is a CPU. The memory 302 is a volatile or non-volatile storage device. The memory 302 is also called a primary storage device or a main memory. For example, the memory 302 is a RAM. Data stored in the memory 302 is saved in the secondary storage device 303 as necessary. The auxiliary storage device 303 is a non-volatile storage device. The auxiliary storage device 303 is also called storage. For example, the auxiliary storage device 303 is a ROM, a HDD, a flash memory, or a combination of these. Data stored in the auxiliary storage device 303 is loaded into the memory 302 as needed. The communication device 304 is a receiver and a transmitter. For example, the communication device 304 is a communication chip or a NIC. The communication of the monitor and control device 300 is performed using the communication device 304. The input / output interface 305 is a port to which an input device and an output device are connected.

[0029] The monitoring control device 300 comprises elements such as a request receiving unit 311, a monitoring control unit 312, a monitoring data generating unit 313, and a monitoring data providing unit 314. These elements are realized by software.

[0030] The auxiliary storage device 303 stores a monitoring program for causing a computer to function as a request receiving unit 311, a monitoring control unit 312, a monitoring data generating unit 313, and a monitoring data providing unit 314. The monitoring program is loaded into the memory 302 and executed by the processor 301. The auxiliary storage device 303 further stores an OS. At least a portion of the OS is loaded into the memory 302 and executed by the processor 301. The processor 301 executes a monitoring program while executing the OS.

[0031] Input and output data of the monitoring program is stored in the storage unit 320 . The memory 302 functions as the storage unit 320. However, a storage device such as the auxiliary storage device 303, a register in the processor 301, or a cache memory in the processor 301 may function as the storage unit 320 instead of the memory 302 or together with the memory 302.

[0032] The monitoring program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0033] Returning to FIG. 1, the explanation will be continued. The autonomous driving control device 400 is a device that manages and controls the autonomous driving of the target moving body 120.

[0034] The configuration of the automatic driving control device 400 will be described with reference to FIG. The autonomous driving control device 400 is a computer including hardware such as a processor 401, a memory 402, an auxiliary storage device 403, a communication device 404, and an input / output interface 405. These pieces of hardware are connected to each other via signal lines.

[0035] The processor 401 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 401 is a CPU. The memory 402 is a volatile or non-volatile storage device. The memory 402 is also called a primary storage device or a main memory. For example, the memory 402 is a RAM. Data stored in the memory 402 is saved in the secondary storage device 403 as necessary. The auxiliary storage device 403 is a non-volatile storage device. The auxiliary storage device 403 is also called storage. For example, the auxiliary storage device 403 is a ROM, a HDD, a flash memory, or a combination of these. Data stored in the auxiliary storage device 403 is loaded into the memory 402 as needed. The communication device 404 is a receiver and a transmitter. For example, the communication device 404 is a communication chip or a NIC. Communication of the autonomous driving control device 400 is performed using the communication device 404. The input / output interface 405 is a port to which an input device and an output device are connected.

[0036] The autonomous driving control device 400 includes elements such as a data receiving unit 411, a driving obstacle determining unit 412, a control data generating unit 413, and a control data providing unit 414. These elements are realized by software.

[0037] The auxiliary storage device 403 stores a control program for causing the computer to function as a data receiving unit 411, a driving obstacle determining unit 412, and a control data generating unit 413. The control program is loaded into the memory 402 and executed by the processor 401. The auxiliary storage device 403 further stores an OS. At least a portion of the OS is loaded into the memory 402 and executed by the processor 401. The processor 401 executes a control program while executing the OS.

[0038] Input and output data for the control program are stored in memory unit 420. The memory 402 functions as the storage unit 420. However, a storage device such as the auxiliary storage device 403, a register in the processor 401, or a cache memory in the processor 401 may function as the storage unit 420 instead of the memory 402 or together with the memory 402.

[0039] The control program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0040] Returning to FIG. 1, the explanation will be continued. The map database 110 is a device that manages the map data 111 . The map data 111 will be described later.

[0041] ***Explanation of Operation*** The operation procedure of the automatic driving system 100 corresponds to an automatic driving method. The operation procedure of the automatic driving system 100 corresponds to a processing procedure by an automatic driving system program. The automatic driving system program includes an automatic driving program, a monitoring program, and a control program.

[0042] The following describes the operation of the vehicle-mounted device 200. First, the premise of the operation of the vehicle-mounted device 200 will be described. The mobile object sensor group 290 periodically observes the traveling conditions of the target mobile object 120 and the surrounding conditions of the target mobile object 120. For example, a depth camera 291 measures the surroundings of the target mobile object 120, a lidar 292 measures the distance to each point around the target mobile object 120, and a positioning sensor 293 performs satellite positioning. The mobile sensor group 290 provides driving condition data and surrounding condition data. The traveling condition data is data that indicates the traveling condition of the target moving object 120. The traveling condition data indicates the position, speed, acceleration, etc. of the target moving object 120. The position of the target moving object 120 is referred to as the self-position. Data that indicates the self-position is referred to as the self-position data. The surrounding situation data is data that indicates the surrounding situation of the target moving object 120. The surrounding situation data indicates various objects and the like observed by the moving object sensor group 290 mounted on the target moving object 120. The driving control unit 213 controls the driving of the target moving object 120 based on the driving plan data and the surrounding situation data. The driving plan data indicates a planned driving route of the target moving object 120 .

[0043] The procedure of the operation of the vehicle-mounted device 200 will be described with reference to FIG. Steps S121 to S125 are repeatedly executed.

[0044] In step S121, the monitoring request unit 211 refers to the self-location data and the map data 111, and determines whether monitoring by the surrounding situation monitoring device 130 is required. The map data 111 is obtained by communication with a map database 110 .

[0045] The map data 111 indicates one or more surveillance areas. The monitoring area is set as an area in which the surrounding conditions of the target moving object 120 should be monitored by the surrounding conditions monitoring device 130 when the target moving object 120 passes through. Specifically, the monitoring area is a place where a blind spot from the target moving object 120 occurs, such as an area before and after a curve.

[0046] Whether or not monitoring by the surroundings monitoring device 130 is required is determined as follows. The monitoring request unit 211 determines whether the target moving object 120 has entered the monitoring area. For example, the monitoring request unit 211 maps the position of the target moving object 120 on the map shown in the map data 111. Then, the monitoring request unit 211 determines whether the position of the target moving object 120 has been mapped within the monitoring area. When the target moving object 120 enters a monitoring area, the monitoring request unit 211 determines that monitoring by the surrounding situation monitoring device 130 is necessary.

[0047] If it is determined that monitoring by the surroundings monitoring device 130 is necessary, the process proceeds to step S122.

[0048] In step S122, the monitoring request unit 211 transmits a monitoring request to the surrounding situation monitoring device . The monitoring request is data for requesting the surrounding situation monitoring device 130 to monitor the surrounding situation of the target moving object 120 . The monitoring request indicates an identifier of the target mobile object 120, the location of the target mobile object 120, and the range of the monitoring area.

[0049] In step S123, the control data acquisition unit 212 transmits the driving plan data and the driving situation data to the autonomous driving control device 400.

[0050] Steps S122 and S123 may be performed in the reverse order.

[0051] In step S124, the control data acquisition unit 212 receives the control data transmitted from the automatic driving control device 400. The control data is data for notifying the target moving body 120 of a driving obstacle. For example, the control data indicates the positions of objects (obstacles) that are impeding the travel of the target moving object 120.

[0052] In step S125, the driving control unit 213 controls the traveling of the target moving object 120 based on the control data and the surrounding situation data.

[0053] The travel of the target moving object 120 is controlled as follows. First, the driving control unit 213 changes the driving plan. For example, the driving control unit 213 changes the planned driving route to a route that can avoid the obstacles indicated in the control data and the objects indicated in the surrounding situation data. Next, the driving control unit 213 determines control (driving control) for the driving device of the target moving body 120 so that the target moving body 120 can travel according to the changed travel plan. Examples of the driving device are a steering device (handle) and a motor (engine). Then, the driving control unit 213 controls the driving device of the target moving object 120 according to the determined driving control.

[0054] The procedure of the operation of the monitoring and control device 300 will be described with reference to FIG. Steps S131 to S134 are executed every time a monitoring request is transmitted from the target moving object 120.

[0055] In step S 131 , the request receiving unit 311 receives a monitoring request transmitted from the target moving object 120 .

[0056] In step S132 , the monitoring control unit 312 causes the monitoring sensors 390 to monitor the surrounding conditions of the target moving object 120 .

[0057] The surroundings of the target moving object 120 are monitored as follows. The monitoring control unit 312 outputs operation commands to the monitoring sensors 390 . The monitoring sensors 390 start operating upon receiving an operation command. For example, the depth camera 391 and the RGB camera 392 each capture images of the surroundings of the target moving object 120.

[0058] The monitoring sensors 390 monitor the surroundings of the target moving object 120 to obtain sensor data.

[0059] In step S133, the monitoring data generation unit 313 uses the sensor data obtained from the monitoring sensor group 390 to generate monitoring data. The monitoring data is data that indicates the surrounding conditions of the target moving object 120. The monitoring data indicates the positions of various objects detected by the monitoring sensors 390, etc.

[0060] The monitoring data is generated as follows: First, the monitoring data generation unit 313 executes object detection using the sensor data at each time, and detects each object present around the target moving object 120. Next, the monitoring data generation unit 313 calculates the speed of each object based on the position of each object at each time, and predicts the moving path of each object. Then, the monitoring data generator 313 generates data indicating the position, speed, and predicted path of each detected object. The generated data is the monitoring data.

[0061] In step S134, the monitoring data providing unit 314 transmits the monitoring data to the autonomous driving control device 400.

[0062] The procedure of the operation of the automatic driving control device 400 will be described with reference to FIG. Steps S141 to S145 are executed every time the target moving object 120 transmits the travel plan data and the travel condition data and the surrounding condition monitoring device 130 transmits the monitoring data.

[0063] In step S141, the data receiving unit 411 receives the driving plan data and the driving status data transmitted from the target moving object 120.

[0064] In step S142, the data receiving unit 411 receives the monitoring data transmitted from the surrounding situation monitoring device .

[0065] Steps S141 and S142 may be performed in the reverse order.

[0066] In step S143, the driving obstacle determination unit 412 determines whether or not there is a driving obstacle for the target moving object 120 based on the driving condition data and the monitoring data.

[0067] The presence or absence of a road obstacle for the target moving object 120 is determined as follows. The traveling obstacle determination unit 412 determines whether an obstacle facing the target moving object 120 exists within a proximity range from the position of the target moving object 120. The proximity range is a range that has the position of the target moving object 120 as its base point, and the size of the proximity range is determined in advance. The position of the target moving body 120 and the position of each object are managed using a position database. The position database is managed by the autonomous driving control device 400. The driving obstacle determination unit 412 maps the position of the target moving body 120 and the position of each object on a map shown in the position database. Then, the driving obstacle determination unit 412 makes a determination by referring to the map on which the position of the target moving body 120 and the position of each object are mapped. If there is an obstacle facing the target moving object 120 within a proximity range from the position of the target moving object 120, the traveling obstacle determination unit 412 determines that there is an obstacle for the target moving object 120 to travel.

[0068] If it is determined that there is an obstacle to the travel of the target moving object 120, the process proceeds to step S144. If it is determined that there is no obstacle to the travel of the target moving object 120, the process ends.

[0069] In step S144, the control data generation unit 413 generates control data for notifying the target moving body 120 of a driving obstacle.

[0070] In step S145 , the control data providing unit 414 transmits the control data to the target moving object 120 .

[0071] ***Advantages of the First Embodiment*** According to the first embodiment, when the target moving object 120 passes through a monitoring area where a blind spot occurs from the target moving object 120, the surroundings of the target moving object 120 can be monitored using the monitoring sensor group 390 provided independently of the target moving object 120. This makes it possible to realize safer automated driving.

[0072] ***Addition to the first embodiment*** The autonomous driving system 100 includes a target moving object 120, a surrounding situation monitoring device 130, an autonomous driving control device 400, and a map database 110. The target moving object 120 is a moving object capable of autonomous driving. The target moving object 120 has sensors for detecting the surrounding environment, such as an optical sensor and a LIDAR, and a communication function. The surrounding situation monitoring device 130 has sensors such as a camera that exists independently of the target moving object 120, and a communication function. The autonomous driving control device 400 has communication and calculation functions, and notifies the target moving body 120 of the control data required for planning a driving plan. The map database 110 has a storage function and a communication function. The map database 110 stores map data 111 used to determine the necessity of checking the surrounding situation. The map database 110 can communicate with the target moving object 120, the surrounding situation monitoring device 130, and the automatic driving control device 400. The target moving object 120 transmits information such as its own position, speed, and acceleration to the autonomous driving control device 400. The target moving object 120 also maps its own position on a map shown in the map data 111, and determines whether or not it is necessary to monitor the surrounding situation. If it is necessary to monitor the surrounding situation, the target moving object 120 transmits a monitoring request to the surrounding situation monitoring device 130. The surrounding situation monitoring device 130 may be fixed to the roadside, or may be an aircraft launched from the target moving object 120. The surrounding situation monitoring device 130 acquires information such as the position, speed, and acceleration of an obstacle that affects the travel plan of the target moving object 120 from the monitoring sensor group 390. Then, the surrounding situation monitoring device 130 transmits information such as the position, speed, and acceleration of the obstacle to the automatic driving control device 400. The surrounding situation monitoring device 130 also maps the position of the obstacle on a map shown in the map data 111. The autonomous driving control device 400 judges a driving obstacle of the target moving body 120 based on information such as the position (including future position) of the target moving body 120 and information such as the position (including future position) of an obstacle. Then, the autonomous driving control device 400 generates control data based on the judgment result of the driving obstacle and transmits the control data to the target moving body 120.

[0073] Embodiment 2 Regarding an embodiment in which the surrounding situation monitoring device 130 is a drone 131, the differences from the first embodiment will be mainly described with reference to Figs. 8 to 10.

[0074] ***Configuration Description*** The configuration of the autonomous driving system 100 will be described with reference to FIG. The autonomous driving system 100 is equipped with a drone 131 as an ambient situation monitoring device 130. Drone 131 is an autonomous flying vehicle that flies in the sky. The target moving body 120 includes a drone port 121 in addition to the elements described in the first embodiment. The drone port 121 is a facility where the processor 301 is stored and where the processor 301 takes off and lands. The drone port 121 has a charging port for charging the drone 131.

[0075] ***Explanation of Operation*** The procedure of the operation of the vehicle-mounted device 200 will be described with reference to FIG. Steps S221 to S225 correspond to steps S121 to S125 in the first embodiment.

[0076] In step S221, the monitoring request unit 211 refers to the self-location data and the map data 111 and determines whether monitoring by the drone 131 is necessary.

[0077] If it is determined that monitoring by the drone 131 is necessary, the process proceeds to step S222.

[0078] In step S222, the monitoring request unit 211 sends a monitoring request to the drone 131.

[0079] The monitoring request includes self-location data and monitoring area data. The monitoring area data indicates the monitoring area through which the target mobile object 120 passes.

[0080] In step S223, the control data acquisition unit 212 transmits the driving plan data and the driving situation data to the autonomous driving control device 400.

[0081] In step S224, the control data acquisition unit 212 receives the control data transmitted from the automatic driving control device 400.

[0082] In step S225, the driving control unit 213 controls the traveling of the target moving object 120 based on the control data and the surrounding situation data.

[0083] The procedure of the operation of the monitor and control device 300 will be described with reference to FIG. Steps S231 to S234 correspond to steps S131 to S134 in the first embodiment.

[0084] In step S 231 , the request receiving unit 311 receives a monitoring request transmitted from the target moving object 120 .

[0085] The monitoring request includes self-location data and monitoring area data.

[0086] In step S232, the monitoring control unit 312 causes the monitoring sensors 390 to monitor the surroundings of the target moving object 120.

[0087] The monitoring control unit 312 causes the drone 131 to fly above the monitoring area in order to have the monitoring sensor group 390 monitor the surrounding conditions of the target moving object 120.

[0088] Drone 131 flies over the surveillance area as follows: First, the monitoring control unit 312 plans a flight route for the drone 131 based on the self-location data and the monitored area data. The planned flight route is a route from taking off from drone port 121 of the target moving body 120, flying over the monitored area, and landing at drone port 121 of the target moving body 120. Then, the monitoring and control unit 312 causes the drone 131 to fly along the planned flight path. To this end, the monitoring and control unit 312 controls the driving device of the drone 131.

[0089] In step S233, the monitoring data generation unit 313 uses the sensor data obtained from the monitoring sensor group 390 to generate monitoring data.

[0090] In step S234, the monitoring data providing unit 314 transmits the monitoring data to the autonomous driving control device 400.

[0091] ***Effects of the second embodiment*** According to the second embodiment, it becomes possible to monitor from the sky a monitoring area that is a blind spot from the target moving object 120 traveling on the ground.

[0092] ***Additional information to the second embodiment*** The ambient situation monitoring device 130 is an autonomous flying vehicle. The autonomous flying object receives a monitoring request, calculates a flight plan based on the information on the monitoring area that triggered the monitoring request, and starts flying. The autonomous flying object obtains the position, speed, and acceleration of an obstacle from data obtained from the onboard sensor, and transmits the position, speed, and acceleration of the obstacle to the autonomous driving control device 400. The target moving object 120 has a port for the autonomous flying object to take off and land. The target moving object 120 acquires the position information of an area where a blind spot occurs from the map data 111, and launches the autonomous flying object when it enters the area where a blind spot occurs. The autonomous flying object then monitors the situation in the blind spot area to determine the position, speed, and acceleration of an obstacle, and transmits the position, speed, and acceleration of the obstacle to the autonomous driving control device 400.

[0093] Embodiment 3 The embodiment in which the position of an obstacle is detected using a color sensor will be described below, focusing mainly on the differences from the first embodiment, with reference to Figs.

[0094] ***Configuration Description*** The configuration of the autonomous driving system 100 will be described with reference to FIG. In the surroundings monitoring device 130 , the monitoring sensor group 390 includes a color sensor 393 . The color sensor 393 is an RGB camera or an RGB-D camera. R stands for red, G stands for green, B stands for blue. D stands for depth.

[0095] ***Explanation of Operation*** The procedure of the operation of the monitoring and control device 300 will be described with reference to FIG. Steps S331 to S334 correspond to steps S131 to S134 in the first embodiment.

[0096] In step S331, the request receiving unit 311 receives a monitoring request transmitted from the target moving object 120.

[0097] In step S332, the monitoring control unit 312 causes the monitoring sensors 390 to monitor the surroundings of the target moving object 120.

[0098] In step S333, the monitoring data generation unit 313 uses the sensor data obtained from the monitoring sensor group 390 to generate monitoring data.

[0099] The monitoring data indicates the positions of each object present around the target moving object 120, etc.

[0100] The position of each object is estimated as follows: The sensor data obtained by the color sensor 393 is referred to as color sensor data. First, the monitoring data generation unit 313 executes object detection using color sensor data to detect objects present around the target moving object 120 and the plane on which the objects are located. The monitoring data generator 313 then estimates the position of the object on the detected plane.

[0101] In step S334, the monitoring data providing unit 314 transmits the monitoring data to the autonomous driving control device 400.

[0102] ***Effects of the Third Embodiment*** According to the third embodiment, it is possible to more accurately detect the position of each object that may become an obstacle.

[0103] ***Additional Note to the Third Embodiment*** If the connected sensor is an RGB sensor, the surroundings monitoring device 130 detects the horizontal plane and estimates the coordinates of the detected obstacle relative to the detected horizontal plane. However, the estimation may also be performed by the autonomous driving control device 400. When the sensor that exists independently of the target moving object 120 is a camera that obtains RGB information, the surrounding situation monitoring device 130 performs object detection, detects a plane based on the RGB information, and estimates the position of the object relative to the detected plane. When the sensor that exists independently of the target moving body 120 is a camera that obtains RGB-D information, the surrounding situation monitoring device 130 performs object detection, detects a plane based on the RGB-D information, and estimates the position of the object relative to the detected plane.

[0104] The third embodiment may be implemented in combination with the second embodiment. That is, the surrounding situation monitoring device 130 may be a drone 131, and the target moving object 120 may include a drone port 121.

[0105] Embodiment 4 The manner in which the target moving object 120 deals with a travel obstacle depending on the type of the obstacle will be described below with reference to Figs. 13 and 14, mainly with respect to the points different from the first embodiment.

[0106] ***Configuration Description** The configuration of the autonomous driving system 100 is similar to that in the first embodiment. However, the map data 111 indicates one or more escape zones.

[0107] ***Explanation of Operation*** The procedure of the operation of the automatic driving control device 400 will be described with reference to FIG. Steps S441 to S445 correspond to steps S141 to S145 in the first embodiment.

[0108] In step S441, the data receiving unit 411 receives the driving plan data and the driving status data transmitted from the target moving object 120.

[0109] In step S442, the data receiving unit 411 receives the monitoring data transmitted from the surrounding situation monitoring device .

[0110] In step S443, the driving obstacle determination unit 412 determines whether or not there is a driving obstacle for the target moving object 120 based on the driving condition data and the monitoring data.

[0111] When it is determined that there is a traveling obstacle for the target moving body 120, the traveling obstacle determination unit 412 determines the type of obstacle based on the monitoring data. Specifically, the traveling obstacle determination unit 412 determines whether the obstacle is a moving object or a non-moving object. Examples of moving objects are vehicles and people. An example of a non-moving object is falling rocks. The traveling obstacle determination unit 412 also determines whether there are multiple obstacles. An example of multiple obstacles is multiple pedestrians. In addition, the driving obstacle determination unit 412 determines the type of road (target road) on which the target moving object 120 and the obstacle exist, based on the position database. Specifically, the driving obstacle determination unit 412 determines whether the target road is a narrow road having a width of only one lane.

[0112] If it is determined that there is an obstacle to the travel of the target moving object 120, the process proceeds to step S444. If it is determined that there is no obstacle to the travel of the target moving object 120, the process ends.

[0113] In step S444, the control data generation unit 413 generates control data for notifying the target moving body 120 of a driving obstacle.

[0114] The control data indicates the type of driving obstacle, etc.

[0115] The type of driving impairment is determined as follows: If the target road is a narrow road and the obstacle is a moving object, the control data generating unit 413 determines the type of driving obstacle to be "no passing". "No passing" means that the target moving body 120 and the obstacle cannot pass each other when traveling. If the target road is a narrow road and the obstacle is a stationary object, the control data generating unit 413 determines the type of driving obstacle as "impassable." "Impassable" means that the target moving body 120 cannot avoid the obstacle and pass through it. When multiple obstacles are present, the control data generation unit 413 transmits control data indicating "slow down required" as control information. "Slow down required" means that slow down is required to avoid the obstacles.

[0116] In step S445, the control data providing unit 414 transmits the control data to the target moving object 120.

[0117] The procedure of the operation of the vehicle-mounted device 200 will be described with reference to FIG. Steps S421 to S425 correspond to steps S121 to S125 in the first embodiment.

[0118] In step S421, the monitoring request unit 211 refers to the self-location data and the map data 111, and determines whether monitoring by the surrounding situation monitoring device 130 is required.

[0119] If it is determined that monitoring by the surroundings monitoring device 130 is necessary, the process proceeds to step S422.

[0120] In step S422, the monitoring request unit 211 transmits a monitoring request to the surrounding situation monitoring device .

[0121] In step S423, the control data acquisition unit 212 transmits the driving plan data and the driving situation data to the autonomous driving control device 400.

[0122] In step S424, the control data acquisition unit 212 receives the control data transmitted from the automatic driving control device 400.

[0123] The control data indicates the type of obstacle to the target moving body 120's travel, etc.

[0124] In step S425, the driving control unit 213 controls the traveling of the target moving object 120 based on the control data and the surrounding situation data.

[0125] The travel of the target moving object 120 is controlled according to the type of travel obstacle as follows. When the type of the driving obstacle is “no passing”, the driving control unit 213 causes the target moving object 120 to retreat to the nearest escape zone shown in the map data 111. If the type of the travel obstacle is “impassable”, the driving control unit 213 makes the target moving object 120 make a U-turn using the nearest escape zone shown in the map data 111. When the type of driving obstacle is "requires slowing down", the driving control unit 213 causes the target moving object 120 to drive slowly.

[0126] ***Effects of the fourth embodiment*** The fourth embodiment makes it possible to realize safe automated driving according to the type of driving obstacle.

[0127] ***Additional information to the fourth embodiment*** The autonomous driving control device 400 obtains information on the position, speed, and acceleration from the target moving body 120, and obtains information on the position, speed, and acceleration of an obstacle from the surroundings monitoring device 130. The autonomous driving control device 400 records future position information calculated based on the position information of the target moving body 120 and the obstacle, together with the predicted time, in a database having version information and space-time axis information. Every time the position information and future predicted positions of the target moving body 120 and obstacles are changed, the autonomous driving control device 400 updates the version and updates the position information of the target moving body 120 and obstacles. If the position of the target moving body 120 is in a close relationship with the position of the obstacle at the time of updating, the autonomous driving control device 400 determines that there is a driving obstacle, and the target moving body 120 modifies the driving plan. When the type of the driving obstacle is determined to be "no passing", the target moving body 120 acquires the position information of the escape zone from the map data 111, and changes the driving plan to take refuge in the escape zone. If the type of driving obstacle is determined to be "impassable", the target moving body 120 acquires the position information of the escape zone from the map data 111, and changes the driving plan to make a U-turn using the escape zone. If the type of driving obstacle is determined to be "required to drive slowly," the target moving body 120 changes its driving plan to drive slowly.

[0128] The fourth embodiment may be implemented in combination with at least one of the second and third embodiments. That is, the surrounding situation monitoring device 130 may be a drone 131, and the target moving object 120 may include a drone port 121. The monitoring control device 300 may also use a color sensor 393 to detect the position of an obstacle.

[0129] Embodiment 5. The embodiment of notifying an obstacle of a travel obstacle of the target moving object 120 will be described with reference to Figs. 15 to 17, mainly focusing on the points different from the first embodiment.

[0130] ***Configuration Description** The configuration of the autonomous driving system 100 will be described with reference to FIG. The autonomous driving system 100 includes one or more moving objects 101 and one or more mobile terminals 102. The moving object 101 is a moving object similar to the target moving object 120 . The mobile terminal 102 is carried by a pedestrian. For example, the mobile terminal 102 is a smartphone. An application program for the autonomous driving system 100 is executed on the smartphone. Information on each moving object 101 and information on each mobile terminal 102 are registered in advance in the automatic driving control device 400.

[0131] The configuration of the automatic driving control device 400 will be described with reference to FIG. The autonomous driving control device 400 further includes an element called a driving obstacle notification unit 415. The automatic driving control program further causes the computer to function as a driving obstacle notification unit 415.

[0132] ***Explanation of Operation*** The following describes the operation of the autonomous driving control device 400. First, the premise of the operation of the autonomous driving control device 400 will be described. Each moving object 101 periodically transmits its own position data to the automatic driving control device 400. The own position data indicates the position of the moving object 101. Each mobile terminal 102 periodically transmits its own position data to the autonomous driving control device 400. The own position data indicates the positions of the pedestrian and the mobile terminal 102.

[0133] The procedure of the operation of the automatic driving control device 400 will be described with reference to FIG. Steps S542 to S546 correspond to steps S141 to S145 in the first embodiment.

[0134] In step S541, the data receiving unit 411 receives the self-location data from each moving object 101. Moreover, the data receiving unit 411 receives self-location data from each mobile terminal 102 .

[0135] In step S542, the data receiving unit 411 receives the driving plan data and the driving status data transmitted from the target moving object 120.

[0136] In step S543, the data receiving unit 411 receives the monitoring data transmitted from the surrounding situation monitoring device .

[0137] In step S544, the driving obstacle determination unit 412 determines whether or not there is a driving obstacle for the target moving object 120 based on the driving condition data and the monitoring data.

[0138] In step S545, the control data generation unit 413 generates control data for notifying the target moving body 120 of a driving obstacle.

[0139] In step S546, the control data providing unit 414 transmits the control data to the target moving object 120.

[0140] In step S547, the traveling obstacle notification unit 415 determines whether the obstacle is any one of the moving bodies 101 based on the self-position data of each of the one or more moving bodies 101. When it is determined that the obstacle is one of the moving objects 101, the traveling obstacle notification unit 415 transmits notification data to the moving object 101 that is the obstacle. The notification data is data for notifying a traveling obstacle of the target moving body 120. For example, the traveling obstacle notification unit 415 generates control data for the moving body 101, which is an obstacle, as the notification data.

[0141] Furthermore, the driving obstacle notification unit 415 determines whether the obstacle is a pedestrian carrying one of the mobile terminals 102, based on the self-position data of each of the one or more mobile terminals 102. When it is determined that the obstacle is a pedestrian carrying any one of the mobile terminals 102, the driving obstacle notification unit 415 transmits notification data to the mobile terminal 102 of the pedestrian who is the obstacle.

[0142] ***Effects of the Fifth Embodiment*** According to the fifth embodiment, moving objects and pedestrians that are obstacles to the target moving object 120 are notified of a road obstacle for the target moving object 120. This makes it possible to realize safer automated driving.

[0143] ***Additional information to the fifth embodiment*** If it is determined that there is a driving obstacle and that the obstacle is a moving object driven by a human or a pedestrian, the autonomous driving control device 400 sends a notification to the smartphone carried by the human. Also, if the obstacle is a moving object capable of automatic driving, the autonomous driving control device 400 sends information to instruct a change to the driving plan. Obstacles may be differentiated into objects that are expected to be movable and objects that are not expected to be movable. For example, potential obstacles include vehicles driven by humans, autonomous vehicles (mobile objects capable of autonomous driving), non-moving objects (parked vehicles, falling rocks, etc.), and other objects (pedestrians, cyclists, vulnerable road users, etc.).

[0144] The fifth embodiment may be implemented in combination with at least one of the second to fourth embodiments. That is, the surrounding situation monitoring device 130 may be a drone 131, and the target moving object 120 may include a drone port 121. The monitoring control device 300 may detect the position of an obstacle using a color sensor 393. The target moving object 120 may deal with a traveling obstacle according to the type of the traveling obstacle.

[0145] Embodiment 6 Regarding an embodiment in which a moving image stream AI processing pipeline system is applied, differences from the first embodiment will be mainly described with reference to Fig. 18 to Fig. 21. AI is an abbreviation for artificial intelligence.

[0146] ***Configuration Description*** The configuration of the autonomous driving system 100 will be described with reference to FIG. The autonomous driving system 100 includes a plurality of surrounding condition monitoring devices 130. In Fig. 18, the autonomous driving system 100 includes four surrounding condition monitoring devices 130, namely, one drone and three roadside units. The number of surrounding condition monitoring devices 130 is not limited to four. The monitoring sensor group 390 of each surroundings monitoring device 130 includes a video image sensor.

[0147] The autonomous driving control device 400 further includes elements such as a switcher unit 431, an object detection unit 432, a position estimation unit 433, an annotation unit 434, and a learning unit 435. These elements are realized by software. Furthermore, the autonomous driving control device 400 includes a database 438. The database 438 has space and time axis information. The autonomous driving control device 400 has a plurality of types of object detection models. The object detection models are learned models generated by machine learning or the like in order to detect objects shown in an image. In FIG. 18, the autonomous driving control device 400 has a general-purpose object detection model, a general-purpose road detection model, and a general-purpose human detection model. The road detection model is an object detection model for detecting roads. The human detection model is an object detection model for detecting people. The object detection models used are not limited to these models.

[0148] ***Explanation of Operation*** The operation of the multiple surrounding condition monitoring devices 130 and the automatic driving control device 400 will be described with reference to FIG.

[0149] The monitoring control device 300 of each surrounding condition monitoring device 130 transmits monitoring data including a video stream obtained by a video sensor to the autonomous driving control device 400 (S133, S134 in FIG. 6).

[0150] In step S142 (see FIG. 7), the autonomous driving control device 400 receives monitoring data transmitted from each surrounding situation monitoring device 130. The monitoring data includes a video stream. In step S143 (see FIG. 7), the autonomous driving control device 400 detects obstacles to the traveling of the target moving object 120 using the object detection model, and determines whether there is an obstacle to the traveling of the target moving object 120 by using the obstacle detection result.

[0151] Obstacles are detected as follows: The input destination of the video stream from each surrounding condition monitoring device 130 is determined in advance in association with the surrounding condition monitoring device 130 that is the transmission source. The switcher unit 431 sends each video stream to an object detection model of an input destination associated with the surrounding condition monitoring device 130 of the transmission source. For example, the switcher unit 431 sends video stream A from roadside unit A to object detection and annotation, sends video stream B from roadside unit B to object detection, road detection, and annotation, and sends video stream C from roadside unit C to human detection and annotation. The object detection model used at the input destination associated with the surrounding situation monitoring device 130 that is the transmission source is called a corresponding model. The object detection unit 432 performs object detection for each video stream by using the corresponding model and inputting the video stream. The position estimation unit 433 estimates the position (coordinate values) of the detected object based on the result of the object detection, and records the detection information of the detected object in the database 438. The detection information indicates the type, position, etc. of the detected object. The user refers to the detection information of each object recorded in the database 438 using the autonomous driving control device 400, selects the object detection model to be updated, and specifies the selected object detection model to the autonomous driving control device 400. The selected object detection model is referred to as a selected model. The switcher unit 431 sends the video stream input to the selection model to the annotation unit 434 . The annotation unit 434 uses a selection model to input a video stream and corrects the detection information of a detected object with annotations. The detection information is corrected as follows. The annotation unit 434 displays the video stream on a display. The user refers to the video stream, annotates the object shown in the video stream, and inputs the annotation information to the autonomous driving control device 400. The annotation information indicates the position and label of the object. The label identifies the type of object. The annotation unit 434 receives the annotation information and corrects the detection information based on the annotation information. The learning unit 435 updates the selection model by learning using the video stream and the corrected detection information as learning data. For example, the parameters of the selection model are updated. As a result, a new version of the object detection model is generated. When a video stream is input to any type of object detection model, the object detection unit 432 performs object detection using the object detection model with the video stream as input for each version of the same type of object detection model. Then, detection information for each version of the object detection model is recorded in the database 438. The user refers to each piece of detection information for each object recorded in the database 438 using the autonomous driving control device 400. The user determines an object detection model to be used for determining a driving obstacle from a plurality of types of object detection models with different versions. Then, the user specifies the determined object detection model to the autonomous driving control device 400. The determined object detection model is referred to as a determined model. Thereafter, the object detection unit 432 performs object detection using a decision model with the video stream as input, and detects obstacles to the running of the target moving object 120. The detection result includes prediction information of the obstacle's future behavior. The obstacle's future behavior is predicted based on the time series of the obstacle's position. The determination model is changed as needed. Two or more determination models may be used. In this case, an obstacle is identified based on two or more detection results corresponding to two or more determination models. In this case, a method such as weighted averaging is used.

[0152] The input destination of the above video streams is determined by the settings of the video stream AI processing pipeline 439. The video stream AI processing pipeline 439 is set for each surrounding situation monitoring device 130 and indicates the flow of the video stream. 19 to 21 show an example of a video stream AI processing pipeline 439. The "x" in "Vx" indicates the version number. In FIG. 19, video stream A from roadside unit A is sent to a general-purpose object detection model (V0), sent to annotation unit 434, annotation information is added, and sent to learning unit 435 as a part of learning data. Then, an object detection model (V1) is generated by updating the general-purpose object detection model (V0). Furthermore, detection information obtained by the general-purpose object detection model (V0) and position estimation is recorded in database 438. Thereafter, video stream A is also sent to the object detection model (V1), sent to annotation unit 434, annotation information is added, and the annotation information of video stream A is recorded in database 438. 20, video stream B from roadside unit B is sent to an object detection model (V2) generated by updating a general-purpose object detection model (V0), and then sent to annotation unit 434, where annotation information is added. Then, the movement of an obstacle is estimated based on the annotation information of video stream B, and detection information of the obstacle including information on the estimated movement is recorded in database 438. In FIG. 21, a video stream D from a drone D is sent to a general-purpose road detection model (V0) and a road detection model (V1) generated by updating the general-purpose road detection model (V0).

[0153] ***Effects of the sixth embodiment*** According to the sixth embodiment, it becomes possible to more accurately judge a driving obstacle using a video stream.

[0154] ***Additional information to embodiment 6*** The autonomous driving control device 400 processes information from the video sensors of the multiple monitoring and control devices 300. To this end, the autonomous driving control device 400 has a video switcher. The video switcher has a function of switching the flow of information from the video sensor between image processing functions such as object detection, road detection, and lane detection, and annotation functions for learning.

[0155] The video switcher acquires video images obtained from a plurality of monitor control devices 300 through a real-time communication stream, and distributes each stream to a required processing function. The autonomous driving control device 400 has the function of setting up a video switcher and an AI processing pipeline.

[0156] The video switcher settings include the following functions: a function to perform various detections and classifications using an AI model, a function to convert the detection results by the AI ​​model into video, a function to perform annotation for learning, a function to save video images, a function to adjust the images of the video images, etc. The video switcher settings are defined by describing a pipeline in the processing order of these functions.

[0157] The video stream AI processing pipeline system using a video switcher can be used for a variety of purposes in addition to autonomous driving. For example, the video stream AI processing pipeline system can be used for the following purposes: (1) Detecting the number of vehicles entering and leaving a parking lot (2) Detection of the number of parked cars in a parking lot (3) Detection of people entering and leaving the facility (4) Detecting the number of pedestrians on the road (uphill and downhill) (5) Detection of age and gender of passersby (6) Estimation of speed of pedestrians (7) Detection of stopping positions in relation to facilities (8) Detection and classification of work performed on facilities (9) Detection of human behavior

[0158] The sixth embodiment may be implemented in combination with at least one of the second to fifth embodiments.

[0159] ***Supplementary explanation of implementation form*** The autonomous driving system 100 includes a target moving object 120, a surrounding situation monitoring device 130, an autonomous driving control device 400, and a map database 110. The target moving object 120 is a moving object capable of autonomous driving. The target moving object 120 has sensors for detecting the surrounding environment, such as an optical sensor and a LIDAR, and a communication function. The surrounding situation monitoring device 130 has sensors such as a camera that exists independently of the target moving object 120, and a communication function. The autonomous driving control device 400 has a computer capable of communicating with the target moving object 120 and the surrounding situation monitoring device 130. The map database 110 stores map data 111 indicating surveillance areas and escape zones. The target moving object 120 estimates its own position using a computer, detects surrounding objects that may be obstacles using sensors, and estimates the position information of the obstacles through calculation. The surrounding situation monitoring device 130 detects the target moving body 120 and surrounding objects that may be obstacles using sensors, and estimates the position information of the obstacles through calculation. The surroundings monitoring device 130 complements the missing information about blind spots of the sensors of the target moving body 120. The automatic driving control device 400 generates information required to change the driving plan. The target moving body 120 determines the intersection of the predicted path of the object and the planned path of the target moving body 120, and recalculates and updates the driving plan, including the speed and path, so as not to hit the object. The updated driving plan is converted into control information and used for driving control of the target moving body 120.

[0160] The autonomous driving system 100 includes a target moving object 120, a drone 131, an autonomous driving control device 400, and a map database 110. The target moving object 120 is a moving object capable of autonomous driving, and has sensors for detecting the surrounding environment, such as optical sensors and LIDAR, and a communication function. The drone 131 is an aircraft with a flying function, and has sensors such as a camera that exist independently of the target moving object 120, and a communication function. The autonomous driving control device 400 has a communication function and a calculation function. The autonomous driving control device 400 notifies the target moving object 120 of control data required for planning a driving plan. The map database 110 has a storage function and a communication function. The map database 110 stores map data 111 used to determine the necessity of checking the surrounding situation. The map data 111 indicates a monitoring area and an escape zone. The map database 110 can communicate with the target moving object 120, the surrounding situation monitoring device 130, and the automatic driving control device 400. The drone 131 has a function of creating a flight plan for a round trip to a position where the blind spot can be viewed when the approach of the target moving body 120 to an area with a large blind spot is detected, and a function of controlling the drone 131 based on the flight plan.

[0161] Each embodiment is an example of a preferred embodiment, and is not intended to limit the technical scope of the present disclosure. Each embodiment may be implemented in part or in combination with other embodiments. The procedures described using flow charts, etc. may be modified as appropriate.

[0162] Each element of the vehicle-mounted device 200, the monitoring and control device 300, and the automatic driving control device 400 may be realized by software, hardware, firmware, or a combination of these. The "part" of each element of the in-vehicle device 200, the monitoring control device 300, and the automatic driving control device 400 may be read as "processing," "step," "circuit," or "processing circuit."

[0163] Various aspects of the present disclosure are described below as appendices. (Appendix 1) An autonomous driving system including a target moving body that is an autonomous driving moving body, a surrounding situation monitoring device, and an autonomous driving control device, The target moving body is A group of mobile sensors; an on-board device that determines whether or not monitoring by the surrounding situation monitoring device is necessary by referring to self-location data obtained by the mobile body sensor group and indicating the position of the target mobile body, and map data indicating a monitoring area set as an area in which the surrounding situation of the target mobile body should be monitored by the surrounding situation monitoring device when the target mobile body passes, and when it determines that monitoring by the surrounding situation monitoring device is necessary, transmits a monitoring request to the surrounding situation monitoring device and transmits driving situation data obtained by the mobile body sensor group and indicating the driving situation of the target mobile body to the automatic driving control device; Equipped with The surrounding situation monitoring device is A monitoring sensor group; a monitoring control device that receives the monitoring request, causes the monitoring sensor group to monitor the surrounding situation of the target moving object, and transmits monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device; Equipped with The autonomous driving control device receives the driving status data and the monitoring data, determines a driving obstacle of the target moving body based on the driving status data and the monitoring data, and transmits control data notifying the target moving body of the driving obstacle, The vehicle-mounted device receives the control data and controls the travel of the target moving object based on the control data and surrounding situation data obtained by the moving object sensor group and indicating the surrounding situation of the target moving object. Autonomous driving system.

[0164] (Appendix 2) The surrounding situation monitoring device is an autonomous flying vehicle. 2. An automated driving system as described in appendix 1.

[0165] (Appendix 3) The target moving body is provided with a drone port on which the flying object takes off and lands. Attachment 2: An automated driving system.

[0166] (Appendix 4) the vehicle-mounted device transmits the monitoring request including the self-location data and monitoring area data indicating the monitoring area; The monitoring control device receives the monitoring request, plans a flight path based on the self-location data and the monitored area data from taking off from the drone port, flying over the monitored area, and landing at the drone port, and causes the aircraft to fly along the flight path.

[0167] (Appendix 5) The surrounding situation monitoring device is a roadside unit installed in the monitoring area. 2. An automated driving system as described in appendix 1.

[0168] (Appendix 6) the monitoring sensor group includes a color sensor; The monitoring and control device includes: Detecting objects present around the target moving object and a plane on which the objects are located based on color sensor data obtained by the color sensor; Estimating the position of the object in the sensed plane. 10. The automated driving system of claim 1.

[0169] (Appendix 7) The color sensor is an RGB camera or an RGB-D camera. 6. An automated driving system as described in Appendix 6.

[0170] (Appendix 8) The autonomous driving control device determines that the driving obstacle exists when an obstacle facing the target moving object exists within a proximity range from the position of the target moving object. 8. The automated driving system according to any one of claims 1 to 7.

[0171] (Appendix 9) When the obstacle is a moving object, the autonomous driving control device transmits the control data indicating that passing is impossible as the type of the driving obstacle to the target moving body, When the type of the obstacle is the impossibility of passing each other, the vehicle-mounted device causes the target moving object to retreat to an escape zone indicated in the map data. 8. An automated driving system as described in Appendix 8.

[0172] (Appendix 10) When the obstacle is a stationary object, the autonomous driving control device transmits the control data indicating impassability as the type of the driving obstacle to the target moving body, When the type of the obstacle is the impassable obstacle, the vehicle-mounted device makes the target moving object make a U-turn by utilizing an escape zone shown in the map data. 10. An automated driving system as set forth in claim 8 or 9.

[0173] (Appendix 11) When a plurality of obstacles are present, the autonomous driving control device transmits the control data indicating a need to slow down as a type of the driving obstacle to the target moving body, The vehicle-mounted device makes the target moving object slow down when the type of the driving obstacle requires the target moving object to slow down. 11. The automated driving system according to any one of claims 8 to 10.

[0174] (Appendix 12) The autonomous driving control device receives self-location data from each of the one or more moving bodies, determines whether the obstacle is any of the one or more moving bodies based on the self-location data of each of the one or more moving bodies, and notifies the moving body that is the obstacle of the driving obstacle when it determines that the obstacle is any of the one or more moving bodies. 12. The automated driving system according to any one of claims 8 to 11.

[0175] (Appendix 13) The autonomous driving control device receives self-location data from one or more mobile devices carried by one or more pedestrians, determines whether the obstacle is any one of the one or more pedestrians based on the self-location data of each of the one or more mobile devices, and notifies the mobile device of the pedestrian who is the obstacle of the driving obstacle when it determines that the obstacle is any one of the one or more pedestrians. 13. The automated driving system according to any one of claims 8 to 12.

[0176] (Appendix 14) The autonomous driving system includes a plurality of the surrounding situation monitoring devices, The group of monitoring sensors of each of the surrounding situation monitoring devices includes a video image sensor, The monitoring control device of each of the surrounding situation monitoring devices transmits the monitoring data including the video stream obtained by the video sensor to the autonomous driving control device, The automatic driving control device includes: For each of the video streams, an object detection model corresponding to the surrounding situation monitoring device that is a transmission source among a plurality of types of object detection models is used to detect an object by inputting the video stream, and detection information of the detected object is recorded in a database; Using a selection model, which is an object detection model selected from the plurality of types of object detection models, the video stream is input and detection information of the detected object is corrected by annotation, and the selection model is updated by learning the video stream and the corrected detection information to generate a new version of the object detection model; The video stream is input and the object detection is performed using a determined model, which is an object detection model determined from the plurality of types of object detection models for each version, to detect an obstacle to the running of the target moving object. 14. The automated driving system of claim 1.

[0177] (Appendix 15) An autonomous driving method using an autonomous driving system including a target moving body that is an autonomous driving moving body, a surrounding situation monitoring device, and an autonomous driving control device, The target moving body is determining whether or not monitoring by the surrounding situation monitoring device is necessary by referring to self-location data indicating the position of the target moving body and map data indicating a monitoring area set as an area in which the surrounding situation of the target moving body should be monitored by the surrounding situation monitoring device when the target moving body passes, and when determining that monitoring by the surrounding situation monitoring device is necessary, transmitting a monitoring request to the surrounding situation monitoring device and transmitting driving situation data indicating the driving situation of the target moving body to the automatic driving control device; The surrounding situation monitoring device is receiving the monitoring request, monitoring a surrounding situation of the target moving object, and transmitting monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device; The automatic driving control device includes: receiving the driving condition data and the monitoring data, determining a driving obstacle for the target moving object based on the driving condition data and the monitoring data, and transmitting control data notifying the target moving object of the driving obstacle; The target moving body is The control data is received, and the target moving body travels in an automatic driving manner based on the control data and surrounding situation data indicating the surrounding situation of the target moving body. Autonomous driving method.

[0178] (Appendix 16) It is a vehicle that runs autonomously, A group of mobile sensors; an on-board device that determines whether or not monitoring by the surrounding conditions monitoring device is necessary by referring to self-position data obtained by the mobile body sensor group and indicating the position of the mobile body, and map data indicating a monitoring area set as an area in which the surrounding conditions of the mobile body should be monitored by a surrounding conditions monitoring device when the mobile body passes, and when it determines that monitoring by the surrounding conditions monitoring device is necessary, transmits a monitoring request to the surrounding conditions monitoring device and transmits driving condition data obtained by the mobile body sensor group and indicating the driving condition of the mobile body to an automatic driving control device, receives control data from the automatic driving control device that notifies of a driving obstacle of the mobile body determined based on the driving condition data and the monitoring data obtained by the surrounding conditions monitoring device and indicating the surrounding conditions of the mobile body, and controls the driving of the mobile body based on the control data and the surrounding conditions data obtained by the mobile body sensor group and indicating the surrounding conditions of the mobile body; A mobile body comprising:

[0179] (Appendix 17) The surrounding situation monitoring device is an autonomous flying object, The mobile body includes a drone port on which the flying object takes off and lands. 17. A moving object as described in appended claim 16.

[0180] (Appendix 18) A monitoring sensor group; a monitoring control device that receives a monitoring request from a target moving body traveling autonomously when the target moving body passes through a monitoring area, causes the monitoring sensor group to monitor the surrounding conditions of the target moving body to obtain monitoring data indicating the surrounding conditions of the target moving body, and transmits the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle of the target moving body determined based on the monitoring data and driving condition data indicating the driving condition of the target moving body; An ambient situation monitoring device comprising:

[0181] (Appendix 19) The target moving object includes a drone port; The surrounding situation monitoring device is an autonomous flying vehicle that takes off and lands using the drone port. 19. The ambient condition monitoring device of claim 18.

[0182] (Appendix 20) The monitoring control device receives the monitoring request including self-location data indicating the position of the target moving object and monitoring area data indicating the monitoring area, and plans a flight path from taking off from the drone port to flying over the monitoring area and landing at the drone port based on the self-location data and the monitoring area data, and makes the flying object fly along the flight path. 20. The ambient conditions monitoring device of claim 19.

[0183] (Appendix 21) A roadside unit installed in the monitoring area. 19. The ambient condition monitoring device of claim 18.

[0184] (Appendix 22) the monitoring sensor group includes a color sensor; The monitoring and control device includes: Detecting objects present around the target moving object and a plane on which the objects are located based on color sensor data obtained by the color sensor; Estimating the position of the object in the sensed plane. 22. The surrounding situation monitoring device according to any one of appendix 18 to appendix 21.

[0185] (Appendix 23) The color sensor is an RGB camera or an RGB-D camera. 23. The ambient condition monitoring device of claim 22.

[0186] (Appendix 24) Receiving driving status data indicating a driving status of a target moving object from the target moving object that is driving autonomously; receiving monitoring data indicating a surrounding situation of the target moving object from a surrounding situation monitoring device which receives a monitoring request from the target moving object when the target moving object passes through a monitoring area and monitors a surrounding situation of the target moving object; determining a traveling obstacle of the target moving object based on the traveling condition data and the monitoring data; Transmitting control data notifying the target moving body of the road obstruction to the target moving body. Autonomous driving control device.

[0187] (Appendix 25) When an obstacle facing the target moving object is present within a proximity range from the position of the target moving object, it is determined that the target moving object has a travel obstacle. 25. An automated driving control device as described in appendix 24.

[0188] (Appendix 26) When the obstacle is a moving object, the control data indicating that passing is not permitted as the type of the travel obstacle is transmitted to the target moving object, thereby causing the target moving object to retreat to an escape zone. 26. An automated driving control device as described in appendix 25.

[0189] (Appendix 27) When the obstacle is a stationary object, the control data indicating that the obstacle is a type of travel obstacle that is impassable is transmitted to the target moving object, thereby making the target moving object make a U-turn using an escape zone. An automated driving control device as described in Appendix 25 or Appendix 26.

[0190] (Appendix 28) When a plurality of obstacles are present, the control data indicating a need to slow down as a type of the travel obstacle is transmitted to the target moving body, thereby causing the target moving body to slow down. 28. An automatic driving control device according to any one of appendix 25 to appendix 27.

[0191] (Appendix 29) receiving self-location data from each of the one or more moving bodies, determining whether the obstacle is any one of the one or more moving bodies based on the self-location data of each of the one or more moving bodies, and notifying the moving body that is the obstacle of the driving obstacle when it is determined that the obstacle is any one of the one or more moving bodies; 29. An automatic driving control device according to any one of appendix 25 to appendix 28.

[0192] (Appendix 30) receiving self-location data from one or more portable devices carried by one or more pedestrians, determining whether the obstacle is any one of the one or more pedestrians based on the self-location data of each of the one or more portable devices, and notifying the portable device of the pedestrian who is the obstacle of the driving obstacle when determining that the obstacle is any one of the one or more pedestrians; 29. An automatic driving control device according to any one of appendixes 25 to 29.

[0193] (Appendix 31) A process of determining whether or not monitoring by the surrounding situation monitoring device is required by referring to self-location data indicating the location of a moving body traveling by automatic driving and map data indicating a monitoring area set as an area in which the surrounding situation of the moving body should be monitored by the surrounding situation monitoring device when the moving body passes; When it is determined that monitoring by the surrounding situation monitoring device is necessary, a process of transmitting a monitoring request to the surrounding situation monitoring device and transmitting driving situation data indicating a driving situation of the moving body to an automatic driving control device; A process of receiving control data notifying a driving obstacle of the moving body from the automatic driving control device, the control data being determined based on the driving situation data and monitoring data indicating the surrounding situation of the moving body obtained by the surrounding situation monitoring device; A process of controlling the travel of the moving object based on the control data and surrounding situation data indicating the surrounding situation of the moving object; An autonomous driving program that causes a computer to execute the above.

[0194] (Appendix 32) A process of receiving a monitoring request from a target moving object when the target moving object travels through a monitoring area by autonomous driving; A process of causing a group of monitoring sensors to monitor the surroundings of the target moving object and obtaining monitoring data indicating the surroundings of the target moving object; A process of transmitting the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle of the target moving body determined based on the monitoring data and driving situation data indicating the driving situation of the target moving body; A monitoring program that causes a computer to execute the following.

[0195] (Appendix 33) A process of receiving driving status data indicating a driving status of a target moving object from the target moving object that is driving autonomously; receiving monitoring data indicating a surrounding situation of the target moving object from a surrounding situation monitoring device that receives a monitoring request from the target moving object when the target moving object passes through a monitoring area and monitors the surrounding situation of the target moving object; A process of determining a traveling obstacle of the target moving object based on the traveling condition data and the monitoring data; A process of transmitting control data notifying the target moving object of the obstacle to the target moving object; A control program that causes a computer to execute the above.

[0196] (Appendix 34) An autonomous driving system including a target moving object that is an autonomous moving object that runs on the ground, an autonomous flying object that flies in the sky, and an autonomous driving control device, The target moving body is A group of mobile sensors; an in-vehicle device that transmits driving status data obtained by the moving body sensor group and indicating a driving status of the target moving body to the automatic driving control device; Equipped with The flying object is A monitoring sensor group; a monitoring control device that causes the monitoring sensor group to monitor the surrounding situation of the target moving object from the sky and transmits monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device; Equipped with The autonomous driving control device receives the driving status data and the monitoring data, determines a driving obstacle of the target moving object based on the driving status data and the monitoring data, and transmits control data notifying the target moving object of the driving obstacle, The vehicle-mounted device receives the control data and controls the travel of the target moving object based on the control data and surrounding situation data obtained by the moving object sensor group and indicating the surrounding situation of the target moving object. Autonomous driving system.

[0197] (Appendix 35) The target moving body is provided with a drone port on which the flying object takes off and lands. 34. The automated driving system described in claim 34.

[0198] (Appendix 36) the vehicle-mounted device transmits a monitoring request including monitoring area data indicating a monitoring area and self-location data obtained by the group of mobile body sensors and indicating a location of the target mobile body; The monitoring control device receives the monitoring request, plans a flight path based on the self-location data and the monitored area data from taking off from the drone port to flying over the monitored area and landing at the drone port, and flies the aircraft along the flight path.An autonomous driving system as described in Appendix 35.

[0199] (Appendix 37) An autonomous driving method using an autonomous driving system including a target moving body that is a moving body that runs on the ground by autonomous driving, an aircraft that flies in the sky by autonomous driving, and an autonomous driving control device, The target moving body is Transmitting driving status data indicating a driving status of the target moving object to the automatic driving control device; The flying object is Monitoring the surroundings of the target moving object from the sky, and transmitting monitoring data indicating the surroundings of the target moving object to the automatic driving control device; The automatic driving control device includes: receiving the driving condition data and the monitoring data, determining a driving obstacle for the target moving object based on the driving condition data and the monitoring data, and transmitting control data notifying the target moving object of the driving obstacle; The target moving body is The control data is received, and the target moving body travels in an automatic driving manner based on the control data and surrounding situation data indicating the surrounding situation of the target moving body. Autonomous driving method.

[0200] (Appendix 38) It is a vehicle that travels on the ground autonomously. A group of mobile sensors; an on-board device that transmits driving status data obtained by the group of mobile body sensors and indicating the driving status of the mobile body to an autonomous driving control device, receives control data from the autonomous driving control device notifying the mobile body of a driving obstacle determined based on the driving status data and monitoring data obtained by an aircraft flying overhead by autonomous driving and indicating the surrounding conditions of the mobile body, and controls the driving of the mobile body based on the control data and surrounding condition data obtained by the group of mobile body sensors and indicating the surrounding conditions of the mobile body; A moving body comprising:

[0201] (Appendix 39) The drone port on which the aircraft takes off and lands is provided. 39. The mobile object according to claim 38.

[0202] (Appendix 40) It is an autonomous flying vehicle that flies through the sky. A monitoring sensor group; a monitoring control device that causes the monitoring sensor group to monitor from the sky the surrounding conditions of a target moving object traveling on the ground in an autonomous driving manner, to obtain monitoring data indicating the surrounding conditions of the target moving object, and transmits the monitoring data to an autonomous driving control device that notifies the target moving object of a traveling obstacle of the target moving object determined based on the monitoring data and traveling condition data indicating the traveling condition of the target moving object; An aircraft equipped with the above.

[0203] (Appendix 41) The target moving object includes a drone port; The aircraft takes off and lands using the drone port. 41. The air vehicle described in Appendix 40.

[0204] (Appendix 42) The monitoring control device receives a monitoring request including monitoring area data indicating a monitoring area and self-location data indicating the position of the target moving object, and plans a flight path from taking off from the drone port to flying over the monitoring area and landing at the drone port based on the self-location data and the monitoring area data, and causes the flying object to fly along the flight path. 42. The air vehicle described in Appendix 41.

[0205] (Appendix 43) Receiving driving status data indicating a driving status of a target moving object from the target moving object traveling on the ground by autonomous driving; receiving monitoring data indicating a surrounding situation of the target moving object from a surrounding situation monitoring device that flies above the target moving object in an autonomous manner and monitors a surrounding situation of the target moving object; determining a traveling obstacle of the target moving object based on the traveling condition data and the monitoring data; Transmitting control data notifying the target moving body of the road obstruction to the target moving body. Autonomous driving control device.

[0206] (Appendix 44) A process of transmitting driving status data indicating a driving status of a moving object traveling on the ground by autonomous driving to an autonomous driving control device; A process of receiving control data notifying a driving obstacle of the moving body from the automatic driving control device, the control data being determined based on the driving status data and monitoring data indicating the surrounding conditions of the moving body obtained by an aircraft flying in the sky by automatic driving; A process of controlling the travel of the moving object based on the control data and surrounding situation data indicating the surrounding situation of the moving object; An autonomous driving program that causes a computer to execute the above.

[0207] (Appendix 45) A process of having a group of monitoring sensors mounted on an aircraft flying autonomously in the sky monitor the surroundings of a target moving object traveling autonomously on the ground, thereby obtaining monitoring data indicating the surroundings of the target moving object; A process of transmitting the monitoring data to an autonomous driving control device that notifies the target moving body of a driving obstacle of the target moving body determined based on the monitoring data and driving situation data indicating the driving situation of the target moving body; A monitoring program that causes a computer to execute the following.

[0208] (Appendix 46) A process of receiving driving status data indicating a driving status of a target moving object from a target moving object traveling on the ground by autonomous driving; A process of receiving monitoring data indicating a surrounding situation of the target moving object from a surrounding situation monitoring device that flies above the target moving object in an autonomous driving manner and monitors the surrounding situation of the target moving object; A process of determining a traveling obstacle of the target moving object based on the traveling condition data and the monitoring data; A process of transmitting control data notifying the target moving object of the obstacle to the target moving object; A control program that causes a computer to execute the above. [Explanation of symbols]

[0209] 100 Autonomous driving system, 101 Moving body, 102 Mobile terminal, 110 Map database, 111 Map data, 120 Target moving body, 121 Drone port, 130 Surrounding situation monitoring device, 131 Drone, 200 Vehicle-mounted device, 201 Processor, 202 Memory, 203 Auxiliary storage device, 204 Communication device, 205 Input / output interface, 211 Monitoring request unit, 212 Control data acquisition unit, 213 Driving control unit, 220 Memory unit, 290 Moving body sensor group, 291 Depth camera, 292 Lider, 293 Positioning sensor, 300 Monitoring control device, 301 Processor, 302 Memory, 303 Auxiliary storage device, 304 Communication device, 305 Input / output interface, 311 Request reception unit, 312 Monitoring control unit, 313 Monitoring data generation unit, 314 Monitoring data provision unit, 320 Memory unit, 390 monitoring sensor group, 391 depth camera, 392 RGB camera, 393 color sensor, 400 autonomous driving control device, 401 processor, 402 memory, 403 auxiliary storage device, 404 communication device, 405 input / output interface, 411 data reception unit, 412 driving obstacle determination unit, 413 control data generation unit, 414 control data provision unit, 415 driving obstacle notification unit, 420 memory unit, 431 switcher unit, 432 object detection unit, 433 position estimation unit, 434 annotation unit, 435 learning unit, 438 database, 439 video stream AI processing pipeline.

Claims

1. An autonomous driving system including a target moving body that is an autonomous driving moving body, a surrounding situation monitoring device, and an autonomous driving control device, The target moving body is A group of mobile sensors; an on-board device that determines whether the target moving object has entered the monitoring area by referring to self-location data obtained by the group of moving object sensors and indicating the position of the target moving object, and map data indicating a monitoring area set as an area in which the surrounding conditions of the target moving object should be monitored by the surrounding conditions monitoring device when the target moving object passes through, and that transmits a monitoring request to the surrounding conditions monitoring device when it is determined that the target moving object has entered the monitoring area, and transmits driving condition data obtained by the group of moving object sensors and indicating the driving conditions of the target moving object to the automatic driving control device; Equipped with The surrounding situation monitoring device is A monitoring sensor group; a monitoring control device that receives the monitoring request, causes the monitoring sensor group to monitor the surrounding situation of the target moving object, and transmits monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device; Equipped with The autonomous driving control device receives the driving status data and the monitoring data, determines a driving obstacle of the target moving body based on the driving status data and the monitoring data, and transmits control data notifying the target moving body of the driving obstacle, The vehicle-mounted device receives the control data and controls the travel of the target moving object based on the control data and surrounding situation data obtained by the moving object sensor group and indicating the surrounding situation of the target moving object. Autonomous driving system.

2. The surrounding situation monitoring device is an autonomous flying vehicle. The automated driving system according to claim 1 .

3. The target moving body is provided with a drone port on which the flying object takes off and lands. The automated driving system according to claim 2.

4. the vehicle-mounted device transmits the monitoring request including the self-location data and monitoring area data indicating the monitoring area; The autonomous driving system of claim 3, wherein the monitoring control device receives the monitoring request, plans a flight path based on the self-location data and the monitored area data from taking off from the drone port to flying over the monitored area and landing at the drone port, and causes the aircraft to fly along the flight path.

5. The surrounding situation monitoring device is a roadside unit installed in the monitoring area. The automated driving system according to claim 1 .

6. the monitoring sensor group includes a color sensor; The monitoring and control device includes: Detecting objects present around the target moving object and a plane on which the objects are located based on color sensor data obtained by the color sensor; A position of the object on the detected plane is estimated, and data indicating the positions of each object present around the target moving object is generated as the monitoring data. The automated driving system according to any one of claims 1 to 5.

7. The color sensor is an RGB camera or an RGB-D camera. The automated driving system according to claim 6.

8. The autonomous driving control device determines that the driving obstacle exists when an obstacle facing the target moving object exists within a proximity range from the position of the target moving object. The automated driving system according to claim 1 .

9. When the obstacle is a moving object, the autonomous driving control device transmits the control data indicating that passing is impossible as the type of the driving obstacle to the target moving body, When the type of the driving obstacle is the non-passing obstacle, the vehicle-mounted device causes the target moving object to retreat to an escape zone indicated in the map data. The automated driving system according to claim 8.

10. When the obstacle is a stationary object, the autonomous driving control device transmits the control data indicating impassability as the type of the driving obstacle to the target moving body, When the type of the obstacle is the impassable obstacle, the vehicle-mounted device makes the target moving object turn around by utilizing an escape zone shown in the map data. The automated driving system according to claim 8 or 9.

11. When a plurality of obstacles are present, the autonomous driving control device transmits to the target moving body the control data indicating a need to slow down as a type of the driving obstacle; The vehicle-mounted device makes the target moving object slow down when the type of the driving obstacle requires the target moving object to slow down. The automated driving system according to claim 8 or 9.

12. The autonomous driving control device receives self-position data from each of one or more moving bodies, A determination is made as to whether the obstacle is any one of the one or more moving bodies based on the self-position data of each of the one or more moving bodies, and when the determination is made that the obstacle is any one of the one or more moving bodies, the traveling obstacle is notified to the moving body that is the obstacle. The automated driving system according to claim 8 or 9.

13. The autonomous driving control device receives self-location data from one or more mobile devices carried by one or more pedestrians, determines whether the obstacle is any one of the one or more pedestrians based on the self-location data of each of the one or more mobile devices, and notifies the mobile device of the pedestrian who is the obstacle of the driving obstacle when it determines that the obstacle is any one of the one or more pedestrians. The automated driving system according to claim 8 or 9.

14. The autonomous driving system includes a plurality of the surrounding situation monitoring devices, The group of monitoring sensors of each of the surrounding situation monitoring devices includes a video image sensor, The monitoring control device of each of the surrounding situation monitoring devices transmits the monitoring data including the video stream obtained by the video sensor to the autonomous driving control device, The automatic driving control device includes: For each of the video streams, an object detection model corresponding to the surrounding situation monitoring device that is a transmission source among a plurality of types of object detection models is used to detect an object by inputting the video stream, and detection information of the detected object is recorded in a database; displaying the video stream on a display, receiving annotation information input to the autonomous driving control device, correcting detection information of an object detected by inputting the video stream using a selection model that is an object detection model selected from the plurality of types of object detection models based on the annotation information, learning the video stream and the corrected detection information to update the selection model and generate a new version of the object detection model; The video stream is input and the object detection is performed using a decision model, which is an object detection model designated for the autonomous driving control device, among the plurality of types of object detection models by version, to detect an obstacle to the travel of the target moving object. The automated driving system according to claim 1 .

15. An autonomous driving method using an autonomous driving system including a target moving body that is an autonomous driving moving body, a surrounding situation monitoring device, and an autonomous driving control device, The target moving body is a monitoring device that monitors the surrounding conditions of the target moving object when the target moving object passes through the target moving object and transmits a monitoring request to the surrounding conditions monitoring device and transmits driving condition data indicating the driving condition of the target moving object to the automatic driving control device; The surrounding situation monitoring device is receiving the monitoring request, monitoring a surrounding situation of the target moving object, and transmitting monitoring data indicating the surrounding situation of the target moving object to the autonomous driving control device; The automatic driving control device includes: receiving the driving condition data and the monitoring data, determining a driving obstacle for the target moving object based on the driving condition data and the monitoring data, and transmitting control data notifying the target moving object of the driving obstacle; The target moving body is The control data is received, and the target moving body travels in an automatic driving manner based on the control data and surrounding situation data indicating the surrounding situation of the target moving body. Autonomous driving method.

16. It is a vehicle that runs autonomously, A group of mobile sensors; an on-board device that determines whether the moving body has entered the monitoring area by referring to self-location data obtained by the moving body sensor group and indicating the position of the moving body, and map data indicating a monitoring area set as an area in which the surrounding conditions of the moving body should be monitored by a surrounding conditions monitoring device when the moving body passes through, and when it is determined that the moving body has entered the monitoring area, transmits a monitoring request to the surrounding conditions monitoring device, and transmits driving condition data obtained by the moving body sensor group and indicating the driving condition of the moving body to an automatic driving control device, receives control data from the automatic driving control device that notifies of a driving obstacle of the moving body determined based on the driving condition data and the monitoring data obtained by the surrounding conditions monitoring device and indicating the surrounding conditions of the moving body, and controls the driving of the moving body based on the control data and the surrounding conditions data obtained by the moving body sensor group and indicating the surrounding conditions of the moving body; A mobile body comprising:

17. The surrounding situation monitoring device is an autonomous flying object, The mobile body includes a drone port on which the flying object takes off and lands. The moving body according to claim 16.

18. A process of referring to self-location data indicating the location of a moving body traveling by autonomous driving and map data indicating a monitoring area set as an area in which the surroundings of the moving body should be monitored by a surroundings monitoring device when the moving body passes through, and determining whether the moving body has entered the monitoring area; When it is determined that the moving object has entered the monitoring area, a process of transmitting a monitoring request to the surrounding situation monitoring device and transmitting driving situation data indicating a driving situation of the moving object to an automatic driving control device; A process of receiving control data notifying a driving obstacle of the moving body from the automatic driving control device, the control data being determined based on the driving situation data and monitoring data indicating the surrounding situation of the moving body obtained by the surrounding situation monitoring device; A process of controlling the travel of the moving object based on the control data and surrounding situation data indicating the surrounding situation of the moving object; An autonomous driving program that causes a computer to execute the above.

Images