Mobile body autonomous travel assistance system

Abstract

The present invention comprises: a sensor device 10 that has a first sensor unit 10a disposed on the environment side of a real space; an edge computer 11 serving as a server device that is connected to the sensor device 10 in a communicable state; and a mobile body 2 that is connected to the edge computer 11 in a communicable state and has a second sensor unit 20. On the basis of first image sensor data composed of a point cloud in the real space acquired by the first sensor unit 10a of the sensor device 10 and second image sensor data composed of a point cloud in the real space acquired by the second sensor unit 20 of the mobile body 2, the mobile body 2 identifies the host position of the mobile body 2 in the real space and recognizes obstacles around the mobile body 2, whereby a travel route for the mobile body 2 is generated.

Description

Autonomous driving support system for mobile vehicles

[0001] The present invention relates to a mobile object autonomous driving support system that supports autonomous driving of a mobile object in real space.

[0002] In recent years, personal mobility vehicles (PMVs) have been expected to provide solutions to transportation needs such as flexible last-mile travel, reducing traffic and carbon emissions, providing exercise and recreation, delivering services, mobility services for residents and tourists, and enabling door-to-door travel.

[0003] Meanwhile, digital twins, which are digital replicas of real-world entities, are gaining attention. Digital twin technology allows physical systems to update their real-time behavior according to feedback generated by the digital twin. The concept of a digital twin framework needs to be discussed not only for cars and trucks, but also for PMVs.

[0004] One of the most important elements of a PMV digital twin is a real-time monitoring system that replicates physical entities in the digital world, and the use of LIDAR (Light-Detection-and-Ranging) technology as a sensor for this purpose is a promising means of collecting spatial information. Data collected by LIDAR can identify the positions of not only PMVs but also other moving objects such as cars and trucks, as well as pedestrians (see Non-Patent Document 1). In particular, the use of multiple LIDAR sensors has been studied because it can reduce blind spots in the monitoring area and increase the amount of information in overlapping areas (see, for example, Non-Patent Document 2).

[0005] G. Bhatti, H. Mohan, and RR Singh, “Towards the future of smart electric vehicles: Digital twin technology,” Renewable and Sustainable Energy Reviews, vol. 141, p. 110801, 2021. C. Li, R. Shinkuma, T. Sato, and E. Oki, ``Real-time Data Selection and Merging for 3D-image Sensing Network with Multiple Sensors,” IEEE Sensors Journal, Vol.21(19), pp.22058-22076, Aug. 2021.

[0006] However, previous research has not addressed safety issues when onboard sensors are disabled. In particular, onboard sensors in PMVs can easily be disabled due to external hardware issues such as dirt or snow, internal hardware issues, or software issues such as malware infection. Such issues could lead to localization failures, a fatal error in autonomous driving. Therefore, PMVs must ensure autonomous driving, enabling the vehicle to reach a safe location even if their onboard sensors are disabled. Such issues can occur not only in PMVs, but also in other moving objects, such as cars, trucks, and electric wheelchairs, that operate autonomously in real space.

[0007] The present invention has been made in consideration of the above-mentioned technical background, and aims to provide a mobile object autonomous driving support system that can support safe autonomous driving of a mobile object in real space.

[0008] In order to achieve the above-mentioned object, the present invention provides a mobile body autonomous driving assistance system that supports the autonomous driving of the mobile body in real space, comprising a sensor device arranged in an environment of real space, a server device connected to the sensor device in a state capable of communicating with the sensor device, and the mobile body connected to the server device in a state capable of communicating with the sensor device, wherein the sensor device comprises a first sensor unit that acquires first image sensor data consisting of a point cloud in real space, the mobile body comprises a second sensor unit that acquires second image sensor data consisting of a point cloud in real space, and the server device or the mobile body comprises a mobile body position identification unit that identifies the position of the mobile body in real space based on the first image sensor data consisting of the point cloud in real space acquired by the first sensor unit of the sensor device and the second image sensor data consisting of the point cloud in real space acquired by the second sensor unit of the mobile body.

[0009] The moving body position determination unit may also include an estimation unit that estimates the self-position of the moving body based on second image sensor data consisting of a point cloud in real space acquired by a second sensor unit of the moving body, and a correction unit that corrects the self-position of the moving body estimated by the estimation unit based on first image sensor data consisting of a point cloud in real space acquired by a first sensor unit of the sensor device.

[0010] The server device or the mobile body may also be provided with an obstacle recognition unit that recognizes obstacles in real space based on first image sensor data consisting of a point cloud in real space acquired by a first sensor unit of the sensor device and second image sensor data consisting of a point cloud in real space acquired by a second sensor unit of the mobile body.

[0011] The obstacle recognition unit may also include a detection unit that detects obstacles around the moving body, a tracking unit that captures the trajectory of the obstacle based on the detection result of the obstacle by the detection unit, and a prediction unit that predicts the position of the obstacle after a predetermined time has passed based on the trajectory of the obstacle captured by the tracking unit.

[0012] The server device or the moving body may also include a path generation unit that generates a travel path for the moving body in real space based on the position of the moving body in real space identified by the moving body position identification unit and the position of an obstacle in real space identified by the obstacle recognition unit.

[0013] Furthermore, the server device or the mobile body may include a selection unit that selects image sensor data consisting of a point cloud in real space, either first image sensor data consisting of a point cloud in real space acquired by a first sensor unit of the sensor device or second image sensor data consisting of a point cloud in real space acquired by a second sensor unit of the mobile body, and the mobile body position identification unit may identify the self-position of the mobile body in real space based on the first image sensor data or the second image sensor data consisting of a point cloud in real space selected by the selection unit.

[0014] In addition, the selection unit may preferentially select second image sensor data consisting of a point cloud in real space acquired by a second sensor unit of the moving body, and if it determines that the second image sensor data consisting of a point cloud in real space is invalid, it may select first image sensor data consisting of a point cloud in real space acquired by a first sensor unit of the sensor device.

[0015] The moving body may also include a control unit that controls the traveling of the moving body based on the traveling path of the moving body in real space generated by the path generation unit.

[0016] The server device may also include an aggregation unit that aggregates first image sensor data made up of a point cloud in real space acquired by the first sensor unit.

[0017] The server device may also include a map generation unit that generates a map of the real space based on first image sensor data consisting of a cloud of points in the real space aggregated by the aggregation unit, and the moving body position identification unit may use information regarding the map of the real space generated by the map generation unit when identifying the position of the moving body.

[0018] The server device may also include a digital twin generation unit that generates a digital twin corresponding to the real space based on first image sensor data consisting of a point cloud in the real space aggregated by the aggregation unit, and the moving body position identification unit may use information regarding the digital twin corresponding to the real space generated by the digital twin generation unit when identifying the position of the moving body.

[0019] In addition, the digital twin generation unit may detect the moving body in real space using a learning model that has been machine-learned in advance to detect the moving body based on first image sensor data consisting of a point cloud in real space.

[0020] According to the present invention, the position of a mobile object in real space is determined based on first image sensor data consisting of a point cloud in real space acquired by a first sensor unit of an environment-side sensor device and second image sensor data consisting of a point cloud in real space acquired by a second sensor unit of the mobile object, thereby enabling the position of the mobile object in real space to be determined with high accuracy. Moreover, even if the second sensor unit arranged on the mobile object is disabled due to an external hardware problem such as dirt or snow, an internal hardware problem, or a software problem such as malware infection, the first image sensor data consisting of a point cloud in real space acquired by the first sensor unit arranged on the environment side of the real space can be used to determine the mobile object's self-position and recognize obstacles, thereby enabling the mobile object to continuously generate a travel path and enable safe autonomous travel of the mobile object.

[0021] 1 is a diagram showing the overall configuration of a mobile object autonomous driving support system according to an embodiment of the present invention. FIG. 1 is a diagram showing the configuration of the edge computing system of FIG. 1. FIG. 2 is a diagram showing the configuration of the mobile object of FIG. 1. FIG. 3 is a diagram showing first image sensor data consisting of a point cloud in real space. FIG. 4 is a diagram showing a digital twin (including detection results of the mobile object) corresponding to real space. FIG. 5 is a diagram showing second image sensor data consisting of a point cloud in real space. FIG. 6 is a diagram showing simulation results of the estimated position and corrected position of the mobile object. FIG. 7 is a diagram showing a state in which obstacles around the mobile object are recognized. FIG. 8 is a diagram showing a state in which a driving path of the mobile object has been generated. FIG. 9 is a diagram showing the configuration of a mobile object of a mobile object autonomous driving support system according to a second embodiment of the present invention. FIG. 10 is a diagram showing the configuration of a mobile object of a mobile object autonomous driving support system according to a third embodiment of the present invention. FIG. 11 is a diagram showing the cumulative distribution of processing time when processing required for self-localization in the present invention is performed by (a) one edge computer and (b) four edge computers. FIG. 12 is a diagram showing the path of a robot in benchmark results for (a) normal scenario and (b) abnormal scenario according to Example 1. FIG. 13 is a diagram showing the path of a robot in results of the present invention in (a) normal scenario and (b) abnormal scenario according to Example 1. FIG. 14 is a diagram showing the DTW distance between the reference point and passing point of the robot in each trial according to Example 1. FIG. 10 is a diagram showing the results when one micromobility vehicle is made to travel autonomously on a simulator according to Example 2. FIG. 11 is a diagram showing the results when ten micromobility vehicles are made to travel autonomously on a simulator according to Example 3. FIG. 12 is a diagram showing the results when ten micromobility vehicles are made to travel autonomously on a simulator according to Example 4. FIG. 13 is a diagram showing the results when one micromobility vehicle is made to travel autonomously in a real environment and ten micromobility vehicles are made to travel autonomously on a simulator according to Example 5. FIG. 14 is a diagram showing the travel trajectories of each micromobility vehicle when a dynamic load balancing method using multiple system statistics is used.

[0022] First Embodiment Next, a first embodiment of a mobile object autonomous driving assistance system according to the present invention will be described with reference to FIGS. 1 to 9. FIG.

[0023] As shown in Fig. 1, this system is composed of an edge computing system 1 and an autonomous mobile object 2, and the edge computing system 1 and the mobile object 2 are connected in a communicable state via a wireless network. The specific configurations of the edge computing system 1 and the mobile object 2 are described below. The mobile object 2 refers to a vehicle that can travel autonomously, such as a PMV (Personal Mobility Vehicle), automobile, truck, electric wheelchair, or delivery robot.

[0024] [Configuration of Edge Computing System 1] As shown in FIG. 2 , the edge computing system 1 includes a plurality of sensor devices 10 arranged in the environment side of the real space, and an edge computer 11 as a server device connected to each of the sensor devices 10 in a communicable state via a local network.

[0025] The sensor device 10 has a first sensor unit 10a placed on a static object such as a pillar in the real-space environment, and after acquiring first image sensor data consisting of a point cloud in the entire real space including the moving object and its surroundings, performs predetermined processing on the first image sensor data and transmits it to the edge computer 11.

[0026] The first sensor unit 10a is, for example, a sensor called LiDAR (light detection and ranging). LiDAR is a type of sensor that uses laser light, and by scanning and irradiating an object with laser light having a higher radiant flux density than radio waves and a short wavelength, it acquires image sensor data consisting of a point cloud in real space and accurately detects not only the distance to the object but also the position and shape of the object.

[0027] Examples of this LiDAR include a method of acquiring image sensor data in all directions of 360 degrees by rotating light emitters that emit multiple laser beams, and a method of acquiring image sensor data by directly irradiating laser beams within a predetermined light irradiation angle range. Furthermore, the more light emitters that emit laser beams, the higher the accuracy of the image sensor data, but the higher the cost, so an inexpensive LiDAR with a small number of light emitters may be used.

[0028] In addition, when transmitting the first image sensor data consisting of a point cloud in real space acquired by the first sensor unit 10a to the edge computer 11, the sensor device 10 may perform filtering on the first image sensor data or fragment the data into multiple packets.

[0029] 2 , the edge computer 11 includes an aggregation unit 111 that aggregates first image sensor data consisting of a point cloud in real space transmitted from the first sensor unit 10a of the sensor device 10, a map generation unit 112 that generates a map of the real space based on the first image sensor data consisting of the point cloud in real space, a digital twin generation unit 113 that generates a digital twin corresponding to the real space based on the first image sensor data consisting of the point cloud in real space, an information storage unit 114 that stores information about the map (map data) and information about the digital twin (digital twin data), and a transmission unit 115 that transmits information about the map (map data) and information about the digital twin (digital twin data) to the mobile object 2. Note that a digital twin refers to a digital reproduction of the real-space environment in cyberspace obtained by utilizing sensors such as LIDAR to acquire information about the real space.

[0030] The aggregation unit 111 aggregates first image sensor data consisting of a point cloud in real space acquired by the first sensor unit 10a of each sensor device 10. Specifically, the aggregation unit 111 aggregates each frame of the first image sensor data consisting of a point cloud in real space by chronologically synthesizing the frames based on timestamps and aligning them in three-dimensional space. Note that when aggregating the image sensor data consisting of a point cloud in real space, the aggregation unit 111 may perform other processing such as smoothing of the point cloud of the first image sensor data.

[0031] The map generation unit 112 generates a two-dimensional or three-dimensional map of the real space based on the first image sensor data consisting of a point cloud in the real space aggregated by the aggregation unit 111, and stores information about the map (map data) in the information storage unit 114. For example, Fig. 4 shows an example of map data obtained by removing the ceiling and floor from the first image sensor data consisting of a point cloud in the real space (a predetermined indoor space for the experiment), converting the xyz coordinates into xy coordinates, and then converting the processed point cloud into a format that can be used for autonomous driving by a ROS (Robot Operating System).

[0032] The digital twin generation unit 113 generates a three-dimensional digital twin corresponding to the real space based on the first image sensor data consisting of a point cloud in the real space aggregated by the aggregation unit 111, and stores information about the digital twin (digital twin data) in the information storage unit 114. When generating a three-dimensional digital twin corresponding to the real space, the digital twin generation unit 113 detects a moving object 2 in the real space, and also includes information about the position, orientation (travel direction), speed, etc. of the moving object 2 in the information about the digital twin (digital twin data). For example, FIG. 5 shows information (digital twin data) about a three-dimensional digital twin corresponding to the real space (a specified indoor space for experimentation) generated based on the first image sensor data consisting of a point cloud in the real space, in which each moving object 2 detected by the digital twin generation unit 113 is surrounded by a box.

[0033] The digital twin generation unit 113 may detect the moving body 2 in real space (or a digital twin corresponding to real space) using a learning model (AI: Artificial Intelligence) that has been machine-learned to detect in advance the position, orientation, speed, etc. of the moving body 2 based on first image sensor data consisting of a point cloud in real space. In this case, the digital twin generation unit 113 detects and tracks the three-dimensional posture (position, orientation, speed) of the moving body 2 from information related to the digital twin (digital twin data).

[0034] The information storage unit 114 stores information (map data) about the map of the real space generated by the map generation unit 112, and information (digital twin data) about the digital twin corresponding to the real space generated by the digital twin generation unit 113.

[0035] The transmitting unit 115 transmits information (map data) about the map of the real space stored in the information storage unit 114 and information (digital twin data) about the digital twin corresponding to the real space to the corresponding mobile body 2 via a wireless network.

[0036] [Configuration of Mobile Body 2] As shown in FIG. 3 , the mobile body 2 includes one or more second sensor units 20 that acquire image sensor data consisting of a point cloud in real space, a first receiving unit 21 that receives information (map data) related to the map of real space and information (digital twin data) related to the digital twin transmitted from the edge computer 11, a second receiving unit 22 that receives second image sensor data consisting of the point cloud in real space acquired by the second sensor unit 20, an on-board computer 23 mounted inside the mobile body 2, and a drive mechanism 24 that drives the mobile body 2.

[0037] The second sensor unit 20 is disposed on the surface of the moving body 2, acquires second image sensor data consisting of a point cloud in real space around the moving body 2, performs predetermined processing on the second image sensor data, and transmits the data to the on-board computer 23. Similar to the first sensor unit 10a disposed on the environment side of real space, a so-called LiDAR (light detection and ranging) sensor is preferably used as the second sensor unit 20. For example, Fig. 6 shows an example of second image sensor data consisting of a point cloud in real space acquired by the second sensor unit 20 disposed on the moving body 2.

[0038] The on-board computer 23 includes a mobile body position identification unit 231 that identifies the self-position of the mobile body 2, an obstacle recognition unit 232 that recognizes obstacles around the mobile body 2 in real space, a route generation unit 233 that generates a travel route for the mobile body 2, and a control unit 234 that controls the motor and various devices of the mobile body 2.

[0039] The moving object position specifying unit 231 includes an estimation unit 231a that estimates the self-position of the moving object 2, and a correction unit 231b that corrects the self-position of the moving object 2 whose position has been estimated by the estimation unit 231a.

[0040] The estimation unit 231a estimates the position of the moving object 2 on the map of real space based on information (map data) related to the map of real space received by the first receiving unit 21 and second image sensor data consisting of a point cloud received by the second receiving unit 22. Note that the information (map data) related to the map of real space received by the first receiving unit 21 is generated in the edge computer 11 based on the first image sensor data consisting of a point cloud in real space, as described above. Note that, although the present embodiment uses map data generated in the edge computer 11 based on the first image sensor data, existing map data generated by another device may also be used.

[0041] The correction unit 231b corrects the position of the moving body 2 on a map of real space by comparing the detected position of the moving body 2 included in information (digital twin data) about the digital twin corresponding to the real space received by the first receiving unit 21 with the position of the moving body 2 estimated by the estimation unit 231a. Note that the correction unit 231b also processes measurement values ​​received with a delay from the edge computer 11 in order to make corrections taking into account communication delays of data from the edge computer 11 to the moving body 2. For example, FIG. 7 shows the estimated position of the moving body 2 by the estimation unit 231a and the corrected position of the moving body 2 by the correction unit 231b.

[0042] Thus, the information (digital twin data) related to the digital twin corresponding to the real space received by the first receiving unit 21 is generated using first image sensor data consisting of a point cloud in the real space acquired by the first sensor unit 10a of the sensor device 10 arranged on the environment side in the real space, while the position of the moving body 2 estimated by the estimation unit 231a is generated using second image sensor data consisting of a point cloud in the real space acquired by the second sensor unit 20 arranged on the moving body 2 in the real space. Therefore, the moving body position identification unit 231 can accurately identify the position of the moving body 2 in the real space based on two different image sensor data: the first image sensor data consisting of a point cloud in the real space acquired by the first sensor unit 10a of each sensor device 10 arranged on the environment side in the real space, and the second image sensor data arranged on the moving body 2 and consisting of a point cloud in the real space.

[0043] The mobile object position identification unit 231 may, for example, predict and correct the three-dimensional attitude (position, orientation, and velocity) of the mobile object 2 using an extended Kalman filter (EKH). This scheme saves a history of the estimated values ​​and measured values ​​of the extended Kalman filter, and if the timestamp embedded in the data from the edge computer 11 is older than the current time, it returns to the last state before the timestamp. In this scheme, all measured values ​​up to the current time are processed, and the estimated values ​​are recalculated.

[0044] The obstacle recognition unit 232 recognizes obstacles such as pillars, walls, other moving bodies 2, pedestrians, etc. in the real space based on information (map data) relating to the map of the real space received by the first receiving unit 21 and second image sensor data consisting of a point cloud in the real space received by the second receiving unit 22, and is composed of a detection unit 232a, a tracking unit 232b, and a prediction unit 232c.

[0045] The detection unit 232a detects the position and size of obstacles around the moving object 2 for each frame of second image sensor data consisting of a point cloud in real space.

[0046] The tracking unit 232b captures the trajectory of the moving obstacle 2 (e.g., another moving object 2 or a pedestrian) by stitching together the obstacle detection results of the detection unit 232a across multiple frames of the second image sensor data consisting of a point cloud in real space.

[0047] The prediction unit 232c predicts the trajectory (position) of the obstacle a predetermined time (for example, several seconds) after the current time based on the trajectory of the obstacle movement captured by the tracking unit 232b.

[0048] 8 shows an image of an obstacle recognized by the obstacle recognition unit 232. Obstacles such as other moving bodies 2 and pedestrians recognized by the obstacle recognition unit 232 are enclosed in boxes.

[0049] The path generation unit 233 generates a travel path for the mobile body 2 on a map of real space based on information about the position of the mobile body 2 in real space identified by the mobile body position identification unit 231 and information about obstacles in real space recognized by the obstacle recognition unit 232. Specifically, when a static obstacle (current position) or a dynamic obstacle (predicted position after a predetermined time has elapsed) is recognized for the mobile body 2 in real space, the path generation unit 233 generates a travel path for the mobile body 2 so as to avoid these static and dynamic obstacles. For example, Fig. 9 is a diagram showing the travel path for the mobile body 2 generated by the path generation unit 233, and shows the travel path for the mobile body 2 on a map of real space.

[0050] The control unit 234 controls the drive mechanism 24 such as a motor based on information about the travel route of the moving body 2 generated by the route generation unit 233.

[0051] Second Embodiment Next, a second embodiment of a mobile object autonomous driving assistance system according to the present invention will be described with reference to Fig. 10. Note that only configurations different from the above embodiment will be described below, and descriptions of the same configurations will be omitted and the same reference numerals will be used.

[0052] In this embodiment, a selection unit 235 is provided on the output side of the first receiving unit 21 and the second receiving unit 22 in the on-board computer 23 of the mobile body 2, and on the input side of the mobile body position identification unit 231 and the obstacle recognition unit 232.

[0053] The selection unit 235 selects either the information (digital twin data) related to the digital twin corresponding to the real space received by the first receiving unit 21, or the second image sensor data consisting of a point cloud in the real space received by the second receiving unit 22. In other words, the information (digital twin data) related to the digital twin corresponding to the real space received by the first receiving unit 21 is generated based on the first image sensor acquired by the first sensor unit 10a arranged on the environment side of the real space, and therefore the selection unit 235 selects image sensor data consisting of a point cloud in the real space from either the first image sensor data consisting of a point cloud in the real space acquired by the first sensor unit 10a, or the second image sensor data consisting of a point cloud in the real space acquired by the second sensor unit 20 of the moving object 2.

[0054] In addition, the selection unit 235 preferentially selects the second image sensor data when the second receiving unit 22 is able to effectively receive the second image sensor data consisting of a point cloud in real space, but when it determines that the second image sensor data consisting of a point cloud in real space is invalid, it selects the information about the digital twin received by the first receiving unit 21 (digital twin data), that is, the first image sensor data consisting of a point cloud in real space acquired by the first sensor unit 10a of each sensor device 10.

[0055] The timing of disabling the second sensor unit 20 is detected, for example, by measuring the time interval between data arrivals. If the data acquired by the selector 235 is from the second receiver 22 (i.e., the second sensor unit 20), the selector 235 records a timestamp indicating the time at which the data was acquired. On the other hand, if the data acquired by the selector 235 is from the first receiver 21 (i.e., the edge computer 11), the selector 235 refers to the timestamp (indicating the time at which the last packet was transmitted by any of the first sensor units 10a) recorded in the data by the edge computer 11. If the difference between the recorded timestamp and the current time is within a predetermined threshold, the selection unit 235 continues to drop the data from the first receiving unit 21 (data from the edge computer 11) and waits for data from the second receiving unit 22 (data from the second sensor unit 20), while if the difference between the recorded timestamp and the current time exceeds a predetermined threshold, the selection unit 235 determines that the second sensor unit 20 has been disabled, and selects and starts transferring the data from the first receiving unit 21 (data from the edge computer 11). Note that if the selection unit 235 selects the first image sensor data consisting of a point cloud in real space, the correction unit 231b does not need to correct the position of the moving object 2 based on information about the digital twin (digital twin data).

[0056] According to this, even if the second sensor unit 20 arranged on the moving body 2 is disabled due to an external hardware problem such as dirt or snow, an internal hardware problem, or a software problem such as a malware infection, the first image sensor data consisting of a point cloud in real space acquired by the first sensor unit 10a arranged on the real space environment side can be used to identify the moving body's own position and recognize obstacles, so that the moving body 2's travel path can be continuously generated, enabling the moving body 2 to travel safely autonomously.

[0057] Third Embodiment Next, a third embodiment of a mobile object autonomous driving assistance system according to the present invention will be described with reference to Fig. 11. Note that only configurations different from the above-described embodiments will be described below, and the same configurations will be denoted by the same reference numerals and will not be described again.

[0058] In this embodiment, as shown in FIG. 11 , a mobile object position identification unit 231, an obstacle recognition unit 232, and a path generation unit 233 are provided in the edge computer 11, and second image sensor data consisting of a point cloud in real space acquired by the second sensor unit 20 of the mobile object 2 is transmitted from the mobile object 2 to the edge computer 11 at any time.

[0059] The moving body position specifying unit 231 specifies the position of the moving body 2 based on information about a map of real space (map data) and information about a digital twin corresponding to the real space (digital twin data) stored in the information storage unit 114, and second image sensor data consisting of a point cloud in real space transmitted from the moving body 2. The method of specifying the self-position of the moving body 2 by the moving body position specifying unit 231 is the same as in the first embodiment.

[0060] The obstacle recognition unit 232 recognizes obstacles such as pillars, walls, other moving bodies 2, pedestrians, etc. in the real space based on information (map data) related to the map of the real space stored in the information storage unit 114 and second image sensor data consisting of a point cloud in the real space transmitted from the moving body 2. The method of recognizing obstacles by the obstacle recognition unit 232 is the same as in the first embodiment.

[0061] The route generation unit 233 generates a travel route for the mobile body 2 based on information regarding the mobile body 2's own position in real space identified by the mobile body position identification unit 231 and information regarding obstacles in real space recognized by the obstacle recognition unit 232.

[0062] The transmitting unit 115 transmits information about the travel route of the mobile object 2 generated by the route generating unit 233 to the mobile object 2 via a wireless network.

[0063] Thereafter, in the moving body 2 , the control unit 234 controls the drive mechanism 24 based on the information about the travel route of the moving body 2 transmitted from the edge computer 11 .

[0064] In the third embodiment, as in the second embodiment, a selection unit 235 may be provided on the data input side of the mobile object position identification unit 231 and the obstacle recognition unit 232. In this case, the selection unit 235 selects, as necessary, information about the digital twin (digital twin data) stored in the information storage unit 114 and second image sensor data consisting of a point cloud in real space transmitted from the mobile object 2. As a result, even if the second sensor unit 20 is disabled, the mobile object position identification unit 231 can identify its own position and the obstacle recognition unit 232 can recognize obstacles based on information about the digital twin (digital twin data), that is, the first image sensor data consisting of a point cloud in real space acquired by the first sensor unit 10a arranged on the environment side of the real space. This makes it possible to continuously generate a path for the mobile object 2 and enable safe autonomous driving of the mobile object 2.

[0065] In the above embodiment, all of the functions of the mobile body position identification unit 231, the obstacle recognition unit 232, and the route generation unit 233 are provided in the mobile body 2 or the edge computer 11, but some of these functions may be provided in the edge computer 11 and other functions may be provided in the mobile body 2, so that the functions of the mobile body position identification unit 231, the obstacle recognition unit 232, and the route generation unit 233 are separately located in the mobile body 2 and the edge computer 11.

[0066] Furthermore, although the edge computer 11 is described as being configured as a single computer, it may also be configured as multiple computers. In particular, when the edge computer 11 is configured as being distributed across multiple computers arranged in a cloud-like manner on the Internet, the calculations of each function of the edge computer 11 can be distributed and processed at high speed. For example, FIG. 12 shows the cumulative distribution of processing time when the processing required for self-localization in this system is performed by (a) one edge computer 11 and (b) four edge computers 11. As shown in FIG. 12(a), when processing is performed by one edge computer 11, the processing time is approximately several seconds. However, as shown in FIG. 12(b), when processing is distributed across four edge computers 11, it can be confirmed that the processing time is reduced to less than one second.

[0067] Next, examples of embodiments of the present invention will be described with reference to FIGS.

[0068] Example 1 (Scenario) The feasibility of this system was verified through simulation. A 3D model called TurtleBot3 World was used as the experimental field, with only one central column remaining out of nine columns. One robot followed designated waypoints while moving. The robot (mobile body 2) was initially positioned at (1,1). Waypoints were set at the four corners of a 2.0 meter square. The experiment consisted of two scenarios. One scenario assumed normal operation, in which the robot's onboard sensor (second sensor unit 20) operated normally without any problems. The other scenario assumed an abnormality, in which the onboard sensor became disabled during the trial for some reason.

[0069] (Experimental Setup) A prototype of the system was implemented using the robot operating system (ROS 2 Humble Hawksbill). The experiment was conducted using a simulation using Gazebo. The robot model used was a 3D model of TurtleBot3 Waffle. This robot is equipped with two sensors: a wheel odometry and a 2D LIDAR unit (second sensor unit 20). A package called Navigation2 was used to enable the robot to navigate autonomously.

[0070] A marker was attached to the top of the robot to indicate its posture (position, orientation, and speed). This marker is a square marker that encodes a unique ID. A sensor unit (first sensor unit 10a) was installed on a pillar in the experimental field, and the real space where the experiment took place was scanned. The AI ​​periodically detects the posture of the robot by detecting the posture of the marker from the image.

[0071] The robot navigated autonomously according to four given waypoints. The waypoints were set in the following order: (1, 1), (-1, 1), (-1, -1), (1, -1). The robot did not replan its route while passing each waypoint. The robot used adaptive Monte Carlo localization (AMCL) to estimate its pose from data from an onboard 2D LIDAR unit. The robot predicted and corrected its pose using the Extended Kalman Filter (EKF) package. In the normal scenario, the EKF received wheel odometry, the pose estimated by AMCL, and the pose estimated by the AI. In the abnormal scenario, the EKF received only wheel odometry and pose estimated by the AI.

[0072] The communication delay between the robot and the edge computer was reproduced by delaying the data transmission. The delay was set in 0.1 second increments from 0.0 to 1.0 seconds.

[0073] (Benchmark and Evaluation Metrics) In this example, we compared our system with a benchmark. In the benchmark, if the embedded timestamp was older than the current time, the posture estimated by the AI ​​was dropped. To evaluate the dissimilarity of the robot's path, we used the DTW (Dynamic Time Warping) method.

[0074] For each trial, the marker detection result was recorded as the pass score. The sequence of passing points in a normal scenario without communication delay was considered as the reference point for each scheme and benchmark. For each trial, the DTW distance between the recorded passing points and the reference point was calculated.

[0075] (Results) Figure 13 shows the path taken by the robot in the benchmark results for (a) the normal scenario and (b) the abnormal scenario. The robot performed autonomous driving according to two waypoints in all trials. In both scenarios, the route taken became more curved as the communication delay increased. Figure 14 shows the path taken by the robot in the results of this system in (a) the normal scenario and (b) the abnormal scenario. The robot performed autonomous driving according to the same four waypoints in all trials. In both scenarios, the route taken did not become more curved, even as the communication delay increased. This is thought to be because the pose estimated by the AI ​​was used in all trials while taking communication delay into account.

[0076] The DTW method was used to evaluate the dissimilarity of the robot's route. Figure 15 shows the DTW distance between the robot's reference point and the waypoint in each trial, representing the results of the benchmark and the present system for the normal and abnormal scenarios. In the benchmark results, the DTW distance increased as the communication delay increased in both scenarios. The maximum distance in the normal scenario of the present system was approximately 4.46 when the communication delay was set to 1.0 seconds. The maximum distance in the abnormal scenario of the present system was approximately 3.25 when the communication delay was set to 0.9 seconds. In this trial, the DTW distance in the abnormal scenario was almost the same as that in the normal scenario. The average absolute difference between the DTW distances in the normal and abnormal scenarios was approximately 2.69. In the results of this example, the DTW distances in all trials were less than 5.0 in both scenarios. The average absolute difference between the DTW distances in the normal and abnormal scenarios was approximately 0.71. These results demonstrate that autonomous driving was supported despite communication delays, even when the robot's onboard sensors were disabled.

[0077] Example 2 [Setting the Experimental and Analysis Conditions (Equipment, Location, and Parameters)] An evaluation experiment was conducted to verify the effectiveness of the proposed system. The experiment involved one micro-mobility vehicle autonomously driving in a simulator environment. The micro-mobility vehicle autonomously drove along a predetermined course. The micro-mobility autonomous driving process was handled by a worker node. The experimental system consisted of one master node, one worker node, and one hub. One desktop PC was used as the master node. The desktop PC was configured with an RTX 3060Ti GPU and an Intel Core i7 12700K CPU. In the experimental system, a simulator environment for running multiple micro-mobilities on the desktop PC was constructed. Four Jetson Xavier NX were used as worker nodes. One LSW6-GT-5EPL / NBK was used as the hub.

[0078] [Explanation of the Results in Figure 16] This shows the results when one micro-mobility vehicle was driven autonomously using one worker node on a simulator. Figure 16 shows the driving trajectory when one micro-mobility vehicle was driven autonomously using one worker node on a simulator. Figure 17 shows each micro-mobility route so that it can be distinguished. R1 refers to the micro-mobility numbered 1. The micro-mobility vehicle drove autonomously from a predetermined start position to a goal position. The time it took to drive autonomously from the predetermined start position to the goal position was 158.37 seconds. The average execution time for self-localization was 0.20 seconds, with a standard deviation of 0.00.

[0079] Example 3 [Setting the Experimental and Analysis Conditions (Equipment, Location, and Parameters)] An evaluation experiment was conducted to verify the effectiveness of the proposed system. The experiment involved 10 micro-mobility vehicles autonomously driving in a simulator environment. The micro-mobility vehicles autonomously drove along a predetermined course. The autonomous driving process for the micro-mobility vehicles was handled by a worker node. The experimental system consisted of one master node, two worker nodes, and one hub. One desktop PC was used as the master node. The desktop PC was configured with an RTX 3060Ti GPU and an Intel Core i7 12700K CPU. In the experimental system, a simulator environment for running multiple micro-mobility vehicles was built on the desktop PC. Four Jetson Xavier NX were used as worker nodes. One LSW6-GT-5EPL / NBK was used as the hub.

[0080] [Explanation of the Results in Figure 17] This shows the results when 10 micro-mobilities were driven autonomously using one worker node on a simulator. Figure 17 shows the driving trajectories when 10 micro-mobilities were driven autonomously using one worker node on a simulator. In Figure 17, symbols are used to distinguish between micro-mobility routes. R1-10 refers to micro-mobilities numbered 1 to 10. None of the micro-mobilities were able to drive autonomously from the predetermined start position to the goal position. The average execution time for localization was 0.62 seconds, with a standard deviation of 0.16. It is believed that processing 10 micro-mobilities autonomously using one worker node increased the processing load, which in turn increased the execution time for localization.

[0081] [Practical benefits to society from the results] It became clear that when a single edge computer processes the autonomous driving of many vehicles, the load on that single edge computer is high, processing delays occur, and safe autonomous driving becomes impossible.

[0082] Example 4 [Setting the Experimental and Analysis Conditions (Equipment, Location, and Parameters)] An evaluation experiment was conducted to verify the effectiveness of the proposed system. The experiment involved 10 micro-mobility vehicles autonomously driving in a simulator environment. The micro-mobility vehicles autonomously drove along a predetermined course. The autonomous driving process for the micro-mobility vehicles was handled by a worker node. The experimental system consisted of one master node, two worker nodes, and one hub. One desktop PC was used as the master node. The desktop PC was configured with an RTX 3060Ti GPU and an Intel Core i7 12700K CPU. In the experimental system, a simulator environment for running multiple micro-mobility vehicles was built on the desktop PC. Four Jetson Xavier NX were used as worker nodes. One LSW6-GT-5EPL / NBK was used as the hub.

[0083] [Explanation of the results in Figure 18] This shows the results when 10 micro-mobility vehicles were driven autonomously using two worker nodes on a simulator. The following three load balancing methods were used. The first is a static load balancing method that uses one system statistic. Here, CPU usage is used as the system statistic. The second is a dynamic load balancing method that uses CPU usage. Here, CPU usage is used as the system statistic. The third is a dynamic load balancing method that uses multiple system statistic values. Here, CPU usage, CPU queue length, memory usage, and network traffic are used as system statistic values.

[0084] Figure 18(a) shows the execution time of localization using three load balancing methods in a box plot. In the figure, static_cpu_x% is the result when a static load balancing method using CPU utilization is used. x indicates the threshold value of CPU utilization that is the basis for load balancing. x was set in increments of 5 between 70 and 95. In the figure, dynamic_cpu is the result when a dynamic load balancing method using one system statistic value is used. In the figure, dynamic_multiple is the result when a load balancing method using multiple system statistic values ​​is used.

[0085] The following can be seen from Figure 18(a). ・The execution time of localization when using a static load balancing method based on CPU utilization was as follows. When the CPU utilization threshold was set to 70, 75...95%, the average (standard deviation) execution time of localization was 0.27 s (0.0), 0.27 s (0.0), 0.27 s (0.0), 0.28 s (0.0), 0.29 s (0.0), and 0.28 s (0.0), respectively. ・The execution time of localization when using a dynamic load balancing method based on a single system statistic was as follows. The average execution time of localization was 0.27 s with a standard deviation of 0.01 s. ・The execution time of localization when using a load balancing method based on multiple system statistics was as follows. The average execution time of localization was 0.29 s with a standard deviation of 0.00 s. This section shows the comparison results of running ten micro-mobility vehicles autonomously on a simulator using one worker node without background load and two worker nodes with three load balancing methods. This section shows the comparison results with a static load balancing method using CPU utilization. At all CPU utilization threshold settings, the localization execution time was reduced compared to running ten micro-mobility vehicles on one worker node, and all micro-mobility vehicles were able to autonomously travel from a predetermined start position to a goal position. This section shows the comparison results with a dynamic load balancing method using a single system statistic. The localization execution time was reduced compared to running ten micro-mobility vehicles on one worker node, and all micro-mobility vehicles were able to autonomously travel from a predetermined start position to a goal position. This section shows the comparison results with a dynamic load balancing method using multiple system statistics. The localization execution time was reduced compared to running ten micro-mobility vehicles on one worker node, and all micro-mobility vehicles were able to autonomously travel from a predetermined start position to a goal position. The following is an explanation of the comparison results: ・This is thought to be because the load on one worker node was reduced by using a load balancing method to process autonomous driving on two worker nodes.

[0086] All the load balancing methods enabled all micro-mobilities to autonomously travel the predetermined route. Figure 18(b) shows the travel trajectory when using the load balancing method that uses multiple system statistics.

[0087] The following can be seen from Figure 18(b). - With the dynamic load balancing method using multiple system statistics, all micro-mobilities were able to travel autonomously from the predetermined start position to the goal position. - With the dynamic load balancing method using multiple system statistics, six robots for each micro-mobility were assigned to worker node1 and four to worker node2. - With the dynamic load balancing method using multiple system statistics, robot1, robot2, robot3, robot4, robot5, and robot6 were assigned to worker node1, and robot7, robot8, robot9, and robot10 were assigned to worker node2. Using the dynamic load balancing method using multiple system statistics, the time taken for robots 1, 2...10 to travel autonomously from a predetermined start position to a goal position was 227.43s, 229.28s, 234.86s, 228.31s, 228.46s, 224.58s, 227.34s, 225.28s, 231.69s, and 225.55s, respectively.

[0088] [Practical benefits to society from the results] When outsourcing autonomous driving processing to the edge, the load of the autonomous driving processing can be distributed using multiple edge computers, making it possible to operate many autonomous vehicles simultaneously.

[0089] Example 5 [Experimental and Analysis Conditions (Devices, Locations, and Parameters)] An evaluation experiment was conducted to verify the effectiveness of the proposed system. The experiment involved 10 micro-mobilities driving autonomously in a real environment and a simulator environment. Nine micro-mobilities were driven autonomously in the simulator environment, and one actual micro-mobility was driven autonomously in the real environment. The micro-mobilities drove autonomously along a predetermined course. The autonomous driving process for the micro-mobility was performed by a worker node. The experimental system consisted of one master node, two worker nodes, one router, and one actual micro-mobility. One laptop PC was used as the master node. The laptop PC was configured with an RTX 3050 Laptop GPU as its GPU and an Intel Core i7-1280P as its CPU. In the experimental system, a simulator environment for running multiple micro-mobilities was constructed on the laptop PC. Two Jetson Xavier NX were used as worker nodes. One tp-link Archer AX73 was used as the router. One Turtlebot 3 burger was used as the actual micro-mobility.

[0090] [Explanation of the results in Figure 19] Figure 19 shows the results of simultaneously running one micro-mobility vehicle in a real environment and nine micro-mobility vehicles in a simulator environment using one worker node with a background load of 50% (CPU and memory). Figure 19 shows the driving trajectory of each micro-mobility vehicle. Figure 19(b) illustrates each driving trajectory so that the micro-mobility routes can be distinguished. Figure 19(a) shows the driving trajectory of one micro-mobility vehicle running autonomously in a real environment. Figure 19(b) shows the driving trajectory of nine micro-mobility vehicles running autonomously in a simulator environment.

[0091] The following can be seen from Figure 19. ・When 10 micro-mobilities were driven autonomously by one worker node, none of the micro-mobilities were able to drive autonomously from the predetermined start position to the goal position. ・The average execution time for self-localization was 1.24 seconds, with a standard deviation of 0.61.

[0092] Example 6 [Setting the Experimental and Analysis Conditions (Devices, Locations, and Parameters)] An evaluation experiment was conducted to verify the effectiveness of the proposed system. The experiment involved 10 micro-mobilities driving autonomously in a real environment and a simulator environment. Nine micro-mobilities were driven autonomously in the simulator environment, and one actual micro-mobility was driven autonomously in the real environment. The micro-mobilities autonomously drove along a predetermined course. The autonomous driving process for the micro-mobility was performed by a worker node. The experimental system consisted of one master node, two worker nodes, one router, and one actual micro-mobility. One laptop PC was used as the master node. The laptop PC was configured with an RTX 3050 Laptop GPU as the GPU and an Intel Core i7-1280P as the CPU. In the experimental system, a simulator environment for running multiple micro-mobilities was built on the laptop PC. Two Jetson Xavier NX were used as worker nodes. One tp-link Archer AX73 was used as the router. One Turtlebot 3 burger was used as the actual micro-mobility.

[0093] [Explanation of the results in Figure 20] Figure 20 shows the driving trajectories of each micro-mobility vehicle when using a dynamic load balancing method using multiple system statistics. Figure 20(b) shows each driving trajectory so that the micro-mobility routes can be distinguished. Figure 20(a) shows the driving trajectory of one micro-mobility vehicle that drove autonomously in a real environment. Figure 20(b) shows the driving trajectories of nine micro-mobility vehicles that drove autonomously in a simulator environment.

[0094] The following can be seen from Figure 20. ・All of the micro-mobility vehicles were able to travel autonomously from the predetermined starting position to the goal position. ・The average execution time for self-localization was 0.78 seconds, with a standard deviation of 0.22.

[0095] The following can be seen from Figure 20(a). - With the dynamic load balancing method using multiple system statistics, all micro-mobilities were able to travel autonomously from the predetermined start position to the goal position. - With the dynamic load balancing method using multiple system statistics, micro-mobilities were assigned to worker node1. - With the dynamic load balancing method using multiple system statistics, the time it took to travel autonomously from the predetermined start position to the goal position was 145.33 seconds.

[0096] The following can be seen from Figure 20(b). - With the dynamic load balancing method using multiple system statistics, all micro-mobilities were able to travel autonomously from the predetermined start position to the goal position. - With the dynamic load balancing method using multiple system statistics, four and five micro-mobilities were assigned to worker node 1 and 2, respectively. - With the dynamic load balancing method using multiple system statistics, robot1, robot2, robot3, and robot4 were assigned to worker node 1, and robot5, robot6, robot7, robot8, and robot9 were assigned to worker node 2. Using the dynamic load balancing method using multiple system statistics, the time taken for robots 1, 2...9 to travel autonomously from a predetermined start position to a goal position was 340.88s, 345.19s, 346.90s, 345.11s, 358.11s, 336.52s, 358.65s, 352.30s, and 329.16s, respectively.

[0097] [Practical benefits to society from the results] We were able to confirm in a real-world environment that when outsourcing autonomous driving processing to the edge, using multiple edge computers to distribute the load of autonomous driving processing makes it possible to operate many autonomous vehicles simultaneously.

[0098] Although the embodiments of the present invention have been described above with reference to the drawings, the present invention is not limited to the illustrated embodiments. Various modifications and variations can be made to the illustrated embodiments within the same scope as the present invention or within an equivalent scope.

[0099] DESCRIPTION OF SYMBOLS 1...Edge computing system 10...Sensor device 10a...First sensor unit 11...Edge computer (server device) 111...Aggregation unit 112...Map generation unit 113...Digital twin generation unit 114...Information storage unit 115...Transmission unit 2...Mobile body 20...Second sensor unit 21...First receiving unit 22...Second receiving unit 23...On-board computer 231...Mobile body position identification unit 231a...Estimation unit 231b...Correction unit 232...Obstacle recognition unit 232a...Detection unit 232b...Tracking unit 232c...Prediction unit 233...Route generation unit 234...Control unit 235...Selection unit 24...Drive mechanism

Claims

1. A mobile object autonomous driving support system comprising a sensor device arranged in the environment of a real space, a server device connected to the sensor device in a state capable of communicating with the server device, and a mobile object connected to the server device in a state capable of communicating with the server device, and supporting the autonomous driving of the mobile object in a real space, wherein the sensor device comprises a first sensor unit that acquires first image sensor data consisting of a point cloud in the real space, the mobile object comprises a second sensor unit that acquires second image sensor data consisting of a point cloud in the real space, and the server device or the mobile object comprises a mobile object position identification unit that identifies the position of the mobile object in the real space based on the first image sensor data consisting of the point cloud in the real space acquired by the first sensor unit of the sensor device and the second image sensor data consisting of the point cloud in the real space acquired by the second sensor unit of the mobile object.

2. The mobile body autonomous driving assistance system described in claim 1, wherein the mobile body position identification unit includes an estimation unit that estimates the position of the mobile body based on second image sensor data consisting of a point cloud in real space acquired by the second sensor unit of the mobile body, and a correction unit that corrects the position of the mobile body estimated by the estimation unit based on first image sensor data consisting of a point cloud in real space acquired by the first sensor unit of the sensor device.

3. The mobile body autonomous driving assistance system described in claim 1, wherein the server device or the mobile body is equipped with an obstacle recognition unit that recognizes obstacles in real space based on first image sensor data consisting of a point cloud in real space acquired by the first sensor unit of the sensor device and second image sensor data consisting of a point cloud in real space acquired by the second sensor unit of the mobile body.

4. A mobile body autonomous driving assistance system as described in claim 3, wherein the obstacle recognition unit comprises a detection unit that detects obstacles around the mobile body, a tracking unit that captures the trajectory of the obstacle based on the detection result of the obstacle by the detection unit, and a prediction unit that predicts the position of the obstacle after a predetermined time has passed based on the trajectory of the obstacle captured by the tracking unit.

5. A mobile body autonomous driving assistance system as described in claim 3, wherein the server device or the mobile body is equipped with a route generation unit that generates a driving route for the mobile body in real space based on the position of the mobile body in real space identified by the mobile body position identification unit and the position of an obstacle in real space identified by the obstacle recognition unit.

6. The mobile body autonomous driving assistance system of claim 1, wherein the server device or the mobile body comprises a selection unit that selects image sensor data consisting of a point cloud in real space, either first image sensor data consisting of a point cloud in real space acquired by the first sensor unit of the sensor device or second image sensor data consisting of a point cloud in real space acquired by the second sensor unit of the mobile body, and the mobile body position identification unit identifies the position of the mobile body in real space based on the first image sensor data or the second image sensor data consisting of a point cloud in real space selected by the selection unit.

7. A mobile body autonomous driving assistance system as described in claim 6, wherein the selection unit preferentially selects second image sensor data consisting of a point cloud in real space acquired by the second sensor unit of the mobile body, and if it determines that the second image sensor data consisting of a point cloud in real space is invalid, selects first image sensor data consisting of a point cloud in real space acquired by the first sensor unit of the sensor device.

8. A mobile autonomous driving assistance system as described in claim 5, wherein the mobile body is equipped with a control unit that controls the traveling of the mobile body based on the traveling path of the mobile body in real space generated by the path generation unit.

9. The mobile autonomous driving assistance system according to claim 1, wherein the server device includes an aggregation unit that aggregates first image sensor data consisting of a point cloud in real space acquired by the first sensor unit.

10. The mobile autonomous driving assistance system described in claim 9, wherein the server device includes a map generation unit that generates a map of real space based on first image sensor data consisting of a cloud of points in real space aggregated by the aggregation unit, and the mobile object position identification unit uses information related to the map of real space generated by the map generation unit when identifying the position of the mobile object.

11. The mobile autonomous driving assistance system described in claim 9, wherein the server device is provided with a digital twin generation unit that generates a digital twin corresponding to the real space based on first image sensor data consisting of a point cloud in the real space aggregated by the aggregation unit, and the mobile body position identification unit uses information regarding the digital twin corresponding to the real space generated by the digital twin generation unit when identifying the position of the mobile body.

12. The mobile autonomous driving assistance system described in claim 11, wherein the digital twin generation unit detects the moving body in real space using a learning model that has been machine-trained in advance to detect the moving body based on first image sensor data consisting of a point cloud in real space.

Images