Mobile body control device and method, learning device and method, and storage medium

Abstract

A mobile body control device and method, a learning device and method, and a storage medium capable of detecting a drivable space of a mobile body based on less learning data without complicating a hardware structure for sensing. The mobile body control device includes: an acquisition unit that acquires an object aerial view image obtained by converting an image of a surrounding situation of a mobile body, which is captured by a camera pair mounted on the mobile body, to an aerial view coordinate system; a solid object detection unit that detects a solid object in the object aerial view image by inputting the object aerial view image to a learned model that is learned in such a manner that at least a solid object in an input aerial view image is output when the aerial view image is input; a space detection unit that detects a drivable space of the mobile body based on the detected solid object; and a travel control unit that causes the mobile body to travel in such a manner that the drivable space is passed through.

Description

Technical Field

[0001] This invention relates to a mobile body control device, a mobile body control method, a learning device, a learning method, and a storage medium. Background Technology

[0002] Previously, techniques for detecting obstacles in the vicinity of a moving body using sensors mounted on the moving body were known. For example, Japanese Patent Application Publication No. 2021-162926 disclosed a technique for detecting obstacles in the vicinity of a moving body based on information obtained from multiple ranging sensors mounted on the moving body.

[0003] The technology described in Patent Document 1 uses multiple ranging sensors, such as ultrasonic sensors and LiDAR, to detect obstacles around a moving object. However, when using a structure with multiple ranging sensors, the hardware structure for sensing becomes complex, thus tending to increase the system cost. On the other hand, to reduce system cost, a simple hardware structure using only a camera has been considered, but in this case, a large amount of learning data is required for sensing to ensure durability in various scenarios. Summary of the Invention

[0004] The present invention was made in consideration of such circumstances, and one of its objectives is to provide a mobile body control device, mobile body control method, learning device, learning method, and storage medium that can detect the drivable space of a mobile body based on less learning data without complicating the hardware structure used for sensing.

[0005] The mobile body control device, mobile body control method, learning device, learning method, and storage medium of the present invention adopt the following structure.

[0006] (1): A mobile body control device according to one aspect of the present invention includes: an acquisition unit that acquires a bird's-eye view image of an object, the bird's-eye view image of the object being obtained by transforming an image of the surrounding conditions of the mobile body captured by a camera mounted on the mobile body into a bird's-eye view coordinate system; a stereoscopic object detection unit that detects stereoscopic objects in the bird's-eye view image of the object by inputting a learned model that has been trained to output at least stereoscopic objects in the bird's-eye view image when the bird's-eye view image is input; a space detection unit that detects the drivable space of the mobile body based on the detected stereoscopic objects; and a driving control unit that causes the mobile body to drive through the drivable space.

[0007] (2): In the above (1) scheme, the learning completed model is a learning completed model that is learned by outputting whether the moving body can cross the three-dimensional objects in the bird's-eye view image when the bird's-eye view image is input.

[0008] (3): In the above scheme (1) or (2), the learning completed model is a learning completed model based on the first teaching data. In the first teaching data, for the area in the bird's-eye view image with radial patterns centered on the lower center of the bird's-eye view image, there is a corresponding annotation indicating that the area is a three-dimensional object.

[0009] (4): In the above scheme (3), the learning completed model is a learning completed model that has been learned based on the second teaching data in addition to the first teaching data. In the second teaching data, for the area in the bird's-eye view image with a monochrome pattern that is different from the color of the road surface, there is a corresponding annotation indicating that the area is a three-dimensional object.

[0010] (5): In the above scheme (3) or (4), the learning completed model is a learning completed model that has been learned based on the third teaching data in addition to the first teaching data. In the third teaching data, for the road markings in the bird's-eye view image, there is a corresponding annotation indicating that the road markings are non-three-dimensional objects.

[0011] (6): In any of the above (1) to (5), the moving body control device further includes a reference mapping generation unit, which identifies objects contained in the image based on the image of the surrounding conditions of the moving body captured by the camera, and generates a reference mapping that reflects the position of the identified objects. The space detection unit detects the drivable space by matching the detected three-dimensional objects in the bird's-eye view image of the object with the generated reference mapping.

[0012] (7): In any of the above (1) to (6), the camera includes a first camera disposed below the moving body and a second camera disposed above the moving body. The stereoscopic object detection unit detects the stereoscopic object based on a first object bird's-eye view image, detects the object in the second object bird's-eye view image along with its position information based on a second object bird's-eye view image, and detects the position of the stereoscopic object by matching the detected stereoscopic object with the detected object having the position information. The first object bird's-eye view image is obtained by transforming the image of the surrounding conditions of the moving body captured by the first camera into a bird's-eye view coordinate system, and the second object bird's-eye view image is obtained by transforming the image of the surrounding conditions of the moving body captured by the second camera into a bird's-eye view coordinate system.

[0013] (8): In any of the above (1) to (7), before the stereoscopic object detection unit transforms the image of the surrounding conditions of the moving body captured by the camera into the bird's-eye view coordinate system, the stereoscopic object detection unit detects hollow objects reflected in the image and assigns identification information to the hollow objects. The space detection unit also detects the drivable space based on the identification information.

[0014] (9): In any of the above schemes (1) to (8), the stereo detection unit detects the same area as a stereo object when the displacement of the same area based on the road surface in multiple bird's-eye view images of the object obtained in a time sequence is above a threshold.

[0015] (10): A mobile body control method of one aspect of the present invention enables a computer to perform the following processing: acquiring an object bird's-eye view image, which is obtained by transforming an image of the surrounding conditions of the mobile body taken by a camera mounted on the mobile body into a bird's-eye view coordinate system; detecting three-dimensional objects in the object bird's-eye view image by inputting a learned model that has been trained to output at least three-dimensional objects in the bird's-eye view image when the bird's-eye view image is input; determining the drivable space of the mobile body based on the detected three-dimensional objects; and causing the mobile body to travel through the drivable space.

[0016] (11): In one aspect of the present invention, a storage medium stores a program that causes a computer to perform the following processing: acquiring an object bird's-eye view image, which is obtained by transforming an image of the surrounding conditions of the moving body taken by a camera mounted on the moving body into a bird's-eye view coordinate system; detecting three-dimensional objects in the object bird's-eye view image by inputting a learned model that has been trained to output at least three-dimensional objects in the bird's-eye view image when the bird's-eye view image is input; determining the drivable space of the moving body based on the detected three-dimensional objects; and causing the moving body to travel through the drivable space.

[0017] (12): A learning device of one aspect of the present invention learns based on teaching data in such a way that when a bird’s-eye view image is input, it outputs at least the three-dimensional objects in the bird’s-eye view image. In the teaching data, for a region in the bird’s-eye view image that has a radial pattern centered at the lower center of the bird’s-eye view image, there is a corresponding annotation indicating that the region is a three-dimensional object.

[0018] (13): A learning method of one aspect of the present invention enables a computer to perform the following processing: based on teaching data, learning in such a way that when a bird’s-eye view image is input, at least the three-dimensional objects in the bird’s-eye view image are output, wherein the teaching data includes annotations indicating that the regions in the bird’s-eye view image with radial patterns centered at the lower center of the bird’s-eye view image are three-dimensional objects.

[0019] (14): In one aspect of the present invention, the storage medium stores a program, wherein the program causes a computer to perform the following processing: learning, based on teaching data, to output at least three-dimensional objects in the bird's-eye view image when the bird's-eye view image is input, wherein the teaching data includes annotations indicating that the regions in the bird's-eye view image having radial patterns centered at the lower center of the bird's-eye view image are three-dimensional objects.

[0020] According to the schemes in (1) to (14), the drivable space of a moving body can be detected based on less learning data without complicating the hardware structure used for sensing.

[0021] According to the schemes (2) to (5) or (12) to (14), the drivable space of a moving body can be detected based on less learning data.

[0022] According to the scheme in (6), the drivable space of a moving body can be detected more reliably.

[0023] According to the scheme in (7), the existence and location of three-dimensional objects can be detected more reliably.

[0024] According to schemes (8) or (9), three-dimensional objects that obstruct the movement of vehicles can be detected more reliably. Attached Figure Description

[0025] Figure 1 This is a diagram illustrating an example of the structure of a vehicle M equipped with a mobile body control device according to an embodiment of the present invention.

[0026] Figure 2 This is a diagram illustrating an example of a reference mapping generated by the reference mapping generation unit based on an image captured by a camera.

[0027] Figure 3 This is a diagram showing an example of a bird's-eye view image acquired by the bird's-eye view image acquisition unit.

[0028] Figure 4 This is a diagram showing an example of drivable space on a reference map detected by the space detection unit.

[0029] Figure 5 This is a flowchart illustrating an example of a process executed by a mobile body control device.

[0030] Figure 6 This is a diagram representing an example of teaching data from a bird's-eye view image used to generate a learned model.

[0031] Figure 7 It is a diagram used to illustrate the difference between the near and far areas of the vehicle in a bird's-eye view image.

[0032] Figure 8 This diagram illustrates a method for detecting hollow objects in bird's-eye view images.

[0033] Figure 9 This diagram illustrates a method for detecting three-dimensional objects based on the displacement of three-dimensional objects in a time series from a bird's-eye view image.

[0034] Figure 10 This is a flowchart illustrating another example of a process executed by a mobile body control device.

[0035] Figure 11 This is a diagram illustrating an example of the structure of the vehicle equipped with a modified version of the present invention's movement control device.

[0036] Figure 12 This is an example of a bird's-eye view image acquired by a bird's-eye view image acquisition unit based on an image captured by a camera.

[0037] Figure 13 This is a flowchart illustrating another example of the process executed by the moving body control device of the modified example. Detailed Implementation

[0038] Hereinafter, with reference to the accompanying drawings, embodiments of the mobile body control device, mobile body control method, learning device, learning method, and storage medium of the present invention will be described. The mobile body control device is a device for controlling the movement of a mobile body. A mobile body includes vehicles with three or four wheels, two-wheeled vehicles, micro-mobilities, etc., and can include all mobile bodies capable of moving on a road surface. In the following description, the mobile body is assumed to be a four-wheeled vehicle, and the vehicle equipped with a driving support device is referred to as vehicle M.

[0039] [summary]

[0040] Figure 1 This is a diagram illustrating an example of the structure of a vehicle M equipped with the mobile body control device 100 according to an embodiment of the present invention. (See diagram for example.) Figure 1As shown, this vehicle M includes a camera 10 and a movement control device 100. The camera 10 and the movement control device 100 are interconnected via multiple communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, and wireless communication networks. It should be noted that... Figure 1 The structure shown is just one example; other structures can also be added.

[0041] Camera 10 is, for example, a digital camera utilizing a solid-state imaging element such as CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor). In this embodiment, camera 10 is, for example, mounted on the front bumper of the vehicle M, but camera 10 can be mounted at any location capable of capturing images of the front of the vehicle M. Camera 10 periodically and repeatedly captures images of the perimeter of the vehicle M. Camera 10 can also be a stereo camera.

[0042] The mobile control device 100 includes, for example, a reference mapping generation unit 110, a bird's-eye view image acquisition unit 120, a three-dimensional object detection unit 130, a space detection unit 140, a driving control unit 150, and a storage unit 160. The storage unit 160 stores, for example, a learned model 162. These components are implemented, for example, by executing a program (software) using a hardware processor such as a CPU (Central Processing Unit). Some or all of these components can also be implemented using hardware (including a circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or through the coordinated use of software and hardware. The program can be pre-saved in a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device with a non-transitory storage medium), or it can be saved in a removable storage medium such as a DVD or CD-ROM (a non-transitory storage medium), and installed by mounting the storage medium onto a drive device. The storage unit 160 is implemented, for example, by ROM (Read Only Memory), flash memory, SD card, RAM (Random Access Memory), HDD (Hard Disk Drive), registers, etc.

[0043] The reference mapping generation unit 110 performs image recognition processing based on well-known methods (binarization, contour extraction, image enhancement, feature extraction, pattern matching, or other processing utilizing a learned model) on an image of the surrounding environment of the vehicle M captured by the camera 10, in order to identify objects contained in the image. Here, "objects" refers to, for example, other vehicles (e.g., surrounding vehicles within a predetermined distance of the vehicle M). Objects may also include pedestrians, bicycles, road structures, etc. Road structures include, for example, road signs, traffic signals, curbs, median strips, guardrails, fences, walls, intersections, etc. Furthermore, objects may include obstacles that hinder the movement of the vehicle M. Moreover, the reference mapping generation unit 110 may not identify all objects contained in the image, but may initially identify road dividing lines contained in the image, and only identify objects existing inside the identified road dividing lines.

[0044] Next, the reference mapping generation unit 110 transforms the image based on the camera coordinate system to the overhead coordinate system and generates a reference mapping that reflects the position of the identified object. Here, the reference mapping refers to information representing the shape of a road, such as road segments and nodes connecting the road segments.

[0045] Figure 2 This diagram illustrates an example of a reference mapping generated by the reference mapping generation unit 110 based on an image captured by the camera 10. Figure 2 The upper part represents the image captured by camera 10. Figure 2 The lower part represents the reference mapping generated by the reference mapping generation unit 110 based on the image. For example... Figure 2 As shown in the upper part, the reference mapping generation unit 110 identifies objects contained in the image captured by the camera 10, which in this case is the vehicle in front, by performing image recognition processing on the image. Next, the reference mapping generation unit 110... Figure 2 As shown in the lower part, a reference map is generated that reflects the position of the identified vehicle ahead.

[0046] The bird's-eye view image acquisition unit 120 acquires a bird's-eye view image by transforming the image captured by the camera 10 into a bird's-eye view coordinate system. Figure 3 This is a diagram showing an example of a bird's-eye view image acquired by the bird's-eye view image acquisition unit 120. Figure 3 The upper part represents the image captured by camera 10. Figure 3 The lower part represents the bird's-eye view image acquired by the bird's-eye view image acquisition unit 120 based on this image. Figure 3 In the bird's-eye view image, reference numeral O indicates the location of camera 10 within the vehicle M. (Comparison) Figure 3The image shown at the top is similar to Figure 3 As can be seen from the bird's-eye view image shown at the bottom: contained in Figure 3 The three-dimensional object in the upper part of the image Figure 3 In the lower bird's-eye view image, it is transformed into an AR pattern with radial lines centered at position O.

[0047] The stereoscopic object detection unit 130 detects stereoscopic objects in a bird's-eye view image by inputting a learned model 162, which is designed to output at least the stereoscopic objects in the bird's-eye view image when the bird's-eye view image is input. A detailed method for generating the learned model 162 will be described later.

[0048] The space detection unit 140 detects the drivable space (driving space) of the vehicle M in the bird's-eye view image by excluding the three-dimensional objects detected by the three-dimensional object detection unit 130 from the bird's-eye view image. Figure 3 In the bird's-eye view image, the reference numeral FS1 indicates the drivable space of the vehicle M. Next, the space detection unit 140 transforms the drivable space FS1 of the vehicle M in the bird's-eye view image to the overhead coordinate system and matches it with the reference mapping, thereby detecting the drivable space FS2 on the reference mapping.

[0049] Figure 4 This diagram illustrates an example of a drivable space FS2 on a reference map, detected by the space detection unit 140. Figure 4 In the diagram, the mesh-like area represents the drivable space FS2 on the reference map. The driving control unit 150 generates a target track TT so that the vehicle M traverses the drivable space FS2, and causes the vehicle M to travel along the target track TT. The target track TT may, for example, include speed elements. For example, the target track is represented by a sequence of locations (track points) that the vehicle M should reach. Track points are locations that the vehicle M should reach at predetermined travel distances (e.g., several meters) along the route. In contrast, target speed and target acceleration are generated as part of the target track at predetermined sampling times (e.g., a fraction of a second). Alternatively, track points may be positions that the vehicle M should reach at predetermined sampling times. In this case, the target speed and target acceleration information are represented by the intervals of the track points. It should be noted that, in this embodiment, as an example, the present invention is illustrated to be applicable to the case of autonomous driving. However, the present invention is not limited to such a structure. It can also be applied to driving support such as displaying a drivable space FS2 without three-dimensional objects on the navigation device of the vehicle M, or assisting the steering wheel in driving in the drivable space FS2.

[0050] Figure 5This is a flowchart illustrating an example of the processing flow performed by the mobile body control device 100. First, the mobile body control device 100 acquires an image of the surrounding environment of the vehicle M captured by the camera 10 (step S100). Next, the reference mapping generation unit 110 performs image recognition processing on the acquired image to identify objects contained in the image (step S102). Then, the reference mapping generation unit 110 transforms the image based on the acquired camera coordinate system to a top-view coordinate system and generates a reference map reflecting the position of the identified objects (step S104).

[0051] Parallel to steps S102 and S104, the following processing is performed: The bird's-eye view image acquisition unit 120 acquires a bird's-eye view image by transforming the image captured by the camera 10 into a bird's-eye view coordinate system (step S106). Next, the stereoscopic object detection unit 130 detects stereoscopic objects in the bird's-eye view image by inputting the bird's-eye view image acquired by the bird's-eye view image acquisition unit 120 into the learned model 162 (step S108). Next, the space detection unit 140 detects the drivable space FS1 of the vehicle M in the bird's-eye view image by excluding the stereoscopic objects detected by the stereoscopic object detection unit 130 from the bird's-eye view image (step S110).

[0052] Next, the space detection unit 140 transforms the drivable space FS1 to the overhead coordinate system and matches it with the reference mapping to detect the drivable space FS2 on the reference mapping (step S112). Then, the driving control unit 150 generates a target track TT with the vehicle M traversing the drivable space FS2, and causes the vehicle M to travel along the target track TT (step S114). Thus, the processing of this flowchart ends.

[0053] [Completed the generation of Model 162]

[0054] Next, refer to Figure 6 This will explain the specific generation method of Model 162 after learning. Figure 6 This is a diagram representing an example of teaching data from a bird's-eye view image used to generate the learned model 162. Figure 6 The upper part represents the image captured by camera 10. Figure 6 The lower part represents the bird's-eye view image acquired by the bird's-eye view image acquisition unit 120 based on this image.

[0055] exist Figure 6 In the lower bird's-eye view image, the reference numeral A1 indicates the... Figure 6The upper part of the image corresponds to the region O1. Region A1 is a region with radial patterns centered on the lower center O of the bird's-eye view image. Thus, for a region with radial patterns centered on the lower center O of the bird's-eye view image, there is a corresponding annotation indicating that the region is a three-dimensional object, which becomes teaching data. This is because: generally, when a camera image is transformed into a bird's-eye view image, the three-dimensional objects in the camera image have radial patterns as noise due to the interpolation of pixels that accompany the stretching into the bird's-eye view image.

[0056] Moreover, in Figure 6 In the lower bird's-eye view image, the reference numeral A2 indicates the... Figure 6 The area corresponding to the road sign tower O2 in the upper part of the image. Area A2 is a region in the bird's-eye view image with a monochrome pattern that is different from the road surface color. Thus, for the region in the bird's-eye view image with a monochrome pattern that is different from the road surface color, there is a corresponding annotation indicating that the region is a three-dimensional object as teaching data. This is because: generally, when converting a camera image to a bird's-eye view image, a clean three-dimensional object with a monochrome pattern in the camera image sometimes does not have a radial pattern, even when subjected to pixel interpolation caused by the stretching to the bird's-eye view image.

[0057] Moreover, in Figure 6 In the lower bird's-eye view image, reference numeral A3 indicates the... Figure 6 The upper part of the image corresponds to the road marking O3. Region A3 is the area in the bird's-eye view image that corresponds to the road marking. Thus, for the area in the bird's-eye view image that corresponds to the road marking, there is a corresponding annotation indicating that the area is not a three-dimensional object as teaching data. This is because: in general, the area corresponding to the road marking is mostly monochromatic, so it is possible to determine that the area is a three-dimensional object by transforming it into a bird's-eye view image.

[0058] The mobile control device 100 learns the teaching data constructed as described above using methods such as DNN (deep neural network) to generate a learned model 162 that outputs at least the three-dimensional objects in the bird's-eye view image when the bird's-eye view image is input. The mobile control device 100 can also generate the learned model 162 by learning teaching data that also includes annotations indicating whether the vehicle M can travel across the three-dimensional object. In addition to the presence and position of the three-dimensional object, the learned model 162 also outputs information indicating whether it can travel across the three-dimensional object, thereby making it more suitable for generating the target track TT performed by the driving control unit 150.

[0059] Figure 7This diagram illustrates the difference between the near and far areas of vehicle M in a bird's-eye view image. Generally, the number of pixels per unit distance in a camera image varies with the distance from camera 10; that is, the further away an area is from camera 10, the fewer pixels it has. On the other hand, the number of pixels per unit distance in a bird's-eye view image is constant. Therefore, as... Figure 7 As shown, the greater the distance from the vehicle M equipped with camera 10, the more difficult it is to detect three-dimensional objects in the bird's-eye view image with pixel interpolation.

[0060] The learned model 162 is a learned model generated by using a DNN to learn annotated teaching data of various regions in the near and far regions of the vehicle M, and therefore takes into account the aforementioned effects. However, the moving body control device 100 can further set a reliability corresponding to the distance for the region of the bird's-eye view image. In this case, the moving body control device 100 can also determine the presence or absence of a three-dimensional object by performing image recognition processing based on well-known methods (binarization processing, contour extraction processing, image enhancement processing, feature extraction processing, pattern matching processing, or other processing that utilizes the learned model, etc.) on the source image captured by the camera 10 for regions where the set reliability is less than a threshold, without using the information related to the three-dimensional object output by the learned model 162.

[0061] [Detection of hollow objects]

[0062] Figure 8 This diagram illustrates a method for detecting hollow objects in bird's-eye view images. (Example) Figure 6 As shown in the bird's-eye view image, hollow objects, such as the barrier connecting two road signs, may not be detected by the learned model 162 due to their small area in the image. As a result, the spatial detection unit 140 may detect the area between the two road signs as a drivable area, and the driving control unit 150 may generate a target track TT so that the vehicle M travels in the drivable area.

[0063] To address the aforementioned issues, before transforming the image captured by the camera 10 into a bird's-eye view image, the stereo object detection unit 130 detects hollow objects mapped into the image using well-known methods (binarization, contour extraction, image enhancement, feature extraction, pattern matching, or other processing methods utilizing learned models), and applies bounding boxes (BBs) to the detected hollow objects. The bird's-eye view image acquisition unit 120 transforms the camera image, including the hollow objects with attached bounding boxes (BBs), into a bird's-eye view image, obtaining... Figure 8The lower part shows a bird's-eye view image. The space detection unit 140 detects the drivable space FS1 of the vehicle M in the bird's-eye view image by excluding the three-dimensional objects and bounding boxes BB detected by the three-dimensional object detection unit 130 from the bird's-eye view image. Therefore, the drivable space can be detected more accurately in conjunction with the detection based on the learned model 162. The bounding box BB is an example of "recognition information".

[0064] [Detection of 3D Objects Based on Time-Series Displacement]

[0065] Figure 9 This diagram illustrates a method for detecting three-dimensional objects based on the displacement of these objects over a time series from a bird's-eye view image. Figure 9 In the attached diagram, reference numeral A4(t1) indicates a landmark at time point t1, and reference numeral A4(t2) indicates a landmark at time point t2. For example... Figure 9 As shown, for example, due to the shape of the road surface on which the vehicle M travels, there is a possibility that the area of ​​a three-dimensional object in the bird's-eye view image may sway over time. On the other hand, there is a tendency for such swaying to be less pronounced closer to the road surface. Therefore, if the displacement of the same area relative to the road surface detected in multiple bird's-eye view images obtained over time is above a threshold, the three-dimensional object detection unit 130 detects that same area as a three-dimensional object. As a result, the drivable space can be detected more accurately in conjunction with the detection based on the learned model 162.

[0066] Figure 10 This is a flowchart illustrating another example of the process executed by the mobile body control device 100. Regarding... Figure 5 The processing of steps S100, S102, S104, S112, and S114 in the flowchart is as follows: Figure 10 The same process is performed in the flowchart, so the explanation is omitted.

[0067] After performing step S100, the stereoscopic object detection unit 130 detects hollow objects based on the camera image and applies bounding boxes (BB) to the detected hollow objects (step S105). Next, the bird's-eye view image acquisition unit 120 acquires a bird's-eye view image by transforming the camera image with bounding boxes (BB) to the bird's-eye view coordinate system (step S106). For hollow objects in the bird's-eye view image obtained at this time, bounding boxes (BB) are similarly applied, and they have been detected as stereoscopic objects.

[0068] Next, the stereo detection unit 130 detects stereo objects by inputting the bird's-eye view image acquired by the bird's-eye view image acquisition unit 120 into the learned model 162 (step S108). Next, the stereo detection unit 130 measures the displacement of each region based on the previous bird's-eye view image and detects regions with displacement amounts above a threshold as stereo objects (step S109). Next, the space detection unit 140 detects the drivable space FS1 of the vehicle M in the bird's-eye view image by excluding the stereo objects detected by the stereo detection unit 130 from the bird's-eye view image (step S112). Afterwards, the process proceeds to step S112. It should be noted that the processing order of step S108 and step S109 can be reversed, or they can be performed in parallel, or either can be omitted.

[0069] Through the above flowchart processing, the 3D object detection unit 130 detects a 3D object by applying a bounding box BB to a hollow object, and detects 3D objects contained in the bird's-eye view image by inputting the bird's-eye view image into the learned model 162, and detects regions where the displacement based on the previous bird's-eye view image is above a threshold as 3D objects. Therefore, compared to detecting 3D objects using only the learned model 162... Figure 5 Compared to the flowchart, it can detect three-dimensional objects more reliably.

[0070] According to this embodiment as described above, the mobile body control device 100 transforms the image captured by the camera 10 into a bird's-eye view image, and inputs the transformed bird's-eye view image into a learned model 162 that has been trained to identify areas with radial patterns as three-dimensional objects, thereby recognizing three-dimensional objects. Thus, without complicating the hardware structure used for sensing, the drivable space of the mobile body can be detected based on less learning data.

[0071] [Variation Example]

[0072] Figure 1 The vehicle M shown here is equipped with a camera 10 as part of its structure. In particular, in the above embodiment, the camera 10 is positioned at the low position of the vehicle M, specifically at the front bumper. However, generally, the bird's-eye view image derived from an image captured by the camera 10 at a low position tends to have stronger noise compared to a bird's-eye view image derived from an image captured by the camera 10 at a high position. This noise manifests as a radial pattern, which is suitable for detecting three-dimensional objects using the learned model 162, but on the other hand, it makes determining the position of the three-dimensional object more difficult. This modified example addresses this problem.

[0073] Figure 11This is a diagram illustrating an example of the structure of the vehicle M equipped with a modified version of the mobile body control device 100 of the present invention. (See diagram for example.) Figure 11 As shown, the vehicle M includes camera 10A, camera 10B, and a movement control device 100. The hardware structure of cameras 10A and 10B is the same as that of camera 10 in the above-described embodiment. Camera 10A is an example of a "first camera," and camera 10B is an example of a "second camera."

[0074] Camera 10A is mounted on the front bumper of the vehicle M in the same manner as camera 10 described above. Camera 10B is mounted at a higher position than camera 10A, for example, as an in-vehicle camera mounted inside the vehicle M.

[0075] Figure 12 This is a diagram illustrating an example of a bird's-eye view image acquired by the bird's-eye view image acquisition unit 120 based on images captured by cameras 10A and 10B. Figure 12 The left side shows the image captured by camera 10A, and the bird's-eye view image transformed from that image. Figure 12 The right side shows the image captured by camera 10B, and the bird's-eye view image transformed from that image. (Comparison) Figure 12 The left-side bird's-eye view image and Figure 12 As can be seen from the bird's-eye view on the right, the bird's-eye view image corresponding to the camera 10A, which is set at a low position, has stronger noise (i.e., the radial patterns are more pronounced) compared to the bird's-eye view image corresponding to the camera 10B, which is set at a high position, making it more difficult to determine the position of the three-dimensional object.

[0076] Given the above context, the stereo detection unit 130 detects stereo objects by inputting a bird's-eye view image corresponding to camera 10A into the learned model 162. It then uses well-known methods (binarization, contour extraction, image enhancement, feature extraction, pattern matching, or other processing methods utilizing the learned model) to detect objects (not necessarily stereo objects) whose position information is determined in the bird's-eye view image corresponding to camera 10B. Next, the stereo detection unit 130 determines the position of the detected stereo object by matching it with other detected objects. This allows for more accurate detection of drivable space in conjunction with detection based on the learned model 162.

[0077] Figure 13This is a flowchart illustrating another example of the processing flow performed by the modified mobile control device 100. First, the mobile control device 100 acquires an image of the surrounding environment of the vehicle M captured by camera 10A, and an image of the surrounding environment of the vehicle M captured by camera 10B (step S200). Next, the reference mapping generation unit 110 performs image recognition processing on the image captured by camera 10B to identify objects contained in the image (step S202). Then, the reference mapping generation unit 110 transforms the image based on the acquired camera coordinate system to a top-view coordinate system and generates a reference mapping reflecting the position of the identified objects (step S204). Since camera 10B is positioned higher than camera 10A, it can identify objects over a wider area, making camera 10B more suitable for generating the reference mapping.

[0078] Parallel to steps S202 and S204, the following processing is performed: The bird's-eye view image acquisition unit 120 acquires two bird's-eye view images by transforming the images captured by camera 10A and camera 10B into the bird's-eye view coordinate system (step S206). Next, the stereo object detection unit 130 detects stereo objects by inputting the bird's-eye view image corresponding to camera 10A into the learned model 162 (step S208). Next, the stereo object detection unit 130 detects objects whose position information has been determined based on the bird's-eye view image corresponding to camera 10B (step S210). It should be noted that the order of the processing in step S208 and step S210 can be reversed, or the processing in step S208 and step S210 can be performed in parallel.

[0079] Next, the stereoscopic object detection unit 130 determines the position of the stereoscopic object by matching the detected stereoscopic object with the object whose position information has been determined (step S212). Next, the space detection unit 140 detects the drivable space FSi of the vehicle M in the bird's-eye view image by excluding the stereoscopic object detected by the stereoscopic object detection unit 130 from the bird's-eye view image (step S214).

[0080] Next, the space detection unit 40 transforms the drivable space FS1 to the overhead coordinate system and matches it with the reference mapping, thereby detecting the drivable space FS2 on the reference mapping (step S216). Then, the driving control unit 150 generates a target track TT with the vehicle M traversing the drivable space FS2, and causes the vehicle M to travel along the target track TT (step S218). Thus, the processing of this flowchart ends.

[0081] According to this modified example as described above, the moving body control device 100 detects three-dimensional objects based on a bird's-eye view image obtained by transforming an image captured by camera 10A, and determines the position of the three-dimensional object by referring to a bird's-eye view image obtained by transforming an image captured by camera 10B. Therefore, the positions of three-dimensional objects existing around the moving body can be determined more accurately, and the drivable space of the moving body can be detected more accurately.

[0082] The implementation methods described above can be performed as follows.

[0083] A mobile body control device, configured to include:

[0084] Storage device, which stores a program; and

[0085] Hardware processor,

[0086] The hardware processor executes the program stored in the storage device to perform the following processing:

[0087] A bird's-eye view image of the object is obtained by transforming an image of the surrounding conditions of the moving body captured by a camera mounted on the moving body into a bird's-eye view coordinate system;

[0088] The three-dimensional objects in the object's bird's-eye view image are detected by a learned model that has been trained to output at least the three-dimensional objects in the bird's-eye view image when the bird's-eye view image is input.

[0089] Based on the detected three-dimensional object, the drivable space of the moving body can be determined;

[0090] The moving body is made to travel through the drivable space.

[0091] The above description illustrates specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A mobile body control device, wherein, The moving body control device includes: The acquisition unit acquires a bird's-eye view image of the object, which is obtained by transforming an image of the periphery of the moving body taken by a camera mounted on the moving body into a bird's-eye view coordinate system; The stereoscopic object detection unit detects stereoscopic objects in the bird's-eye view image of the object by learning a model that has been trained to output at least the stereoscopic objects in the bird's-eye view image when the bird's-eye view image is input. The space detection unit detects the drivable space of the moving body based on the detected three-dimensional object; as well as A driving control unit that enables the moving body to travel through the drivable space. The learned model is a learned model that was trained based on the following teaching data as teaching data related to the bird's-eye view image. The teaching data includes at least: The first teaching data includes annotations indicating that a region in a bird's-eye view image with radial patterns centered at the lower center of the image is a three-dimensional object; and / or The third set of instructional data includes annotations indicating that the road markings in the bird's-eye view image are non-three-dimensional objects. Before transforming the image of the surrounding environment of the moving object captured by the camera into a bird's-eye view coordinate system, the stereoscopic object detection unit detects hollow objects reflected in the image and assigns recognition information to the hollow objects. The identification information is the bounding box of the hollow object. The space detection unit detects the drivable space by removing the regions corresponding to the detected three-dimensional objects and the regions corresponding to the hollow objects determined based on the bounding boxes from the drivable space in the bird's-eye view image of the object.

2. The moving body control device according to claim 1, wherein, The learned model is a model that has been trained by outputting whether the moving body can travel across three-dimensional objects in the bird's-eye view image when the bird's-eye view image is input.

3. The moving body control device according to claim 1, wherein, The learned model is a learned model that has been learned based on the second teaching data in addition to the first teaching data. In the second teaching data, for areas in the bird's-eye view image that have monochrome patterns that are different from the color of the road surface, there are corresponding annotations indicating that the area is a three-dimensional object.

4. The moving body control device according to any one of claims 1 to 3, wherein, The moving body control device further includes a reference mapping generation unit, which identifies objects contained in the image based on the image captured by the camera of the surrounding environment of the moving body, and generates a reference mapping reflecting the position of the identified objects. The space detection unit detects the drivable space by matching the detected three-dimensional objects in the bird's-eye view image of the object with the generated reference map.

5. The moving body control device according to any one of claims 1 to 3, wherein, The camera includes a first camera disposed below the moving body and a second camera disposed above the moving body. The stereoscopic object detection unit detects the stereoscopic object based on a first object bird's-eye view image, and detects the objects in the second object bird's-eye view image along with their position information based on a second object bird's-eye view image. The unit detects the position of the stereoscopic object by matching the detected stereoscopic object with the detected object having the position information. The first object bird's-eye view image is obtained by transforming the image of the surrounding environment of the moving object captured by the first camera into a bird's-eye view coordinate system, and the second object bird's-eye view image is obtained by transforming the image of the surrounding environment of the moving object captured by the second camera into a bird's-eye view coordinate system.

6. The moving body control device according to any one of claims 1 to 3, wherein, The stereoscopic object detection unit detects the same area as a stereoscopic object when the displacement of the same area relative to the road surface in multiple bird's-eye view images of the object obtained in a time series is above a threshold.

7. A method for controlling a moving body, wherein, The moving body control method causes the computer to perform the following processing: A bird's-eye view image of the object is obtained by transforming an image of the surrounding conditions of the moving body captured by a camera mounted on the moving body into a bird's-eye view coordinate system; The three-dimensional objects in the object's bird's-eye view image are detected by a learned model that has been trained to output at least the three-dimensional objects in the bird's-eye view image when the bird's-eye view image is input. Based on the detected three-dimensional object, the drivable space of the moving body can be determined; The moving body is made to travel through the drivable space. The learned model is a learned model that was trained based on the following teaching data as teaching data related to the bird's-eye view image. The teaching data includes at least: The first teaching data includes annotations indicating that a region in a bird's-eye view image with radial patterns centered at the lower center of the image is a three-dimensional object; and / or The third set of instructional data includes annotations indicating that the road markings in the bird's-eye view image are non-three-dimensional objects. Before transforming the image of the surroundings of the moving object captured by the camera into a bird's-eye view coordinate system, hollow objects projected into the image are detected, and identification information is assigned to the hollow objects. The identification information is the bounding box of the hollow object. The drivable space is detected by removing the regions in the bird's-eye view image of the object that correspond to the detected three-dimensional object and the regions that correspond to the hollow object determined based on the bounding box from the drivable space.

8. A storage medium storing a program, wherein, The program causes the computer to perform the following processing: A bird's-eye view image of the object is obtained by transforming an image of the surrounding conditions of the moving body captured by a camera mounted on the moving body into a bird's-eye view coordinate system; The three-dimensional objects in the object's bird's-eye view image are detected by a learned model that has been trained to output at least the three-dimensional objects in the bird's-eye view image when the bird's-eye view image is input. Based on the detected three-dimensional object, the drivable space of the moving body can be determined; The moving body is made to travel through the drivable space. The learned model is a learned model that was trained based on the following teaching data as teaching data related to the bird's-eye view image. The teaching data includes at least: The first teaching data includes annotations indicating that a region in a bird's-eye view image with radial patterns centered at the lower center of the image is a three-dimensional object; and / or The third set of instructional data includes annotations indicating that the road markings in the bird's-eye view image are non-three-dimensional objects. Before transforming the image of the surroundings of the moving object captured by the camera into a bird's-eye view coordinate system, hollow objects projected into the image are detected, and identification information is assigned to the hollow objects. The identification information is the bounding box of the hollow object. The drivable space is detected by removing the regions in the bird's-eye view image of the object that correspond to the detected three-dimensional object and the regions that correspond to the hollow object determined based on the bounding box from the drivable space.

9. A mobile body control device, wherein, The moving body control device includes: The acquisition unit acquires a bird's-eye view image of the object, which is obtained by transforming an image of the periphery of the moving body taken by a camera mounted on the moving body into a bird's-eye view coordinate system; The stereoscopic object detection unit detects stereoscopic objects in the bird's-eye view image of the object by learning a model that has been trained to output at least the stereoscopic objects in the bird's-eye view image when the bird's-eye view image is input. The space detection unit detects the drivable space of the moving body based on the detected three-dimensional object; as well as A driving control unit that enables the moving body to travel through the drivable space. The camera includes a first camera disposed below the moving body and a second camera disposed above the moving body. The stereoscopic object detection unit detects the stereoscopic object based on a first object bird's-eye view image, and detects the objects in the second object bird's-eye view image along with their position information based on a second object bird's-eye view image. The unit detects the position of the stereoscopic object by matching the detected stereoscopic object with the detected object having the position information. The first object bird's-eye view image is obtained by transforming the image of the surrounding environment of the moving object captured by the first camera into a bird's-eye view coordinate system, and the second object bird's-eye view image is obtained by transforming the image of the surrounding environment of the moving object captured by the second camera into a bird's-eye view coordinate system.

10. A method for controlling a moving body, wherein, The moving body control method causes the computer to perform the following processing: A bird's-eye view image of the object is obtained by transforming an image of the surrounding conditions of the moving body captured by a camera mounted on the moving body into a bird's-eye view coordinate system; The three-dimensional objects in the object's bird's-eye view image are detected by a learned model that has been trained to output at least the three-dimensional objects in the bird's-eye view image when the bird's-eye view image is input. Based on the detected three-dimensional object, the drivable space of the moving body can be determined; The moving body is made to travel through the drivable space. The camera includes a first camera disposed below the moving body and a second camera disposed above the moving body. The stereoscopic object detection unit detects the stereoscopic object based on a first object bird's-eye view image, and detects the objects in the second object bird's-eye view image along with their position information based on a second object bird's-eye view image. The unit detects the position of the stereoscopic object by matching the detected stereoscopic object with the detected object having the position information. The first object bird's-eye view image is obtained by transforming the image of the surrounding environment of the moving object captured by the first camera into a bird's-eye view coordinate system, and the second object bird's-eye view image is obtained by transforming the image of the surrounding environment of the moving object captured by the second camera into a bird's-eye view coordinate system.

11. A storage medium storing a program, wherein, The program causes the computer to perform the following processing: A bird's-eye view image of the object is obtained by transforming an image of the surrounding conditions of the moving body captured by a camera mounted on the moving body into a bird's-eye view coordinate system; The three-dimensional objects in the object's bird's-eye view image are detected by a learned model that has been trained to output at least the three-dimensional objects in the bird's-eye view image when the bird's-eye view image is input. Based on the detected three-dimensional object, the drivable space of the moving body can be determined; The moving body is made to travel through the drivable space. The camera includes a first camera disposed below the moving body and a second camera disposed above the moving body. The stereoscopic object detection unit detects the stereoscopic object based on a first object bird's-eye view image, and detects the objects in the second object bird's-eye view image along with their position information based on a second object bird's-eye view image. The unit detects the position of the stereoscopic object by matching the detected stereoscopic object with the detected object having the position information. The first object bird's-eye view image is obtained by transforming the image of the surrounding environment of the moving object captured by the first camera into a bird's-eye view coordinate system, and the second object bird's-eye view image is obtained by transforming the image of the surrounding environment of the moving object captured by the second camera into a bird's-eye view coordinate system.

12. A learning device, wherein, The learning device learns based on teaching data, in a manner that when an input bird's-eye view image is input, it outputs at least the three-dimensional objects in the bird's-eye view image. In this teaching data, for regions in the bird's-eye view image with radial patterns centered at the lower center of the image, there are annotations indicating that the region is a three-dimensional object. The bird's-eye view image includes a first object bird's-eye view image obtained by transforming an image of the surrounding environment of the moving body captured by a first camera positioned below the moving body into a bird's-eye view coordinate system, and a second object bird's-eye view image obtained by transforming an image of the surrounding environment of the moving body captured by a second camera positioned above the moving body into a bird's-eye view coordinate system. The three-dimensional object is detected based on the first object bird's-eye view image, and the objects in the second object bird's-eye view image are detected together with their position information based on the second object bird's-eye view image. The position of the three-dimensional object is detected by matching the detected three-dimensional object with the detected object having the position information.

13. A learning method, wherein, The learning method enables the computer to perform the following processing: Based on teaching data, the learning process involves outputting at least three-dimensional objects from a bird's-eye view image when that image is input. In this teaching data, regions within the bird's-eye view image exhibiting radial patterns centered at the lower center of the image are annotated to indicate that these regions are three-dimensional objects. The bird's-eye view image includes a first object bird's-eye view image obtained by transforming an image of the surrounding environment of the moving body captured by a first camera positioned below the moving body into a bird's-eye view coordinate system, and a second object bird's-eye view image obtained by transforming an image of the surrounding environment of the moving body captured by a second camera positioned above the moving body into a bird's-eye view coordinate system. The three-dimensional object is detected based on the first object bird's-eye view image, and the objects in the second object bird's-eye view image are detected together with their position information based on the second object bird's-eye view image. The position of the three-dimensional object is detected by matching the detected three-dimensional object with the detected object having the position information.

14. A storage medium storing a program, wherein, The program causes the computer to perform the following processing: Based on teaching data, the learning process involves outputting at least three-dimensional objects from a bird's-eye view image when that image is input. In this teaching data, regions within the bird's-eye view image exhibiting radial patterns centered at the lower center of the image are annotated to indicate that these regions are three-dimensional objects. The bird's-eye view image includes a first object bird's-eye view image obtained by transforming an image of the surrounding environment of the moving body captured by a first camera positioned below the moving body into a bird's-eye view coordinate system, and a second object bird's-eye view image obtained by transforming an image of the surrounding environment of the moving body captured by a second camera positioned above the moving body into a bird's-eye view coordinate system. The three-dimensional object is detected based on the first object bird's-eye view image, and the objects in the second object bird's-eye view image are detected together with their position information based on the second object bird's-eye view image. The position of the three-dimensional object is detected by matching the detected three-dimensional object with the detected object having the position information.

Images