Image region classification method, travel control method, program, image region classification device, and mobile body

Abstract

This system accurately classifies image regions corresponding to 3D point clouds. [Solution] The image region classification method includes a point cloud acquisition step S21 which acquires a 3D point cloud containing distance information from a reference point to distance measurement targets located around the reference point; an image acquisition step S22 which acquires an orthoimage corresponding to the 3D point cloud; a feature extraction step S30 which extracts feature quantities from the 3D point cloud and orthoimage using a trained first machine learning model; and a classification step S41 which uses a trained second machine learning model to classify each region included in the predicted image corresponding to the orthoimage into multiple classes based on the feature quantities extracted in the feature extraction step S30.

Description

Technical Field

[0001] The present invention relates to an image region classification method, a travel control method, a program, an image region classification device, and a moving object.

Background Art

[0002] Currently, research on automated driving of moving objects such as automobiles is being actively advanced. As a technology necessary for realizing automated driving, detection of a travelable region around the host vehicle can be cited. The travelable region means a region obtained by excluding a blind spot region caused by obstacles from the travel path on which the host vehicle can pass. In order to realize this technology, it is necessary to detect obstacles around the host vehicle using data obtained by sensors such as cameras and LiDAR (Light Detection and Ranging), and accurately detect the road region which is the other region. As a major premise for realizing the detection of the travelable region in this way, it is necessary to accurately detect the road region.

[0003] For example, a known technology assigns type information to point groups within a predetermined region of a three-dimensional point group by performing processing such as semantic segmentation on the three-dimensional point group obtained by LiDAR used in automated driving (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0005] However, in three-dimensional point groups, data becomes sparse particularly in regions at a large distance from LiDAR. For this reason, when semantic segmentation is performed using a machine learning model, it is difficult to increase the estimation accuracy of the machine learning model.

[0006] This invention has been made in view of the above problems, and aims to accurately classify the regions of an image corresponding to a three-dimensional point cloud. [Means for solving the problem]

[0007] To achieve the above objective, an image region classification method according to one aspect of the present invention is an image region classification method performed by a computer, comprising: a point cloud acquisition step of acquiring a 3D point cloud including distance information from a reference point to distance measurement targets located around the reference point; an image acquisition step of acquiring an orthoimage corresponding to the 3D point cloud; a feature extraction step of extracting feature quantities from the 3D point cloud and the orthoimage using a trained first machine learning model; and a classification step of classifying each region included in a predicted image corresponding to the orthoimage into a plurality of classes based on the feature quantities extracted in the feature extraction step using a trained second machine learning model.

[0008] Furthermore, in order to achieve the above objective, a driving control method according to one aspect of the present invention includes the above-described image region classification method and a driving control step of performing driving control of a moving body based on the drivable region classified by the image region classification method.

[0009] Furthermore, in order to achieve the above objective, a program according to one aspect of the present invention is a program for causing a computer to execute the above-described image region classification method.

[0010] Furthermore, in order to achieve the above objective, an image region classification device according to one aspect of the present invention includes: a point cloud acquisition unit that acquires a 3D point cloud including distance information from a reference point to distance measurement targets located around the reference point; an image acquisition unit that acquires an orthoimage corresponding to the 3D point cloud; a feature extraction unit that extracts feature quantities from the 3D point cloud and the orthoimage using a trained first machine learning model; and a classification unit that uses a trained second machine learning model to classify each region included in the predicted image corresponding to the orthoimage into a plurality of classes based on the feature quantities extracted by the feature extraction unit.

[0011] Furthermore, in order to achieve the above objective, a mobile body according to one aspect of the present invention comprises an image region classification device and a travel control unit that controls the travel of the mobile body based on the travelable region classified by the image region classification device.

[0012] These comprehensive or specific embodiments may be implemented as a system, method, integrated circuit, computer program, or recording medium such as a non-temporary computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium. [Effects of the Invention]

[0013] According to the image region classification method of the present invention, it is possible to accurately classify the image regions corresponding to a 3D point cloud. [Brief explanation of the drawing]

[0014] [Figure 1] This is a block diagram showing an example of the configuration of an image region classification device and a mobile body according to Embodiment 1. [Figure 2] This is a block diagram showing an example of the configuration of the feature extraction unit included in the image region classification device according to Embodiment 1. [Figure 3] This figure shows an example of a 3D point cloud acquired by the point cloud acquisition unit according to Embodiment 1. [Figure 4] This figure shows an example of an orthomosaic image acquired by the image acquisition unit according to Embodiment 1. [Figure 5] This is a schematic diagram showing a three-dimensional point cloud divided into columnar sections. [Figure 6] This is a schematic diagram showing an example of the first image according to Embodiment 1. [Figure 7] This diagram illustrates the processing in the second conversion unit according to Embodiment 1. [Figure 8] This diagram illustrates the processing at the joint according to Embodiment 1. [Figure 9]This is a diagram for explaining processing in the two-dimensional convolution unit according to Embodiment 1. [Figure 10] This is a diagram for explaining processing in the classification unit according to Embodiment 1. [Figure 11] This is a diagram showing an example of a predicted image obtained by the classification unit according to Embodiment 1. [Figure 12] This is a diagram showing an example of a correct image that is teacher data of a predicted image used in learning of the classification unit and the like according to Embodiment 1. [Figure 13] This is a diagram showing positions of one or more moving objects detected by the detection unit according to Embodiment 1. [Figure 14] This is a flowchart showing the image region classification method and the travel control method according to Embodiment 1. [Figure 15] This is a flowchart showing the feature extraction step according to Embodiment 1. [Figure 16] This is a diagram showing an example of weights assigned to teacher data of each class in the learning step according to Embodiment 1. [Figure 17] This is a diagram showing the relationship between the presence or absence of weighting in the learning step according to Embodiment 1 and the IoU of the classification result for each region of the predicted image. [Figure 18] This is a diagram showing the relationship between the presence or absence of a moving object class in classification of each region of a predicted image according to Embodiment 2 and the IoU of the classification result for each region of the predicted image. [Figure 19] This is a diagram showing Precision and Recall of the classification result for each region of a predicted image according to Embodiment 2. [Figure 20] This is a diagram showing an example of a predicted image obtained by the image region classification device and the image region classification method according to Embodiment 2. [Figure 21] This is a diagram showing a correct image corresponding to the predicted image shown in FIG. 20. [Figure 22] This is a diagram showing another example of a predicted image obtained by the image region classification device and the image region classification method according to Embodiment 2. [Figure 23]This figure shows the ground truth image corresponding to the predicted image shown in Figure 22. [Modes for carrying out the invention]

[0015] Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings.

[0016] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components.

[0017] Furthermore, each figure is a schematic diagram and not necessarily a strictly accurate representation. Also, the same component is denoted by the same reference numeral in each figure.

[0018] (Embodiment 1) The image region classification method and the like according to Embodiment 1 will be described.

[0019] [Configuration of the image region classification device and mobile unit] The configuration of the image region classification device and mobile body according to this embodiment will be explained with reference to Figures 1 and 2. Figure 1 is a block diagram showing an example of the configuration of the image region classification device 100 and mobile body V10 according to this embodiment. Figure 2 is a block diagram showing an example of the configuration of the feature extraction unit 30 included in the image region classification device 100 according to this embodiment. Figure 2 also shows the point cloud acquisition unit 21, image acquisition unit 22, classification unit 41, and detection unit 42 included in the image region classification device 100.

[0020] The mobile body V10 is a moving object comprising an image region classification device 100 and a driving control unit 50. In this embodiment, the mobile body V10 further comprises a measurement unit 10 and a storage unit 60. The mobile body V10 is not particularly limited as long as it is a mobile body that can be at least partially controlled by the driving control unit 50. In this embodiment, the mobile body V10 is a vehicle that performs autonomous driving.

[0021] The image region classification device 100 is a device that classifies each region contained in an image. In this embodiment, the image region classification device 100 also detects moving objects contained in a 3D point cloud. As shown in Figure 1, the image region classification device 100 comprises a point cloud acquisition unit 21, an image acquisition unit 22, a feature extraction unit 30, a classification unit 41, and a detection unit 42. The image region classification device 100 may also include a measurement unit 10.

[0022] The measurement unit 10 is a sensor that acquires a three-dimensional point cloud containing distance information from a reference point to distance measurement targets located around the reference point. In this embodiment, the reference point is included in the mobile body V10. That is, the reference point is located on the mobile body V10 and can move together with the mobile body V10. More specifically, the reference point corresponds to a measurement reference point, which is the position of the sensor of the measurement unit 10 on the mobile body V10. Distance measurement targets include roads, sidewalks, buildings, and mobile bodies located around the reference point. Mobile bodies include, for example, vehicles such as automobiles and motorcycles, and pedestrians. Other movable objects may be included in the mobile bodies located around the reference point.

[0023] In this embodiment, the measurement unit 10 sequentially generates a plurality of 3D point clouds, which are time-series information at different points in time. Using a reference point as the measurement reference point, the measurement unit 10 measures 3D information as distance information, including the direction (horizontal direction) in which the object to be measured is located, the distance to the surface of the object to be measured, and the height. For example, a LiDAR can be used as the measurement unit 10. In this embodiment, the measurement unit 10 includes a LiDAR. The measurement unit 10 may also include a system for estimating the position and attitude angle of the moving object V10. For example, the measurement unit 10 may include a GNSS / INS (Global Navigation Satellite System / Internal Navigation System) combined navigation system.

[0024] LiDAR measures the distance between an object and a measurement reference point included in the LiDAR by irradiating the object with laser light and acquiring the laser light reflected from the object. The measurement unit 10 acquires 3D point cloud data showing the relationship between the horizontal position and height of objects around the measurement reference point from the measured distance, the position of the LiDAR, and the direction of laser light irradiation.

[0025] LiDAR, for example, emits 128 laser beams vertically (i.e., in the upright direction) at different irradiation angles. The vertical field of view of LiDAR is, for example, between -25 degrees and 15 degrees, with the horizontal direction (i.e., the direction perpendicular to the vertical direction) being 0 degrees. LiDAR can acquire 3D point cloud data in all directions by rotating the direction of these laser beams 360 degrees horizontally. The scan rate of the laser beams is, for example, between 2 Hz and 20 Hz. In this embodiment, the scan rate is set to 10 Hz. In other words, in this embodiment, LiDAR acquires 10 frames of 3D point cloud data per second.

[0026] The point cloud acquisition unit 21 is a processing unit that acquires a 3D point cloud containing distance information from a reference point to distance measurement targets located around the reference point. In this embodiment, the point cloud acquisition unit 21 acquires a 3D point cloud from the measurement unit 10. The point cloud acquisition unit 21 may acquire all of the 3D point cloud acquired by the measurement unit 10, or it may acquire only a part of it. In this embodiment, the point cloud acquisition unit 21 acquires 10 frames of 3D point cloud per second. An example of a 3D point cloud acquired by the point cloud acquisition unit 21 will be explained using Figure 3. Figure 3 is a diagram showing an example of a 3D point cloud acquired by the point cloud acquisition unit 21 according to this embodiment. As shown in Figure 3, the 3D point cloud according to this embodiment includes a plurality of points indicating the 3D positions of measurement targets on distance measurement targets.

[0027] The image acquisition unit 22 is a processing unit that acquires orthophotos corresponding to 3D point clouds. An orthophoto is an image obtained by orthorectifying an aerial photograph, satellite image, 3D point cloud, etc., and correcting for distortion. Orthophotos are also called orthographic (photographic) images, orthophotos, orthoimages. An orthophoto is an image without tilt, without positional displacement, and with correct positional information, as if the ground were viewed from directly above. An orthophoto corresponding to a 3D point cloud is an orthophoto that represents an area corresponding to at least a part of the area indicated by the 3D point cloud. Orthophotos may be stored in advance in the storage unit 60, for example. The orthophoto used in this embodiment will be explained with reference to Figure 4. Figure 4 is a diagram showing an example of an orthophoto acquired by the image acquisition unit 22 according to this embodiment. In this embodiment, the orthophoto is an image generated by converting a 3D point cloud acquired in advance using LiDAR or the like. The orthophoto is, for example, an image (in other words, a map) of the road surface on which the moving object V10 is located, viewed vertically from above. Multiple 3D point cloud data are used to generate a single orthomosaic image. The image acquisition unit 22 acquires an orthomosaic image corresponding to the 3D point cloud acquired by the point cloud acquisition unit 21 from the storage unit 60, based on, for example, the position information of the mobile body V10 measured by the measurement unit 10. The image acquisition unit 22 may acquire the orthomosaic image by other methods. For example, the orthomosaic image may be acquired from outside the mobile body V10 using wireless communication or the like.

[0028] The feature extraction unit 30 is a processing unit that extracts feature quantities from the 3D point cloud and orthoimage using a pre-trained first machine learning model. Feature quantities are variables that quantitatively represent the characteristics of the input data.

[0029] The first machine learning model receives a 3D point cloud and an orthophoto as input, extracts features from the 3D point cloud and orthophoto, and outputs the extracted features. The first machine learning model may be a deep learning model that performs deep learning. As shown in Figure 2, the feature extraction unit 30 in this embodiment has a first transformation unit 31, a second transformation unit 32, a coupling unit 33, and a 2D convolution unit 34.

[0030] The first conversion unit 31 shown in Figure 2 is a processing unit that converts a 3D point cloud input from the point cloud acquisition unit 21 into a 2D image, which is the first image. The processing in the first conversion unit 31 will be explained using Figures 5 and 6. Figure 5 is a schematic diagram showing a 3D point cloud divided into columnar shapes. Figure 6 is a schematic diagram showing an example of the first image according to this embodiment.

[0031] The first transformation unit 31 first divides the three-dimensional point cloud, as shown in Figure 3, into columnar sections based on two-dimensional positions, as shown in Figure 5. Subsequently, the first transformation unit 31 extracts feature quantities from each divided columnar section. Examples of extracted feature quantities include the coordinate information of the point cloud, reflection intensity, relative coordinates and height distribution (maximum height, minimum height, average height, etc.) of points within the columnar section, and point density.

[0032] The feature quantities corresponding to each 2D position in the 3D point cloud, extracted in this manner, are associated with those 2D positions by the first transformation unit 31. As a result, the first transformation unit 31 creates a first image, which is a 2D image of size (H, W, C) as shown in Figure 6. Here, a 2D image refers to a set of information containing feature quantities corresponding to each 2D position, and does not necessarily have to be a visible image. For example, the first image contains feature quantities corresponding to each position on a 2D plane extending horizontally. In the first transformation unit 31 according to this embodiment, a method employed in PointPillars' Pillar Feature Net, which is an example of an object detection processing model from a 3D point cloud, may be used. Alternatively, a pseudo-image used in PointPillars may be used as the first image according to this embodiment. H and W, which indicate the size of the first image, represent the vertical and horizontal dimensions of the 2D image, respectively, and C represents the feature quantity (in other words, the number of channels).

[0033] The second conversion unit 32 shown in Figure 2 is a processing unit that converts an orthophoto into a second image, which is a two-dimensional image. The processing in the second conversion unit 32 will be explained with reference to Figure 7. Figure 7 is a diagram illustrating the processing in the second conversion unit 32 according to this embodiment.

[0034] As shown in Figure 7, the second transformation unit 32 repeatedly performs convolution on the orthoimage. This allows the second transformation unit 32 to generate a second image, which is a two-dimensional image with the input size (H,W) of the orthoimage and the number of channels C. In the second image, the feature quantities corresponding to each two-dimensional position in the orthoimage are associated with those two-dimensional positions. Examples of feature quantities corresponding to each two-dimensional position include the coordinates of points included in the second image.

[0035] The coupling unit 33 shown in Figure 2 is a processing unit that combines the first image and the second image. The processing in the coupling unit 33 will be explained using Figure 8. Figure 8 is a diagram illustrating the processing in the coupling unit 33 according to this embodiment.

[0036] As shown in Figure 8, the merging unit 33 generates a combined image, which is a 2D image of size (H, W, 2C), by combining the first feature quantities, which are feature quantities corresponding to each 2D position of the first image, and the second feature quantities, which are feature quantities corresponding to each 2D position of the second image. Here, the first and second images are combined such that the absolute positions indicated by each 2D position of the first image match the absolute positions indicated by each 2D position of the second image. Next, the merging unit 33 integrates the first and second images by performing convolution on the combined first and second images. In other words, the merging unit 33 performs convolution on the combined first and second feature quantities. As a result, the merging unit 33 generates a unified image, which is a 2D image of size (H, W, C).

[0037] The two-dimensional convolution unit 34 shown in Figure 2 is a processing unit that extracts features by performing two-dimensional convolution on the integrated image generated by the merging unit 33. The processing in the two-dimensional convolution unit 34 will be explained with reference to Figure 9. Figure 9 is a diagram illustrating the processing in the two-dimensional convolution unit 34 according to this embodiment.

[0038] As shown in Figure 9, the two-dimensional convolution unit 34 performs convolution to convert the integrated image, which is a two-dimensional image of size (H,W,C), into a two-dimensional image of size (H / 2,W / 2,C). Subsequently, by performing further convolution, the two-dimensional image of size (H / 2,W / 2,C) is converted into a two-dimensional image of size (H / 4,W / 4,2C). Subsequently, by performing further convolution, the two-dimensional image of size (H / 4,W / 4,2C) is converted into a two-dimensional image of size (H / 8,W / 8,4C). Subsequently, by performing transpose convolution, each of the two-dimensional images of size (H / 2,W / 2,C), (H / 4,W / 4,2C), and (H / 8,W / 8,4C) is converted into a two-dimensional image of size (H / 2,W / 2,2C). Finally, three 2D images of size (H / 2, W / 2, 2C) are combined to create a 2D image of size (H / 2, W / 2, 3×2C). Transposed convolution is a process that expands the spatial size of the input data by performing convolution while expanding the input data.

[0039] In the feature extraction process described above, for example, the method used in the PointPillars Backbone mentioned above may be used. In other words, the PointPillars Backbone may be used in the 2D convolution section 34.

[0040] As shown in Figure 2, the two-dimensional convolution unit 34 outputs a two-dimensional image of size (H / 2, W / 2, 3×2C) to the classification unit 41 and the detection unit 42.

[0041] The classification unit 41 shown in Figures 1 and 2 is a processing unit that uses a trained second machine learning model to classify each region included in the predicted image corresponding to the orthomosaic image into multiple classes based on the features extracted by the feature extraction unit 30. The second machine learning model receives the features extracted by the feature extraction unit 30 as input and outputs information indicating the class of each region included in the predicted image. The second machine learning model may be a deep learning model that performs deep learning. In this embodiment, a two-dimensional image containing features is input to the second machine learning model. The multiple classes classified by the classification unit 41 include drivable regions, which are regions that the mobile body V10, including a reference point, can travel through.

[0042] The processing in the classification unit 41 will be explained using Figure 10. Figure 10 is a diagram illustrating the processing in the classification unit 41 according to this embodiment. As shown in Figure 10, the classification unit 41 performs convolution on a two-dimensional image of size (H / 2,W / 2,3×2C) generated by the two-dimensional convolution unit 34, thereby converting a two-dimensional image of size (H / 2,W / 2,3×2C) into a two-dimensional image of size (H / 2,W / 2,C). Furthermore, the classification unit 41 performs convolution on the two-dimensional image of size (H / 2,W / 2,C), thereby converting a two-dimensional image of size (H / 2,W / 2,C) into a two-dimensional image of size (H / 2,W / 2,C / 4). In this way, the classification unit 41 reduces the number of channels by performing multiple convolutions on the 2D image of size (H / 2, W / 2, 3×2C) generated by the 2D convolution unit 34, and outputs a predicted image which is an image semantically segmented pixel by pixel.

[0043] The number of convolutions in the classification unit 41 is, for example, three or less. If the number of convolutions is too high, the number of parameters increases, which may hinder the learning of parameters for extracting the first image from the 3D point cloud in the first transformation unit 31. By limiting the number of convolutions in the classification unit 41 to three or less, it becomes possible to reliably advance the learning of parameters in the first transformation unit 31.

[0044] The predicted images obtained by the classification unit 41 will be explained using Figures 11 and 12. Figure 11 is a diagram showing an example of a predicted image obtained by the classification unit 41 according to this embodiment. Figure 12 is a diagram showing an example of a ground truth image, which is the training data for the predicted images used in the learning of the classification unit 41 according to this embodiment. Figure 12 shows a ground truth image that indicates the region corresponding to the predicted image shown in Figure 11. Each of the predicted image shown in Figure 11 and the ground truth image shown in Figure 12 is an image corresponding to an orthomosaic image. Each region of the predicted image and the ground truth image is classified into classes. In the example shown in Figures 11 and 12, each region of the predicted image and the ground truth image is classified into five classes: road, sidewalk, median strip, vegetation, and unlabeled (i.e., a class representing a region that is not classified into any other class). Here, a road is an example of a drivable region, which is the region on which a mobile body V10 including a reference point can travel. The class classification in the predicted image in Figure 11 is almost identical to the class classification in the ground truth image shown in Figure 12. Thus, the image region classification device 100 according to this embodiment can accurately classify each region of a predicted image.

[0045] The detection unit 42 shown in Figures 1 and 2 is a processing unit that uses a trained third machine learning model to detect moving objects included in a 3D point cloud based on the features extracted by the feature extraction unit 30. Here, the moving objects are included in the distance measurement targets located around the reference point. The detection unit 42 extracts the features from the 2D image output by the feature extraction unit 30 using the trained third machine learning model. The third machine learning model receives the features extracted by the feature extraction unit 30 as input and outputs the positions of one or more moving objects located around the reference point. The third machine learning model may be a deep learning model that performs deep learning. In this embodiment, the third machine learning model receives a 2D image containing the features as input.

[0046] Specifically, the detection unit 42 detects the position of the moving object by performing convolution on the 2D image output by the feature extraction unit 30. The detection unit 42 may use, for example, the method employed in the PointPillars Detection Head described above. In other words, the PointPillars Detection Head may be used in the third machine learning model. In this embodiment, the detection unit 42 estimates the 3D coordinates as the position of one or more moving objects. The detection unit 42 may also estimate the size, tilt (yaw angle), and class of one or more moving objects.

[0047] An example of the positions of one or more moving objects detected by the detection unit 42 will be explained using Figure 13. Figure 13 is a diagram showing the positions of one or more moving objects V1 detected by the detection unit 42 according to this embodiment. In Figure 13, the positions of the one or more detected moving objects V1 are shown as 3D boxes in the 3D point cloud. A 3D box is a cuboid-shaped virtual box that indicates the position and size of an object. The size of the 3D box is such that it can contain the corresponding object. In this way, the detection unit 42 can detect the positions of one or more moving objects included in the 3D point cloud.

[0048] The travel control unit 50 controls the movement of the mobile body V10. The travel control unit 50 controls the movement of the mobile body V10 based on the drivable area classified by the image area classification device 100. In this embodiment, the travel control unit 50 also controls the movement of the mobile body V10 based on the position of the mobile body detected by the detection unit 42. For example, the travel control unit 50 may control the movement of the mobile body V10 to travel within the drivable area, or it may control the movement of the mobile body V10 to avoid interference with other mobile bodies.

[0049] The storage unit 60 stores information for at least one of the image region classification device 100 and the driving control unit 50. In this embodiment, the storage unit 60 stores orthomosaic images, map information including information about the roads driven by the mobile vehicle V10, and the like.

[0050] [Configuration of image region classification method and driving control method] The configuration of the image region classification method and the travel control method according to this embodiment will be explained with reference to Figure 14. Figure 14 is a flowchart of the image region classification method and the travel control method according to this embodiment. Below, the image region classification method using the image region classification device 100 and the travel control method using the mobile body V10 according to this embodiment will be explained.

[0051] The image region classification method according to this embodiment is a method for classifying each region contained in an image, and is executed by a computer. In this embodiment, the image region classification method also detects moving objects contained in a 3D point cloud.

[0052] As shown in Figure 14, the image region classification method according to this embodiment includes a point cloud acquisition step S21, an image acquisition step S22, a feature extraction step S30, a classification step S41, and a detection step S42. The image region classification method may also include a learning step S10. The driving control method according to this embodiment includes each step included in the image region classification method and a driving control step S50.

[0053] As shown in Figure 14, in the image region classification method and driving control method according to this embodiment, first, the first machine learning model and the second machine learning model of the image region classification device 100 are trained (training step S10). For example, the first and second machine learning models are trained using a 3D point cloud, an orthoimage, and information indicating the class of each region of the image corresponding to the orthoimage as training data. In this embodiment, in training step S10, the third machine learning model of the image region classification device 100 is also trained. For example, the first, second, and third machine learning models are trained using a 3D point cloud, an orthoimage, information indicating the class of each region of the image corresponding to the orthoimage, and information indicating the position, size, and yaw angle of the moving object in the 3D point cloud as training data. This makes it possible to prepare the first, second, and third machine learning models, as well as the first, second, and third machine learning models, which have already been trained.

[0054] Furthermore, in this embodiment, in learning step S10, weights are applied to the training data for each of the multiple classes based on the area ratio of each class in the training data of the predicted image. This suppresses the decrease in classification accuracy for classes with small area ratios.

[0055] Next, the point cloud acquisition unit 21 of the image region classification device 100 acquires a 3D point cloud that includes distance information from the reference point to distance measurement targets located around the reference point (point cloud acquisition step S21). In point cloud acquisition step S21, the measurement unit 10 may generate a 3D point cloud by measuring distance information from the reference point to distance measurement targets, and the point cloud acquisition unit 21 may acquire this 3D point cloud. In this embodiment, the measurement unit 10 sequentially generates a plurality of 3D point clouds, which are time-series information at different points in time, and the point cloud acquisition unit 21 sequentially acquires these plurality of 3D point clouds. In this embodiment, distance measurement targets include moving objects located around the reference point.

[0056] Next, the image acquisition unit 22 of the image region classification device 100 acquires an orthomosaic image corresponding to the 3D point cloud (image acquisition step S22). In image acquisition step S22, the image acquisition unit 22 acquires an orthomosaic image corresponding to the 3D point cloud acquired by the point cloud acquisition unit 21 from the storage unit 60, based on, for example, the position information of the moving object V10 measured by the measurement unit 10.

[0057] The timing of the execution of the point cloud acquisition step S21 and the image acquisition step S22 is not particularly limited. The point cloud acquisition step S21 and the image acquisition step S22 may be executed in either order, or they may be executed simultaneously.

[0058] Next, the feature extraction unit 30 of the image region classification device 100 extracts features from the 3D point cloud and orthoimage using a pre-trained first machine learning model (feature extraction step S30). The feature extraction step S30 according to this embodiment will be explained with reference to Figure 15. Figure 15 is a flowchart of the feature extraction step S30 according to this embodiment. As shown in Figure 15, the feature extraction step S30 includes a first transformation step S31, a second transformation step S32, a merging step S33, and a 2D convolution step S34.

[0059] As shown in Figure 15, in the feature extraction step S30, the first transformation unit 31 of the feature extraction unit 30 transforms the 3D point cloud into a 2D image, which is the first image (first transformation step S31). As described in the explanation of the first transformation unit 31, in the first image, the feature quantities corresponding to each 2D position of the 3D point cloud are associated with those 2D positions.

[0060] Next, the second transformation unit 32 of the feature extraction unit 30 transforms the orthoimage into a second image, which is a two-dimensional image (second transformation step S32). As described in the explanation of the second transformation unit 32, in the second image, the feature quantities corresponding to each two-dimensional position in the orthoimage are associated with those two-dimensional positions.

[0061] The timing of the execution of the first conversion step S31 and the second conversion step S32 is not particularly limited. The first conversion step S31 and the second conversion step S32 may be executed in either order, or they may be executed simultaneously.

[0062] Next, the merging unit 33 of the feature extraction unit 30 merges the first image and the second image (merging step S33). In merging step S33, the first feature quantities, which are feature quantities corresponding to each 2D position of the first image, and the second feature quantities, which are feature quantities corresponding to each 2D position of the second image, are merged, and convolution is performed on the merged first feature quantities and second feature quantities. In this embodiment, in merging step S33, the merging unit 33 generates a merged image by merging the first feature quantities and second feature quantities, and generates an integrated image by performing convolution on the merged image.

[0063] Next, the two-dimensional convolution unit 34 of the feature extraction unit 30 extracts features by performing two-dimensional convolution on the integrated image generated in the merging step S33 (two-dimensional convolution step S34). In the two-dimensional convolution step S34 according to this embodiment, a two-dimensional image is generated that includes the extracted features by performing convolution and transpose convolution on the integrated image.

[0064] Next, as shown in Figure 14, the classification unit 41 of the image region classification device 100 uses a trained second machine learning model to classify each region included in the predicted image corresponding to the orthomosaic image into multiple classes based on the features extracted in the feature extraction step S30 (classification step S41). The multiple classes include drivable regions, which are areas that a mobile body V10 including a reference point can travel through. In this embodiment, the multiple classes include roads as an example of drivable regions.

[0065] Next, the detection unit 42 of the image region classification device 100 uses a trained third machine learning model to detect moving objects included in the 3D point cloud based on the features extracted in the feature extraction step S30 (detection step S42).

[0066] The timing of the execution of the classification step S41 and the detection step S42 is not particularly limited. The classification step S41 and the detection step S42 may be executed in either order, or they may be executed simultaneously.

[0067] Next, in the driving control method according to this embodiment, the driving control unit 50 of the mobile body V10 controls the driving of the mobile body V10 based on the drivable area classified by the image area classification method according to this embodiment (driving control step S50). In this embodiment, in driving control step S50, the driving control unit 50 also controls the driving of the mobile body V10 based on the position of the mobile body detected by the detection unit 42.

[0068] Next, the process returns to the point cloud acquisition step S21 and the same process is repeated. Note that the learning step S10 may be repeated two or more times using different training data.

[0069] As described above, the image region classification method according to this embodiment can accurately classify each region of the predicted image corresponding to the orthomosaic image. Furthermore, the image region classification method according to this embodiment can detect the positions of one or more moving objects included in the 3D point cloud.

[0070] Furthermore, the travel control method according to this embodiment can be used to control the movement of the mobile body V10.

[0071] [effect] The effects of the image region classification device 100 and the image region classification method according to this embodiment will be described.

[0072] As described above, the image region classification method according to this embodiment is an image region classification method performed by a computer, and includes a point cloud acquisition step S21 in which a 3D point cloud is acquired that includes distance information from a reference point to distance measurement targets located around the reference point; an image acquisition step S22 in which an orthoimage corresponding to the 3D point cloud is acquired; a feature extraction step S30 in which features are extracted from the 3D point cloud and the orthoimage using a trained first machine learning model; and a classification step S41 in which each region included in the predicted image corresponding to the orthoimage is classified into a plurality of classes based on the features extracted in the feature extraction step S30 using a trained second machine learning model.

[0073] Thus, in the image region classification method according to this embodiment, by using orthomosaic images in addition to 3D point clouds, it is possible to combine the dense data of orthomosaic images with the sparse data of 3D point clouds. This allows for efficient training of the first and second machine learning models, and thus enables accurate classification of the class of each region included in the predicted image corresponding to the 3D point cloud.

[0074] Furthermore, the image region classification device 100 according to this embodiment includes a point cloud acquisition unit 21 that acquires a 3D point cloud including distance information from a reference point to distance measurement targets located around the reference point, an image acquisition unit 22 that acquires orthoimages corresponding to the 3D point cloud, a feature extraction unit 30 that extracts feature quantities from the 3D point cloud and orthoimages using a trained first machine learning model, and a classification unit 41 that uses a trained second machine learning model to classify each region included in the predicted image corresponding to the orthoimage into a plurality of classes based on the feature quantities extracted by the feature extraction unit 30.

[0075] This image region classification device 100 achieves the same effect as the image region classification method described above.

[0076] Furthermore, the multiple classes according to this embodiment may include a drivable area which is an area on which a mobile body V10 including a reference point can travel.

[0077] This information about the drivable area can be used, for example, to control the movement of a mobile object V10 that includes a reference point.

[0078] Furthermore, the objects to be measured for distance according to this embodiment may include moving objects located around a reference point. The image region classification method according to this embodiment may include a detection step S42 that detects moving objects included in the 3D point cloud based on the feature quantities extracted in the feature extraction step S30 using a trained third machine learning model. Furthermore, the image region classification device 100 according to this embodiment may include a detection unit 42 that detects moving objects included in the 3D point cloud based on the feature quantities extracted in the feature extraction unit 30 using a trained third machine learning model.

[0079] This allows for the detection of moving objects located around a reference point. Information such as the position of these moving objects can be used, for example, to control the movement of the moving object V10 that includes the reference point. In other words, it is possible to classify the area around the moving object V10 while detecting the position information of other moving objects required by the autonomously driving moving object V10.

[0080] Furthermore, the feature extraction step S30 according to this embodiment includes a first transformation step S31 that converts a 3D point cloud into a first image which is a 2D image, a second transformation step S32 that converts an orthoimage into a second image which is a 2D image, and a merging step S33 that combines the first image and the second image, wherein in the first image, first feature quantities, which are feature quantities corresponding to each 2D position of the 3D point cloud, are associated with the 2D position, and in the second image, second feature quantities, which are feature quantities corresponding to each 2D position of the orthoimage, are associated with the 2D position, and in the merging step S33, the first feature quantities and the second feature quantities are combined, and convolution may be performed on the combined first feature quantities and second feature quantities.

[0081] Furthermore, the feature extraction unit 30 according to this embodiment includes a first transformation unit 31 that converts a 3D point cloud into a first image which is a 2D image, a second transformation unit 32 that converts an orthoimage into a second image which is a 2D image, and a merging unit 33 that combines the first image and the second image. In the first image, first feature quantities, which are feature quantities corresponding to each 2D position of the 3D point cloud, are associated with the 2D position. In the second image, second feature quantities, which are feature quantities corresponding to each 2D position of the orthoimage, are associated with the 2D position. The merging unit 33 combines the first feature quantities and the second feature quantities, and may perform convolution on the combined first and second feature quantities.

[0082] In this way, by performing convolution on the first and second features, the two features are integrated. Therefore, it is possible to suppress the estimation of the image region class by relying on only one of the first or second features.

[0083] Furthermore, in the learning step S10 of the image region classification method according to this embodiment, weights may be applied to the training data for each of the multiple classes based on the area ratio of each of the multiple classes in the training data of the predicted image.

[0084] The effect of this weighting will be explained using Figures 16 and 17. Figure 16 is a diagram showing an example of the weights assigned to the training data for each class in the learning step S10 according to this embodiment. Figure 17 is a diagram showing the relationship between the presence or absence of weighting in the learning step S10 according to this embodiment and the IoU (Intersection over Union) of the classification result for each region of the predicted image. Figure 17 also shows the average IoU (mIoU) for all classes.

[0085] IoU is a metric defined as the ratio of the area of ​​the intersection of the region indicated by the ground truth label and the region indicated by the predicted label to the area of ​​the union of the regions indicated by the ground truth label and the region indicated by the predicted label. A larger IoU indicates higher prediction accuracy.

[0086] In an example of image classification using the image region classification method and image region classification device 100 of this embodiment, weights were assigned to the training data for each of the multiple classes based on the area ratio of each of the multiple classes in the training data of the predicted image, as shown in Figure 16. In this embodiment, the larger the area ratio of each of the multiple classes in the training data of the predicted image, the smaller the weight assigned to the training data for that class. For example, in the training data used in this embodiment, the area of ​​the unlabeled class is the largest, and the area of ​​the vegetation class is the smallest. Therefore, as shown in Figure 16, the weight for the unlabeled data in the training data is the smallest, and the weight for the vegetation data is the largest.

[0087] The presence or absence of such weighting results in the classification IoU of the predicted image. As shown in Figure 17, the average IoU can be increased when weighting is applied compared to when no weighting is applied. Comparing the IoU of each class, the IoU of the unlabeled image, which has the smallest weight, is lower when weighting is applied compared to when no weighting is applied. However, for other classes, the IoU is higher when weighting is applied compared to when no weighting is applied. Thus, in this embodiment, weighting of the training data can suppress the decrease in classification accuracy of classes with small area proportions. Furthermore, it is possible to increase the average classification accuracy of the entire image.

[0088] Furthermore, the driving control method according to this embodiment includes an image region classification method according to this embodiment and a driving control step S50 which controls the driving of a moving body based on the drivable region classified by the image region classification method.

[0089] By using this highly accurate classification of drivable area information, it becomes possible to precisely control the movement of the mobile unit V10.

[0090] Furthermore, the mobile body V10 according to this embodiment includes an image area classification device 100 according to this embodiment and a travel control unit 50 that controls the travel of the mobile body V10 based on the travelable area classified by the image area classification device 100.

[0091] By using this highly accurate classification of drivable area information, it becomes possible to precisely control the movement of the mobile unit V10.

[0092] (Embodiment 2) The image region classification device and image region classification method according to Embodiment 2 will be described below. In this embodiment, the moving object class is included among the multiple classes into which the predicted image is classified. The image region classification device and image region classification method according to this embodiment will be described below, focusing on the differences from the image region classification device 100 and image region classification method according to Embodiment 1.

[0093] In the image region classification device and image region classification method according to this embodiment, similar to Embodiment 1, each region included in the predicted image corresponding to the orthomosaic image is classified into multiple classes, and the multiple classes include a moving object class that indicates the region where a moving object is located. In this embodiment, the multiple classes include six classes: unlabeled, road, sidewalk, median strip, vegetation, and moving object.

[0094] This allows for the classification of roads representing drivable areas and moving objects in the predictive image. Therefore, for example, in the driving control of a moving object V10 including a reference point, using the predictive image allows for the acquisition of information on both the drivable area and the area of ​​the moving object that should be avoided, thereby simplifying the driving control.

[0095] The experimental results of classifying each region of a predicted image using the image region classification device and image region classification method according to this embodiment will be explained below with reference to Figures 18 to 23. Figure 18 is a diagram showing the relationship between the presence or absence of a moving object class in the classification of each region of the predicted image according to this embodiment and the IoU of the classification result for each region of the predicted image. Figure 18 also shows the average IoU (mIoU) of all classes. Figure 19 is a diagram showing the Precision and Recall of the classification result for each region of the predicted image according to this embodiment. Precision is a value that represents the low number of false detections, and Recall is a value that represents the low number of undetected objects. Figures 20 and 21 are diagrams showing an example of a predicted image obtained by the image region classification device and image region classification method according to this embodiment, and the ground truth image corresponding to the predicted image, respectively. Figures 22 and 23 are diagrams showing another example of a predicted image obtained by the image region classification device and image region classification method according to this embodiment, and the ground truth image corresponding to the predicted image, respectively.

[0096] In each of the above experiments, as in Embodiment 1, weighting is performed in learning step S10 based on the area ratio of each of the multiple classes in the training data. The weights are set as shown in Figure 16. The weight of the moving object is set to 144.93.

[0097] As shown in Figure 18, the IoU for the mobile object class is lower than that for other classes. Therefore, the average IoU is lower when the mobile object class is present than when it is not. Also, as shown in Figure 19, the precision of the mobile object class is low and the recall is high. As described above, in this embodiment, although there are a relatively large number of false detections of mobile objects, it can be seen that the area where the mobile object is located can be generally detected. As shown in Figures 20 and 21, the white area representing the position of the mobile object shown in the ground truth image of Figure 21 is generally detected as the white area representing the position of the mobile object in the predicted image of Figure 20. Also, as shown in Figures 22 and 23, the white area representing the position of the mobile object shown in the ground truth image of Figure 23 is generally detected as the white area representing the position of the mobile object in the predicted image of Figure 22.

[0098] Figure 22 shows moving objects that are not present in the ground truth image in Figure 23. In other words, there are some false detections of moving objects in Figure 22, but this is thought to indicate that the features of moving objects were learned in learning step S10.

[0099] As described above, according to the image region classification device and image region classification method of this embodiment, it is possible to obtain a predictive image in which roads representing drivable areas and moving objects are classified.

[0100] (Other variations, etc.) Although an image region classification method and the like according to one aspect of the present invention have been described above based on embodiments, the present invention is not limited to these embodiments. Various modifications to the embodiments that a person skilled in the art can conceive of may also be included within the scope of the present invention, as long as they do not depart from the spirit of the present invention.

[0101] For example, in the above embodiment, an example of a mobile body V10, which is a vehicle, was shown, but the mobile body is not limited to a vehicle. The mobile body can be any object that can be controlled to move.

[0102] Furthermore, the following forms may also be included within the scope of one or more aspects of this disclosure.

[0103] (1) Some of the components included in the image region classification device 100 and the mobile body V10 described above may be a computer system consisting of a microprocessor, ROM, RAM, hard disk unit, etc., or a server connected by a network. A computer program is stored in the RAM or hard disk unit. The microprocessor achieves its function by operating in accordance with the computer program. Here, the computer program is composed of a combination of multiple instruction codes that indicate commands to the computer in order to achieve a predetermined function.

[0104] (2) Some of the components included in the image region classification device 100 and the mobile body V10 described above may be made up of a single system LSI (Large Scale Integration). The system LSI is a multi-functional LSI manufactured by integrating multiple components onto a single chip, and specifically, it is a computer system that includes a microprocessor, ROM, RAM, etc. A computer program is stored in the RAM. The system LSI achieves its function by operating the microprocessor in accordance with the computer program.

[0105] (3) Some of the components included in the image region classification device 100 and the mobile body V10 described above may consist of detachable IC cards or standalone modules attached to each device. The IC card or module is a computer system consisting of a microprocessor, ROM, RAM, etc. The IC card or module may include the above-mentioned multi-functional LSI. The microprocessor operates according to a computer program, thereby enabling the IC card or module to achieve its function. The IC card or module may be tamper-resistant.

[0106] (4) Furthermore, some of the components included in the image region classification device 100 and the mobile body V10 may be recorded on a recording medium that can be read by a computer, such as a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, BD (Blu-ray® Disc), semiconductor memory, etc. Alternatively, the digital signal may be recorded on one of these recording media.

[0107] Furthermore, some of the components included in the image region classification device 100 and the mobile device V10 may transmit the computer program or the digital signal via telecommunications lines, wireless or wired communication lines, networks such as the Internet, data broadcasting, etc.

[0108] (5) The present disclosure may be an image region classification method or a driving control method as described above. Alternatively, it may be a computer program for implementing the above image region classification method or driving control method using a computer, or it may be a digital signal consisting of the computer program. Furthermore, the present disclosure may be implemented as a non-temporary computer-readable recording medium such as a CD-ROM on which the computer program is recorded.

[0109] (6) The Disclosure may also provide a computer system comprising a microprocessor and memory, wherein the memory stores the computer program, and the microprocessor operates in accordance with the computer program.

[0110] (7) Alternatively, the program or the digital signal may be carried out by another independent computer system by recording it on the recording medium and transferring it, or by transferring the program or the digital signal via the network or the like.

[0111] (8) The above embodiments and the above modified examples may be combined.

[0112] (Note) Furthermore, the following technologies are disclosed based on the above description.

[0113] (Technology 1) A computer-based image region classification method comprising: a point cloud acquisition step of acquiring a 3D point cloud including distance information from a reference point to distance measurement targets located around the reference point; an image acquisition step of acquiring an orthoimage corresponding to the 3D point cloud; a feature extraction step of extracting feature quantities from the 3D point cloud and the orthoimage using a trained first machine learning model; and a classification step of classifying each region included in a predicted image corresponding to the orthoimage into a plurality of classes based on the feature quantities extracted in the feature extraction step using a trained second machine learning model.

[0114] (Technical 2) The image region classification method according to Technical 1, wherein the plurality of classes include a drivable region which is a region on which a moving body including the reference point can travel.

[0115] (Technology 3) The image region classification method according to Technology 1 or 2, wherein the object to be measured for distance includes a moving object located around the reference point, and the image region classification method includes a detection step of detecting the moving object included in the 3D point cloud based on the feature quantities extracted in the feature extraction step using a trained third machine learning model.

[0116] (Technical 4) The image region classification method according to Technical 3, wherein the multiple classes include a mobile body class that indicates the region in which the mobile body is located.

[0117] (Technical 5) An image region classification method according to any one of Technical 1 to 4, wherein the feature extraction step includes a first transformation step of converting the 3D point cloud into a first image which is a 2D image, a second transformation step of converting the orthoimage into a second image which is a 2D image, and a joining step of combining the first image and the second image, wherein in the first image, a first feature quantity which is a feature quantity corresponding to each 2D position of the 3D point cloud is associated with the 2D position, and in the second image, a second feature quantity which is a feature quantity corresponding to each 2D position of the orthoimage is associated with the 2D position, and in the joining step, the first feature quantity and the second feature quantity are combined, and convolution is performed on the combined first feature quantity and the second feature quantity.

[0118] (Technical 6) The image region classification method according to any one of Technical 1 to 5, wherein the image region classification method includes a learning step of training the first machine learning model and the second machine learning model, and in the learning step, weights are applied to the training data relating to each of the multiple classes based on the area ratio of each of the multiple classes in the training data of the predicted image.

[0119] (Technical 7) A driving control method comprising: an image region classification method described in Technical 2; and a driving control step of performing driving control of the moving body based on the drivable region classified by the image region classification method.

[0120] (Technical 8) A program for causing a computer to execute one of the image region classification methods described in Technical 1 to 6.

[0121] (Technical 9) An image region classification device comprising: a point cloud acquisition unit that acquires a 3D point cloud including distance information from a reference point to distance measurement targets located around the reference point; an image acquisition unit that acquires an orthoimage corresponding to the 3D point cloud; a feature extraction unit that extracts feature quantities from the 3D point cloud and the orthoimage using a trained first machine learning model; and a classification unit that classifies each region included in a predicted image corresponding to the orthoimage into a plurality of classes based on the feature quantities extracted by the feature extraction unit using a trained second machine learning model.

[0122] (Technical 10) The image region classification device according to Technical 9, wherein the plurality of classes include a drivable region which is a region on which a moving body including the reference point can travel.

[0123] (Technical 11) A mobile body comprising an image region classification device described in Technical 10, and a travel control unit that controls the travel of the mobile body based on the travelable region classified by the image region classification device. [Industrial applicability]

[0124] An image region classification method, etc., according to one aspect of the present invention can be applied, for example, to a moving object such as a vehicle for autonomous driving. [Explanation of Symbols]

[0125] 10 Measuring part 21 Point cloud acquisition section 22 Image acquisition unit 30 Feature extraction unit 31 First Conversion Unit 32 Second conversion section 33 Joint part 34 Two-dimensional convolution section 41 Classification Department 42 Detection unit 50 Driving control unit 60 Storage section 100 Image region classification device V1 and V10 mobile bodies

Claims

1. A computer-based method for classifying image regions, A point cloud acquisition step involves acquiring a 3D point cloud that includes distance information from a reference point to objects located around the reference point. Image acquisition step to acquire an orthoimage corresponding to the three-dimensional point cloud, A feature extraction step in which features are extracted from the 3D point cloud and the orthoimage using a pre-trained first machine learning model, The process includes a classification step in which, using a pre-trained second machine learning model, each region included in the predicted image corresponding to the orthoimage is classified into one of several classes based on the features extracted in the feature extraction step. Image region classification method.

2. The aforementioned multiple classes include a drivable area which is an area on which a moving object including the reference point can travel. The image region classification method according to claim 1.

3. The object to be measured for distance includes a moving object located around the reference point. The image region classification method includes a detection step that uses a trained third machine learning model to detect the moving object included in the three-dimensional point cloud based on the features extracted in the feature extraction step. The image region classification method according to claim 1 or 2.

4. The aforementioned plurality of classes include a mobile body class that indicates the region in which the mobile body is located. The image region classification method according to claim 3.

5. The feature extraction step described above is: A first transformation step that converts the three-dimensional point cloud into a two-dimensional image, A second transformation step involves converting the orthophoto into a second image which is a two-dimensional image, The step includes combining the first image and the second image, In the aforementioned first image, the first feature quantity, which is a feature quantity corresponding to each two-dimensional position in the three-dimensional point cloud, is associated with that two-dimensional position. In the second image, the second feature quantity, which is a feature quantity corresponding to each two-dimensional position of the orthoimage, is associated with that two-dimensional position. In the aforementioned merging step, the first feature and the second feature are merged, and convolution is performed on the merged first and second feature. The image region classification method according to claim 1 or 2.

6. The aforementioned image region classification method is: This includes a learning step for training the first machine learning model and the second machine learning model, In the learning step, weights are applied to the training data for each of the multiple classes based on the area ratio of each of the multiple classes in the training data of the predicted image. The image region classification method according to claim 1 or 2.

7. The image region classification method described in claim 2, The method includes a driving control step of controlling the movement of the moving body based on the drivable area classified by the image area classification method. A method for controlling vehicle movement.

8. To cause a computer to execute the image region classification method according to either claim 1 or 2. program.

9. A point cloud acquisition unit acquires a 3D point cloud including distance information from a reference point to objects located around the reference point, An image acquisition unit that acquires an orthophoto corresponding to the three-dimensional point cloud, A feature extraction unit that extracts feature quantities from the 3D point cloud and the orthoimage using a pre-trained first machine learning model, The system includes a classification unit that uses a pre-trained second machine learning model to classify each region included in the predicted image corresponding to the orthomosaic image into multiple classes based on the features extracted by the feature extraction unit. Image region classification device.

10. The aforementioned multiple classes include a drivable area which is an area on which a moving object including the reference point can travel. The image region classification device according to claim 9.

11. The image region classification device according to claim 10, The system includes a travel control unit that controls the movement of the mobile body based on the drivable area classified by the image area classification device. A mobile object.

Images