Learning system, method, program, and area estimation system

Abstract

To accurately estimate a passable region including a region behind an object in a captured image by efficiently generating a data set for learning to be used for machine learning of an estimation model.SOLUTION: A learning system (1) includes: a generation unit (143) that uses arrangement information (41) on an object (31) arranged at an arbitrary position in a virtual space (3) to generate a correct answer label image (1513) including information relating to whether or not a moving body can pass through a site behind the object in the perspective from an arbitrary imaging position; and a learning unit (144) that uses a color information image (1511) obtained by imaging an imaging region including the object from an imaging position, a depth image (1512) from the imaging position to the imaging region, and the correct answer label image (1513) to perform learning of an estimation model (160) for estimating a passable region through which the moving body can pass from among the imaging region.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a learning system, method, program, and region estimation system. [Background technology]

[0002] In recent years, there has been a demand for the use of AI (Artificial Intelligence) estimation models such as neural networks to estimate passable areas for vehicles and other vehicles from captured images of a given traffic environment.To improve the estimation accuracy of the estimation models, machine learning of the estimation models is being carried out using training data such as captured images of the traffic environment.

[0003] Patent Document 1 discloses a technique related to a teacher data generation device. The teacher data generation device acquires external environment information (photographed images) collected by a camera mounted on a vehicle, etc. Thereafter, if the vehicle can pass through an actual area corresponding to a road surface area in the photographed image without taking any evasive action, the teacher data generation device attaches a label indicating passable to the photographed image. On the other hand, if the vehicle takes evasive action for the road surface area, the teacher data generation device does not attach a label to the photographed image, or attaches a label indicating impassable to the photographed image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 116423 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology disclosed in Patent Document 1 determines whether a road surface area is passable or not by actually driving a vehicle on the road. Therefore, in order to determine whether an area behind an obstacle included in a captured image is passable or not, it is necessary to actually drive a vehicle on the road or capture an image of the area behind the obstacle. However, there is a limit to how many points a vehicle can actually drive or capture. Therefore, there is a problem in that it is difficult to machine-train an estimation model to accurately estimate whether a road surface area is passable or not.

[0006] The present disclosure has been made to solve such problems, and by efficiently generating a training dataset to be used for machine learning of an estimation model, it is possible to accurately estimate passable areas, including areas behind objects in a captured image. [Means for solving the problem]

[0007] A learning system according to a first aspect of the present disclosure includes a generation unit that uses placement information of an object placed at an arbitrary position in a virtual space to generate correct label information including information regarding whether a moving body can pass through a point that is behind the object from a viewpoint from an arbitrary shooting position, and a learning unit that uses a color information image of a shooting area including the object photographed from the shooting position, depth information from the shooting position to the shooting area, and the correct label information to learn an estimation model that estimates a passable area for the moving body from within the shooting area.

[0008] In this manner, by using information about the placement of objects in a virtual space, this aspect can identify whether a point is hidden behind an object even from any viewpoint in the image capture, thereby generating correct label information that accurately labels whether a moving object can pass through. Therefore, a set of images captured from the same viewpoint and correct label information can be efficiently generated as a training dataset. By using an estimation model machine-learned using such a training dataset, it is possible to accurately estimate the passable area, including the area behind the object in the captured image.

[0009] Furthermore, it is desirable that the generation unit generates the correct label information by regarding, among the object regions in the shooting region corresponding to the object, regions other than the horizontal projection plane when the object is viewed from above based on the placement information as points that are on the back side of the object from the viewpoint from the shooting position. In this way, even in regions that appear to be object regions from the viewpoint from any shooting position, the horizontal projection plane when the object is viewed from above can be identified using the placement information, so that it is possible to accurately label the passability of moving bodies for at least regions other than the horizontal projection plane.

[0010] Furthermore, the generation unit may generate the correct label information by defining, within the object region, at least the ground surface of the object identified based on the placement information as impassable, and defining, at least, the region other than the horizontal projection plane as passable. Even if the object region is an object region from a viewpoint from an arbitrary shooting position, if the placement information defines a ground surface, it is at least impassable for a moving body, while there is a high probability that the region other than the horizontal projection plane is at least passable. Thus, it is possible to generate highly accurate correct label information.

[0011] Furthermore, the generation unit may generate the correct label information by treating the area of ​​the horizontal projection plane that includes the ground surface as impassable within the object area. The area below the horizontal projection plane includes the ground surface and non-ground surface of the object, both of which are highly likely to be impassable for moving objects. This allows for the generation of more accurate correct label information.

[0012] Furthermore, the generation unit may generate the correct label information so that a different label indicating whether or not a point included in the object area is passable is assigned depending on whether or not the point is on the back side of the object from the viewpoint from the shooting position. For example, even if a point is in the object area from the viewpoint from a given shooting position, the passable status of a point that can be identified as the back side of an object from the placement information often differs from the point where the object actually exists. Therefore, it is possible to generate highly accurate correct label information.

[0013] Furthermore, it is desirable that the generation unit generates, as the correct label information, a passable area image in which a label indicating whether or not the passable area is permitted is assigned to each pixel of an image of the photographed area photographed from the photographing position. This clarifies the pixel-by-pixel correspondence between the photographed image from the same viewpoint and the correct label information, thereby improving the accuracy of machine learning of the estimation model.

[0014] Furthermore, the generation unit may identify planar layout information including the object in the virtual space from the layout information, and generate the passable area image using the planar layout information so that the passable area image is viewed from the shooting position. This makes it possible to generate correct label information that accurately approximates the coordinate system of an image captured from an arbitrary shooting position.

[0015] Alternatively, the generation unit may generate the correct label information so that a label indicating that confirmation of passability is required is attached to a point on the back side of the object, because there are cases where it is difficult to identify passability even based on the placement information, and it is sometimes desirable to actually prompt confirmation of passability.

[0016] The learning unit may perform machine learning on the estimation model using a learning dataset in which the color information image and the depth information are input data and the correct label information is correct data, thereby clarifying the format of training data for the estimation model and improving the efficiency of machine learning processing.

[0017] The virtual space simulator may further include a positioning unit that positions the object at an arbitrary position within the virtual space and positions a camera at the shooting position, and an acquisition unit that acquires the color information image in which the shooting area is shot from the shooting position by the camera and acquires a depth image in which the distance from the shooting position to the shooting area is measured as the depth information. By setting the types of input data for the estimation model to be color information images and depth images acquired from the same viewpoint, it is possible to reduce the discrepancy between the results of machine learning using data acquired in the virtual space on the simulator and the estimation results in a real environment.

[0018] A learning method according to a second aspect of the present disclosure involves a computer using placement information of an object placed at an arbitrary position in a virtual space to generate correct label information including information regarding whether a moving body can pass through a point that is behind the object from the viewpoint of an arbitrary shooting position, and then using a color information image of a shooting area including the object photographed from the shooting position, depth information from the shooting position to the shooting area, and the correct label information, learning an estimation model that estimates the passable area of ​​the moving body within the shooting area.

[0019] A learning program according to a third aspect of the present disclosure causes a computer to execute a generation process that uses placement information of an object placed at an arbitrary position in a virtual space to generate correct label information including information regarding whether a moving body can pass through a point that is behind the object from a viewpoint from an arbitrary shooting position, and a learning process that uses a color information image of a shooting area including the object photographed from the shooting position, depth information from the shooting position to the shooting area, and the correct label information to learn an estimation model that estimates a passable area for the moving body from within the shooting area.

[0020] The second and third aspects can be expected to have the same effects as the first aspect.

[0021] An area estimation system according to a fourth aspect of the present disclosure includes an estimation unit that estimates the passable area in the second area by inputting a second color information image of a second area including a second object photographed from a second photographing position in a specified traffic environment and second depth information from the second photographing position to the second area to the estimation model trained by the learning system described in the first aspect.

[0022] In this way, in the fourth aspect, by using a machine-learned estimation model similar to that in the first aspect, it is possible to accurately estimate the passable area, including the area behind objects in the captured image, from the captured image and depth information taken from the same viewpoint in the real environment. [Effects of the Invention]

[0023] According to the present disclosure, by efficiently generating a learning dataset to be used for machine learning of an estimation model, it is possible to accurately estimate passable areas, including areas behind objects in a captured image. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a block diagram showing a configuration including a learning system and a region estimation system according to a first embodiment. [Figure 2] 1 is a block diagram showing the internal configuration of a learning system according to a first embodiment. [Figure 3] 1 is a block diagram showing the internal configuration of a moving body according to a first embodiment. [Figure 4] 4 is a flowchart showing the flow of a learning process according to the first embodiment. [Figure 5] FIG. 10 is a diagram showing an example of a color information image captured in a virtual space. [Figure 6] FIG. 10 is a diagram showing an example of a depth image captured in a virtual space. [Figure 7] FIG. 10 is a diagram showing an example of a previous correct label image in a state where the back side of an object in a virtual space cannot be identified. [Figure 8] FIG. 2 is a diagram showing an example of a correct label image of a passable area in a virtual space according to the first embodiment. [Figure 9] FIG. 10 is a diagram showing an example in which different labels are assigned to grounded surfaces and non-grounded surfaces in an object region. [Figure 10] 10 is a flowchart showing the flow of an area estimation process according to the first embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of a color information image. [Figure 12] FIG. 10 is a diagram illustrating an example of a depth image. [Figure 13] FIG. 10 is a diagram showing an example of a previous region estimation image in a state where the back side of an object cannot be identified. [Figure 14] FIG. 2 is a diagram showing an example of an area estimation image of a passable area according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, specific embodiments of the present invention, including the above-mentioned aspects, will be described in detail with reference to the drawings. In each drawing, the same elements are designated by the same reference numerals, and for clarity of explanation, duplicate explanations will be omitted as necessary.

[0026] <Embodiment 1> FIG. 1 is a block diagram showing a configuration including a learning system 1 and an area estimation system 2 according to the first embodiment. The learning system 1 is an information system that generates a learning dataset 151 and the like using captured images of objects 31 and the like in a virtual space 3 and placement information of the objects 31 and the like, and performs machine learning on an area estimation model 160 using the learning dataset 151 and the like. Here, the area estimation model 160 is an AI model that estimates a passable area for a predetermined moving body (e.g., moving body 20) from within the captured area. Details of the area estimation model 160 will be described later.

[0027] The learning system 1 includes a placement unit 141, an acquisition unit 142, a generation unit 143, and a learning unit 144. The placement unit 141 places objects 31 to 3n (n is a natural number equal to or greater than 1) at arbitrary positions within the virtual space 3, and places the camera 30 at an arbitrary shooting position. The virtual space 3 can be considered a virtual three-dimensional environment on a predetermined simulator, such as a virtual traffic environment. The objects 31 and the like are virtual objects, and the camera 30 can be considered a shooting device, particularly a virtual shooting device. The camera 30 images color information in the shooting area. Furthermore, the camera 30 may image depth information from the shooting position to the shooting area as a depth image. For example, the camera 30 may have a function to capture a depth image for the same shooting area as the color information image. The depth information does not have to be a depth image. Therefore, the placement unit 141 may place a virtual depth measurement sensor or the like at the same shooting position as the camera 30. The depth measurement sensor may measure the distance from the shooting position to the shooting area as depth information or may capture an image as a depth image. For example, the depth measurement sensor may emit a light beam to measure the distance to an object. Note that the camera 30 and the depth measurement sensor are not limited to these.

[0028] Furthermore, the placement unit 141 may perform, for example, the following processing using the functions of the simulator. First, the placement unit 141 randomly selects at least any of object information, such as the type of object to be placed, the three-dimensional shape, and the texture to be applied to the surface. The placement unit 141 also randomly selects the number of objects to be placed and the position in the virtual space 3 at which each object is to be placed. The placement unit 141 also randomly selects the position (photography position) of the camera 30 to be placed. Then, the placement unit 141 generates placement information 41 including the selected object information, the number and positions of the objects, and the position of the camera 30. Then, the placement unit 141 places the objects 31 to 3n and the camera 30 in the virtual space 3 based on the placement information 41. For example, the placement unit 141 may generate the placement information 41 using domain randomization in the simulator and place each object 31, etc. and the camera 30. This reduces the gap between the simulator and the real world. Alternatively, the placement unit 141 may generate the placement information 41 by selecting the object information, the number and positions of the objects, and the positions of the camera 30 and the like based on a predetermined rule.

[0029] The acquisition unit 142 acquires a color information image 42 in which a shooting area is captured from a shooting position by the camera 30 arranged in the virtual space 3. That is, the camera 30 captures an image of a shooting area including the object 31, etc. at the shooting position and outputs the image to the acquisition unit 142 as a color information image 42. The acquisition unit 142 also acquires a depth image 43 in which the distance from the shooting position to the shooting area is measured by the camera 30 arranged in the virtual space 3 or a depth measurement sensor, as depth information. That is, the camera 30, etc. measures the distance from the shooting position to the object 31, etc., and outputs the depth image 43 to the acquisition unit 142. In other words, the acquisition unit 142 acquires depth information of each pixel in a shooting range corresponding to the color information image 42, starting from the shooting position of the camera 30. The color information image 42 and the depth image 43 are assumed to have a common shooting range (coordinate system). At this time, the acquisition unit 142 stores the acquired color information image 42 and depth image 43 in a memory or storage unit (to be described later) as a color information image 1511 and a depth image 1512. It can also be said that the acquisition unit 142 renders the color information image 42 and the depth image 43 using the function of the simulator.

[0030] The generation unit 143 uses the placement information 41 to generate correct label information including information regarding the passability of a moving object at a point behind the object 31, etc., from the viewpoint of the shooting position of the camera 30. Here, the "point behind the object 31, etc." refers to a point where the ground, etc., is hidden (invisible) by the object 31, etc., from the viewpoint of the shooting position of the camera 30. Furthermore, the "information regarding the passability of a moving object" refers to information indicating either passability, impassability, or the need to confirm passability. Note that the "information regarding the passability of a moving object" may be a binary value, either passable or impassable. Furthermore, the "correct label information" refers to information in which information regarding the passability of a moving object is assigned as a label to each point corresponding to the shooting area in the virtual space 3. Note that the generation unit 143 may use the color information image 42 or the depth image 43 acquired by the acquisition unit 142, together with the placement information 41, when generating the correct label information.

[0031] In particular, it is desirable that the generation unit 143 generates correct label information by regarding, among the object regions corresponding to the object in the shooting region, regions other than the horizontal projection plane when the object is viewed from above as points that are on the back side of the object from the viewpoint from the shooting position based on the arrangement information 41. As a result, even if a region appears to be an object region from the viewpoint from any shooting position, the horizontal projection plane when the object is viewed from above can be identified using the arrangement information 41, and therefore it is possible to accurately label at least the regions other than the horizontal projection plane as passable or unpassable for the moving body.

[0032] Furthermore, the generation unit 143 may generate correct label information by designating at least the ground surface of the object identified based on the placement information 41 as impassable, and by designating at least the area other than the horizontal projection plane as passable. Even if the object area is viewed from an arbitrary shooting position, if it is a ground surface in the placement information 41, it is at least impassable for a moving object. On the other hand, there is a high probability that the area other than the horizontal projection plane is at least passable. Therefore, it is possible to generate highly accurate correct label information. Furthermore, the generation unit 143 may generate correct label information by designating at least the area of ​​the horizontal projection plane (including the ground surface) as impassable, of the object area. The area below the horizontal projection plane includes both the ground surface and non-ground surface of the object, and both are highly likely to be impassable for a moving object. Therefore, it is possible to generate more accurate correct label information. Note that even if the non-ground surface of the object is below the horizontal projection plane, it may be passable for a moving object. In this case, for example, the generation unit 143 may generate correct label information indicating that a moving object is passable for a non-grounded surface of the object under the horizontal projection plane based on the three-dimensional shape of the object in the placement information 41. Alternatively, the generation unit 143 may generate correct label information such that a label indicating that confirmation of passability is required is assigned to a point on the back side of the object. This is because there are cases where it is difficult to determine passability even based on the placement information 41, and it may be desirable to actually prompt confirmation of passability.

[0033] Here, the generation unit 143 may generate correct label information by regarding the non-ground surface of the object as a point on the back side of the object, instead of the "area other than the horizontal projection plane." In this way, even if an area appears to be an object area from a viewpoint from any shooting position, the running area of ​​the moving body and the non-ground surface of the object can be identified by the arrangement information, and therefore, it is possible to accurately label the non-ground surface as to whether the moving body can pass through.

[0034] Furthermore, the generation unit 143 may generate correct label information by treating the ground surface of the object identified based on the placement information 41 as impassable and the non-ground surface as passable. Even if the object area is an object area from the viewpoint of an arbitrary shooting position, if the placement information 41 indicates a ground surface, the moving body cannot pass through, whereas the non-ground surface is highly likely to be passable. Therefore, it is possible to generate highly accurate correct label information.

[0035] Furthermore, the generation unit 143 may generate correct label information so that different labels regarding passability are assigned to each point included in the object area depending on whether it is on the back side of the object from the viewpoint from the shooting position. For example, even if it is an object area from the viewpoint from an arbitrary shooting position, the passability of a point that can be identified as the back side of the object from the placement information 41 often differs from the point where the object actually exists. Therefore, it is possible to generate highly accurate correct label information.

[0036] Furthermore, it is desirable that the generation unit 143 generates, as the correct label information, a passable area image in which a label indicating passability is assigned to each pixel of an image of the shooting area photographed from the shooting position. For example, the generation unit 143 generates a correct label image 1513 in which a label indicating either passability or passability is assigned to each pixel of the color information image 1511. This clarifies the pixel-by-pixel correspondence between the image photographed from the same viewpoint and the correct label information, improving the accuracy of machine learning of the area estimation model 160. Here, the passable area image can be said to be an image in which a label indicating passability is assigned to an area that is behind an object and cannot be seen when viewed from the angle of the camera 30.

[0037] Furthermore, the generation unit 143 may identify planar arrangement information including the object in the virtual space 3 from the arrangement information 41, and generate a passable area image that is a viewpoint from the shooting position using the planar arrangement information. This makes it possible to generate correct label information that accurately approximates the coordinate system of a photographed image taken from an arbitrary shooting position.

[0038] The generation unit 143 generates the generated correct answer labeled image 1513 as a training dataset 151 by associating it with the color information image 1511 and the depth image 1512. At this time, the generation unit 143 stores the training dataset 151 in a memory or storage unit, which will be described later. Similarly, the generation unit 143 generates a correct answer labeled image using other arrangement information in the arrangement unit 141. Then, the generation unit 143 similarly generates a training dataset 15m (m is a natural number equal to or greater than 1) for the other arrangement information and stores it in a memory or the like. Note that it is sufficient for the learning system 1 to generate at least one training dataset.

[0039] The learning unit 144 learns the area estimation model 160 using a color information image 1511 in which a shooting area including an object is photographed from the shooting position, a depth image 1512 which is depth information from the shooting position to the shooting area, and a correct label image 1513.

[0040] The region estimation model 160 is a model formula such as a function defined by parameters and variables, or a computer program. The region estimation model 160 is, for example, an estimation model that estimates a passable region of a moving object from within a captured region corresponding to the color information image 1511. In particular, the region estimation model 160 may receive a color information image and depth information as input data and an image indicating the estimated passable region as output data. The region estimation model 160 is typically configured as a neural network. For example, the region estimation model 160 may be a neural network that performs semantic segmentation. In particular, the region estimation model 160 is preferably a CNN (Convolutional Neural Network) type. For example, the region estimation model 160 may set each pixel value of the color information image and the depth image to each element (neuron) of the input layer, and output a label of each pixel of the image indicating the passable region to each element of the output layer. In addition, various models and networks whose inputs and outputs conform to the above can be applied to the region estimation model 160.

[0041] Specifically, the learning unit 144 performs machine learning on the estimation model using a learning dataset in which the color information image and the depth information are input data and the correct label information is correct data (teacher data). The learning unit 144 may apply a technique equivalent to statistical machine learning to the estimation model. For example, the learning unit 144 performs deep learning on the area estimation model 160 using the learning datasets 151 to 15m. That is, the learning unit 144 optimizes and updates the parameters 161 of the area estimation model 160 using the learning datasets 151 to 15m so as to improve the estimation accuracy of the passable area.

[0042] Fig. 2 is a block diagram showing the internal configuration of the learning system 1 according to the present embodiment 1. Fig. 2 shows an example in which the learning system 1 is implemented as a single information processing device, i.e., a computer. However, the learning system 1 may be implemented redundantly with multiple computers, or may be a computer system implemented with multiple computers each having different functional blocks.

[0043] The learning system 1 includes a storage unit 11, a memory 12, a communication unit 13, and a control unit 14. The storage unit 11 is an example of a non-volatile storage device such as a hard disk or a flash memory. The storage unit 11 stores a simulator program 111, a learning program 112, placement information 113, learning datasets 151 to 15m, and a region estimation model 160. The learning program 112 may include the simulator program 111. Some or all of the placement information 113, learning datasets 151 to 15m, and region estimation model 160 may be stored in a storage device different from the storage unit 11. For example, some or all of the placement information 113, learning datasets 151 to 15m, and region estimation model 160 may be stored in an external storage device connected to the learning system 1.

[0044] The simulator program 111 is a computer program in which processing equivalent to the simulator is implemented. For example, the simulator program 111 may be implemented with processing for arranging objects 31 and the camera 30 in the virtual space 3 and taking pictures using the camera 30. The learning program 112 is a computer program in which at least processing for generating correct answer label information and processing for learning the area estimation model 160 are implemented. Note that the learning program 112 may further be implemented with processing for generating placement information using the simulator program 111, placement processing based on the placement information, and processing for acquiring a color information image and a depth image. In other words, the learning program 112 may be implemented with processing equivalent to the placement unit 141, acquisition unit 142, generation unit 143, and learning unit 144 described above.

[0045] The placement information 113 is information equivalent to the above-mentioned placement information 41. That is, the placement information 113 includes object information such as the type of object to be placed in the virtual space 3, its three-dimensional shape, and the texture to be attached to the surface, the number and positions (three-dimensional coordinates, etc.) of the objects to be placed, and the position of the camera 30, etc. (photographing position, such as three-dimensional coordinates). Furthermore, when the placement process is performed multiple times, the placement information 113 may be a collection of multiple pieces of placement information corresponding to each time.

[0046] The training dataset 151 is a set of a color information image 1511, a depth image 1512, and a correct answer label image 1513, and these are associated information. As described above, the color information image 1511 and the depth image 1512 are used as input data for the region estimation model 160. The correct answer label image 1513 is used as correct answer data in machine learning of the region estimation model 160. The correct answer label image 1513 is image data in which each pixel is labeled with a label indicating whether it is passable or not. The training datasets 152 to 15m have the same configuration as the training dataset 151. The region estimation model 160 is as described above.

[0047] The memory 12 is a volatile storage device such as RAM (Random Access Memory), and is a storage area for temporarily storing information when the control unit 14 is operating. The communication unit 13 is a communication interface between the learning system 1 and an external device or a communication network. The communication unit 13 may be realized, for example, by a general-purpose or dedicated circuit realized by a semiconductor device.

[0048] The control unit 14 controls each component of the learning system 1. The control unit 14 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), or a quantum processor (quantum computer control chip). The control unit 14 loads the simulator program 111 and the learning program 112 from the storage unit 11 into the memory 12, and executes the simulator program 111 and the learning program 112. In this way, the control unit 14 realizes the functions of the arrangement unit 141, acquisition unit 142, generation unit 143, and learning unit 144 described above.

[0049] Furthermore, when some or all of the components of the learning system 1 are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally or decentralized. For example, the information processing devices, circuits, etc. may be realized as a client-server system, cloud computing system, etc., in a form in which each device is connected via a communication network.

[0050] Returning to Figure 1, the explanation continues. The area estimation system 2 is an information system that estimates a passable area from images captured by a camera mounted on a moving object 20 in a predetermined traffic environment 200 using an area estimation model 160 trained by the learning system 1, and controls the operation of the moving object 20 based on the passable area.

[0051] Traffic environment 200 is a real environment in which physical objects 201 to 20k (k is a natural number equal to or greater than 1) are arranged and in which mobile body 20, which is a robot, travels autonomously and performs predetermined operations. Traffic environment 200 is typically a factory where cargo, machines, etc. are arranged in various places, but it may also be an ordinary road, sidewalk, etc.

[0052] The area estimation system 2 performs area estimation using an area estimation model 160 in order to control the movement plan and operation of the moving body 20 in the traffic environment 200. In the following description, it is assumed that the trained area estimation model 160 is stored in the moving body 20. However, the moving body 20 may refer to the trained area estimation model 160 stored in an external storage device via wireless communication.

[0053] 3 is a block diagram showing the internal configuration of the moving body 20 according to the first embodiment. The moving body 20 includes a storage unit 21, a memory 22, a communication unit 23, a control unit 24, a camera 25, a traveling unit 26, and a grip unit 27. The storage unit 21 is an example of a non-volatile storage device such as a hard disk or a flash memory. The storage unit 21 stores a movement planning program 211, a depth image estimation model 212, and an area estimation model 213. The movement planning program 211 is a computer program in which area estimation processing, movement planning processing, and operation control processing, which will be described later, are implemented.

[0054] The depth image estimation model 212 is an AI model that estimates depth information, which is depth information, for each pixel in the same captured area as the input color information image and generates a depth image. The depth image estimation model 212 is a model formula such as a function defined by parameters and variables, or a computer program. In particular, the depth image estimation model 212 receives a color information image as input data, estimates depth information, which is depth information, for each pixel in the same captured area as the color information image, and outputs a depth image generated from the estimated depth information as output data. The depth image estimation model 212 may estimate and output a depth image from a pair of color information images captured by a stereo camera of the moving object 20. Alternatively, the depth image estimation model 212 may estimate and output a depth image from a color information image captured by a monocular camera of the moving object 20. The depth image estimation model 212 is assumed to have been machine-trained in advance using a large number of training datasets. The depth image estimation model 212 is typically configured as a neural network, but is not limited to this.

[0055] The area estimation model 213 is the area estimation model 160 that has been trained by the above-described training system 1 and stored in the moving body 20.

[0056] The memory 22 is a volatile storage device such as a RAM, and is a storage area for temporarily storing information during operation of the control unit 24. The communication unit 23 is a communication interface between the mobile object 20 and an external device or a communication network. The communication unit 23 may be realized, for example, by a general-purpose or dedicated circuit realized by a semiconductor device.

[0057] The control unit 24 controls each component of the moving object 20. The control unit 24 is, for example, a processor such as a CPU, a GPU, an FPGA, or a quantum processor. The control unit 24 loads the movement planning program 211 from the storage unit 21 into the memory 12 and executes the movement planning program 211. The control unit 14 also loads the depth image estimation model 212 and the region estimation model 213 into the memory 12 as needed and executes the depth image estimation model 212 and the region estimation model 213. As a result, the control unit 14 realizes the functions of the image acquisition unit 241, the depth acquisition unit 242, the estimation unit 243, the movement planning unit 244, and the operation control unit 245 described above. The depth image estimation model 212 and the region estimation model 213 may be stored in an external storage device connected to the moving object 20. In this case, the control unit 14 transmits and receives data to and from the depth image estimation model 212 and the region estimation model 213 via the communication unit 13, thereby acquiring a depth image and a region estimation image.

[0058] The image acquisition unit 241 controls the camera 25 during movement planning, etc., to acquire a captured image captured by the camera 25. The image acquisition unit 241 acquires at least a color information image. The depth acquisition unit 242 inputs the color information image acquired by the image acquisition unit 241 to the depth image estimation model 212 and acquires a depth image from the depth image estimation model 212. The estimation unit 243 inputs the color information image and the depth image to the area estimation model 213 and acquires an estimated result of an area passable by the moving object 20 within the captured area as an area estimation image. The movement planning unit 244 generates a movement plan for the moving object 20 using the area estimation image. Here, the "movement plan" may, for example, mean generating a route for moving from the current position of the moving object 20 through a passable area to a destination based on the area estimation image. The operation control unit 245 controls the operation of the moving object 20 based on the generated movement plan. Specifically, the operation control unit 245 instructs the traveling unit 26 to travel into the passable area. Furthermore, the operation control unit 245 instructs the gripping unit 27 to perform a gripping operation as necessary.

[0059] The camera 25 is a physical imaging device. The camera 25 may be a stereo camera, which is a set of multiple camera devices. The camera 25 periodically captures an image of a capture area within the traffic environment 200 or in response to a control signal or the like from the control unit 24, and outputs the captured image to the control unit 24. At this time, it is assumed that the capture area may include part or all of the object 201, etc.

[0060] The running unit 26 is a wheel, a leg, or the like for running on the ground, which is the running area in the traffic environment 200. The gripping unit 27 is a physical configuration for gripping the object 201, etc. The gripping unit 27 is, for example, a robot hand, but is not limited to this. The running unit 26 and the gripping unit 27 operate in response to control signals, instructions, commands, etc. from the control unit 24.

[0061] FIG. 4 is a flowchart showing the flow of the learning process according to the first embodiment. First, the placement unit 141 generates random placement information 41 of any objects 31 to 3n and cameras 30 in the virtual space 3 (S11). Specifically, the placement unit 141 randomly selects the number of objects to be placed, and randomly selects the above-mentioned object information and positions (three-dimensional coordinates) for each of the selected objects. At this time, the placement unit 141 may randomly select the type, three-dimensional shape (size), and texture information for each selected object. The placement unit 141 also randomly selects the type (function) and position (three-dimensional coordinates) of the camera 30 to be placed. Then, the placement unit 141 generates placement information 41 including the object information and positions of each of the selected objects, as well as the type and position of the camera 30, and stores the generated information in the storage unit 11 or the memory 12.

[0062] Next, the placement unit 141 places the object 31 and the like and the camera 30 in the virtual space 3 based on the generated placement information 41 (S12). For ease of explanation, it is assumed below that the shooting range of the camera 30 includes a plurality of objects including the objects 31 and 32.

[0063] Then, the acquisition unit 142 acquires the color information image 42 and the depth image 43 captured by the camera 30 in the virtual space 3 (S13).

[0064] FIG. 5 is a diagram showing an example of a color information image 42 captured within the virtual space 3. The color information image 42 is data including the XY coordinates and color information of each pixel. The color information image 42 is an image expressed in multiple colors, such as RGB, CMYK, or grayscale. For example, the color information image 42 shows that an object region 310 and an object region 320 partially overlap from the viewpoint of the shooting position of the camera 30. Here, the object region 310 is the region within the shooting region of the camera 30 that corresponds to the object 31, and the object region 320 is the region within the shooting region of the camera 30 that corresponds to the object 32. The color information image 42 also shows that a point behind the object 31 cannot be identified within the object region 310.

[0065] FIG. 6 is a diagram showing an example of a depth image 43 captured within the virtual space 3. The depth image 43 is an image captured from the same capturing position as the camera 30, in the same capturing area (captured range) as the color information image 42. The depth image 43 is an image in which the depth of each object, wall, and floor from the capturing position is indicated by shades of gray. The depth image 43 shows an example in which the closer to the capturing position the darker the color, and the farther from the capturing position the lighter the color. Note that the depth image 43 may also indicate the distance (depth length) from the capturing position to each point for each pixel as a numerical value, etc. However, the manner in which the depth image 43 and depth information are expressed is not limited to this.

[0066] Returning to FIG. 4, the description will be continued. The generation unit 143 generates a passable area image (correct label image 45) (distinguishing whether it is the back side of an object) using the placement information 41 (S14). For example, the generation unit 143 identifies placement information on a two-dimensional plane in the virtual space 3 from the placement information 41. For example, the generation unit 143 identifies a plan layout diagram from the placement information 41. At this time, the plan layout diagram may be an image binarized into object areas and object-absent areas on the plane. In other words, the plan layout diagram is assigned labels indicating impassability for object areas and labels indicating passability for object-absent areas on the plane. The generation unit 143 then generates a passable area image by converting the plan layout diagram into a coordinate system in which the captured area of ​​the color information image 42 is captured from the capturing position of the camera 30. In other words, the generation unit 143 converts the plan layout diagram obtained by viewing the virtual space 3 from above into the coordinate system of the viewpoint of the camera 30, and deforms the object shapes on the plane. The generation unit 143 may then mask the transformed (converted) figure on the color information image 42 in accordance with the coordinate system of the color information image 42 to generate a passable area image. In this case, areas on the plane where no object exists (areas of the non-ground surface other than at least the horizontal projection plane) are originally empty areas, and therefore remain object-free areas even after the conversion, and are labeled as passable. On the other hand, areas on the plane where an object exists (at least the ground surface) remain object-present areas even after the conversion, and are labeled as impassable. In other words, the generation unit 143 generates, as a passable area image, an image in which the height direction of each object included in the color information image 42 is compressed. Note that the generation unit 143 also labels areas in the color information image 42 that are recognized as impassable areas, such as wall areas, with impassable labels to generate a passable area image. This is because wall areas, etc., may not appear in the floor plan layout.

[0067] Here, first, an example of a case where a correct label image is generated by existing technology without using the generation unit 143 according to this embodiment is shown. Fig. 7 is a diagram showing an example of a previous correct label image 44 in a state where the back side of an object in the virtual space 3 cannot be identified. The previous correct label image 44 shows that the object region 310, the object region 320, and the wall region in Fig. 5 described above are not distinguished, and all regions are labeled as impassable by moving objects.

[0068] Next, FIG. 8 is a diagram showing an example of a correct label image 45 of a passable area in the virtual space 3 according to the first embodiment. The correct label image 45 distinguishes between the object area 310, the object area 320, and the wall area shown in FIG. 5 described above, and particularly shows that the point behind the object 31 is labeled as passable by a moving object. FIG. 9 is a diagram showing an example in which different labels are assigned to the ground surface and the non-ground surface of the object area 310 to clarify this point. Here, the object area 310 corresponding to the object 31 from the viewpoint from the shooting position of the camera 30 is distinguished into a ground surface 311 and a non-ground surface 312. The ground surface 311 is the area where the object 31 contacts the ground. The non-ground surface 312 is the area where the object 31 does not contact the ground. However, the non-ground surface 312 excludes the area where the ground surface 321 of the object area 320 overlaps with the object area 310. Here, non-ground surface 312 is essentially an area that cannot be distinguished from ground surface 311 from the viewpoint from the shooting position of camera 30. However, generation unit 143 can identify non-ground surface 312 from object area 310 by using planar arrangement information based on arrangement information 41. Then, correct labeled image 45 can be said to be an image in which passable areas and impassable areas of a moving object are classified, that is, labeled, even from the viewpoint from the shooting position of camera 30.

[0069] Returning to FIG. 4, the explanation will be continued. The generation unit 143 stores the color information image 42 (1511) and depth image 43 (1512) acquired in step S13 and the passable area image (correct label image 45 (1513)) generated in step S14 in the storage unit 11 as a training dataset 151 (S15). Then, the generation unit 143 determines whether the number of sets in the training dataset is a specified number (S16). If the number of sets does not reach the specified number in step S16, the processing returns to step S11. Then, the learning system 1 executes the processing of steps S11 to S16 again.

[0070] On the other hand, if the number of sets meets the specified number in step S16, the learning unit 144 performs machine learning on the region estimation model 160 using the stored training datasets 151 to 15m (S17). For example, the learning unit 144 sets the color information image 1511 and the depth image 1512 included in the training dataset 151 to the input layer of the region estimation model 160, performs calculations between each layer using the parameters 161, and acquires the calculation results for the output layer. Then, the learning unit 144 calculates the difference between the label of each pixel included in the calculation results and the label of the corresponding pixel in the correct label image 1513, and determines whether the difference is equal to or greater than a predetermined range. If the difference is equal to or greater than the predetermined range, the learning unit 144 corrects the parameters 161 and performs the same calculation and determination as above again. If the difference is less than the predetermined range, the learning unit 144 repeats the same process for each of the untrained training datasets 152 to 15m. Then, the learning unit 144 determines whether the convergence condition for machine learning is satisfied. For example, if the difference between the calculation result and the correct labeled image falls within a predetermined range or exceeds a predetermined number of repetitions for all of the learning data sets 152 to 15m, the learning unit 144 determines that the convergence condition is satisfied. Note that the above-described process is an example of machine learning. Therefore, the learning unit 144 may perform machine learning by applying various algorithms, such as backpropagation. Furthermore, the generation unit 143 may use, for example, binary cross-entropy loss in the loss calculation. However, the loss calculation method is not limited to this. If the machine learning satisfies the convergence condition, the learning unit 144 updates the parameter 161 in the region estimation model 160 to the value at the time of convergence. Then, the learning system 1 ends the learning process.

[0071] 10 is a flowchart showing the flow of the area estimation process according to the present embodiment 1. Here, it is assumed that the area estimation model 160 learned by the learning process of FIG. 4 described above is registered as the area estimation model 213 in the moving body 20. Then, the moving body 20 starts the area estimation process in order to perform a predetermined operation in the traffic environment 200.

[0072] First, the image acquisition unit 241 acquires a color information image 52 captured by the camera 25 of the moving object 20 in the traffic environment 200 (S21). For ease of explanation, it is assumed below that the capturing range of the camera 25 includes a plurality of objects including objects 201 to 20k. Fig. 11 is a diagram showing an example of the color information image 52 acquired in step S21.

[0073] Next, the depth acquisition unit 242 inputs the color information image 52 to the depth image estimation model 212 to acquire a depth image 53 (S22). At this time, the depth image estimation model 212 estimates depth information of each pixel from the input color information image 52, and generates and outputs a depth image 53 using the estimated depth information. In this way, the depth acquisition unit 242 acquires the depth image 53 from the depth image estimation model 212. FIG. 12 is a diagram showing an example of the depth image 53.

[0074] Next, the estimation unit 243 inputs the color information image 52 and the depth image 53 to the area estimation model 213 to obtain an area estimation image 55 of the passable area (S23). At this time, the estimation unit 243 sets the color information image 52 and the depth image 53 to the input layer of the area estimation model 213, performs calculations between each layer using learned parameters, and obtains the calculation results of the output layer. Here, the calculation results are referred to as the area estimation image 55. The area estimation image 55 can also be said to be a mask image of the passable area.

[0075] First, an example will be shown in which a region estimation image is generated using existing technology without using the trained region estimation model 213 according to this embodiment. Fig. 13 is a diagram showing an example of a previous region estimation image 54 in a state in which the back side of an object cannot be identified. The previous region estimation image 54 is an example in which a passable region 541 is superimposed on a depth image 53. In the previous region estimation image 54, the regions behind the chair and the box are not identified, and it is shown that the entire object region, including the regions behind the chair and the box, is considered impassable.

[0076] Next, FIG. 14 is a diagram showing an example of an area estimation image 55 of a passable area according to the first embodiment. The area estimation image 55 is an example in which a passable area 551 is superimposed on the depth image 53. In the area estimation image 55, the areas behind the chair and the box are identified, indicating that the areas behind the chair and the box are passable. Note that in the area estimation image 55 of FIG. 14, the passable area 551 and other areas (impassable areas) may be displayed in a reversed manner. For example, similar to the above-described correct label image 45, the passable area 551 may be displayed in white, and the impassable areas of the area estimation image 55 may be displayed in black or hatched. In other words, the representation of the area estimation image 55 is not limited to these.

[0077] Returning to FIG. 10 , the description continues. The movement planning unit 244 generates a movement plan using the region estimation image 55 (S24). Specifically, the movement planning unit 244 generates route information for the moving object 20 to reach the destination from the current position via the passable region 551 of the region estimation image 55. Note that the destination may be inside or outside the region estimation image 55. Alternatively, the movement planning unit 244 can be realized in various forms based on the region estimation image 55. For example, the movement planning unit 244 may overlay the depth image 53 and the passable region 551, convert unmasked pixels in the passable region 551 into three-dimensional positions (coordinate system) of the shooting position of the camera 25 based on the depth values, and generate a point set of obstacles in the path. Note that the point set of obstacles may be called an obstacle point cloud. The movement planning unit 244 may then generate route information to the destination so as to avoid the obstacle point cloud. Alternatively, the movement planning unit 244 may learn an AI model that directly calculates the speed of the moving object 20 from the area estimation image 55 and the direction of the destination. Then, the movement planning unit 244 may perform movement planning using the learned model.

[0078] Thereafter, the movement control unit 245 controls the movement of the moving object 20 based on the movement plan generated in step S24 (S25). For example, the movement control unit 245 instructs the traveling unit 26 to travel to the destination according to the route information included in the movement plan. Furthermore, the movement control unit 245 may instruct the gripping unit 27 to perform a predetermined movement after reaching the destination.

[0079] In this way, the learning system 1 of this embodiment uses information about the placement of objects in virtual space to accurately assign labels indicating whether the area behind an object in a captured image is passable. Learning datasets containing such correct label information can be efficiently generated in large quantities. Using such learning datasets improves the learning efficiency of the estimation model, thereby improving estimation accuracy. Therefore, the trained estimation model can be used to accurately estimate passable areas, including areas behind objects in a captured image.

[0080] Here, the training dataset of this embodiment uses images captured in a virtual space, which is an artificial environment, as input data. In other words, the images are generated by a simulator and therefore have little noise. On the other hand, since the traffic environment 200 is a real environment, images captured in the real environment are affected by brightness, such as lighting and sunlight, unlike the virtual space 3 described above. Therefore, images captured in the real environment may contain noise. Therefore, simply using color information images to input training data for the estimation model may not sufficiently improve the accuracy of estimating the passable area for images captured in the real environment. This type of problem is sometimes called the Sim2Real problem.

[0081] Therefore, in this embodiment, in addition to the color information image, depth information such as a depth image of the shooting range corresponding to the color information image is used as input data for learning the estimation model. Then, in the region estimation process, a highly accurate depth image is obtained from the color information image using the depth image estimation model 212. This reduces the influence of brightness, such as lighting, in the real environment, thereby preventing the occurrence of the Sim2Real problem.

[0082] The following methods can be used to acquire highly accurate depth images in a real environment. First, the input layer of the depth image estimation model corresponds to each pixel value of one color information image. In this case, the depth acquisition unit 242 inputs an image captured by a monocular camera of the moving body 20 to the depth image estimation model to acquire a depth image. Alternatively, the input layer of the depth image estimation model may correspond to each pixel value of multiple color information images. In this case, the depth acquisition unit 242 inputs an image captured by a stereo camera of the moving body 20 to the depth image estimation model to acquire a depth image. Alternatively, the moving body 20 may be equipped with an RGBD sensor. In this case, the image acquisition unit 241 and the depth acquisition unit 242 acquire a color information image and a depth image using the RGBD sensor. Alternatively, the moving body 20 may further be equipped with 3DLiDAR (Light Detection And Ranging). In this case, the image acquisition unit 241 and the depth acquisition unit 242 acquire a color information image and a depth image from a combination of a monocular camera and a 3DLiDAR.

[0083] Furthermore, the region estimation process according to this embodiment may be applied not only to movement planning but also to manipulation planning and task planning for robots and the like. For example, the mobile object 20 may include an operation planning unit. The operation planning unit may estimate a workspace where manipulation by the gripper 27 is possible based on the region estimation image estimated by the estimation unit 243, and generate an operation plan for controlling the movement of the gripper 27 by placing it behind an object. Alternatively, the operation planning unit may use the region estimation image to add the region behind an object to candidate locations when selecting a location for an object when changing its placement. This allows for more flexible manipulation plans to be generated. The mobile object 20 may also include a task planning unit. The task planning unit may use the region estimation image to plan various tasks for the mobile object 20, explicitly considering the region behind the object.

[0084] Note that the "information regarding the passability of the moving object" according to this embodiment, i.e., the label, may use three values: passable, impassable, or the need to confirm whether the object is passable. Specifically, the generation unit 143 may assign a label indicating whether the area behind the object is passable or whether confirmation of passability is required, to the area behind the object. In this case, the estimation unit 243 may acquire, from the area estimation model 213, an area estimation image including a passable area and an area where confirmation of passability is required. The movement planning unit 244 may then generate a movement plan to move the moving object 20 toward the area where confirmation of passability is required in the area estimation image, capture an image of the area behind the object with a camera, and confirm the actual passability. This is because if the area behind the object is large, there is high uncertainty regarding the actual passability.

[0085] The moving body 20 is not limited to the autonomously moving robot as described above, but may also be applied to an automatically driven vehicle, a vehicle driven by a human, or the like.

[0086] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0087] The present disclosure is not limited to the above-described embodiments, and may be modified as appropriate without departing from the spirit and scope of the present disclosure. In addition, the present disclosure may be implemented by appropriately combining the respective embodiments. [Explanation of symbols]

[0088] 1. Learning System 11 Storage section 111 Simulator Program 112 Study Programs 113 Placement information 151 training datasets 1511 Color Information Image 1512 depth images 1513 correct labeled images 15m training dataset 160 Area Estimation Model 161 parameters 12 Memory 13 Communications Department 14 Control Unit 141 Placement section 142 Acquisition Department 143 Generation part 144 Learning Department 2. Area estimation system 200 Traffic environment 201 Object 20k objects 20 Mobile 21 Memory section 211 Mobility Planning Program 212 Depth Image Estimation Model 213 Area Estimation Model 22 Memory 23 Communications Department 24 Control Unit 241 Image Acquisition Unit 242 Depth acquisition section 243 Estimation Department 244 Mobility Planning Department 245 Motion control section 25 Camera 26 Running part 27 Gripping part 3 Virtual Space 30 Camera 31 Object 310 Object area 311 Ground plane 312 Non-contact surface 32 Object 320 Object area 321 Ground plane 3n object 41 Placement information 42 Color Information Image 43 Depth image 44 Old correct label image 45 Correct label images 52 Color Information Image 53 Depth image 54 Old area estimation image 541 Passable area 55 Region Estimation Image 551 Passable area

Claims

1. a generation unit that uses placement information of an object placed at an arbitrary position in the virtual space to generate correct label information including information regarding whether a moving object can pass through a point that is on the back side of the object from a viewpoint from an arbitrary shooting position; a learning unit that learns an estimation model that estimates a passable area of ​​the moving object from within the photographed area using a color information image of a photographed area including the object from the photographed position, depth information from the photographed position to the photographed area, and the correct label information; Equipped with The generation unit generates the correct label information by regarding, among the object regions corresponding to the object in the photographing region, a region other than a horizontal projection plane when the object is viewed from above based on the arrangement information, as a point on the back side of the object from a viewpoint from the photographing position. Learning system.

2. The generation unit generates the correct label information by defining at least a ground surface of the object identified based on the placement information as impassable and defining at least a region other than the horizontal projection plane as passable. The learning system of claim 1 .

3. The generation unit generates the correct label information by designating an area of ​​the horizontal projection plane that includes the ground surface as impassable within the object area. The learning system according to claim 2 .

4. The generation unit generates the correct label information so that a different label regarding passability is assigned to each point included in the object area depending on whether the point is on the back side of the object from a viewpoint from the shooting position. The learning system of claim 1 .

5. The generation unit generates, as the correct label information, a passable area image in which a label regarding passability is assigned to each pixel of an image of the shooting area photographed from the shooting position. The learning system of claim 1 .

6. The generation unit specifies planar arrangement information including the object in the virtual space from the arrangement information, and generates the passable area image as a viewpoint from the shooting position using the planar arrangement information. The learning system according to claim 5 .

7. The generation unit generates the correct label information so that a label indicating that it is necessary to confirm whether the object is passable or not is assigned to a point on the back side of the object. The learning system of claim 1 .

8. The learning unit performs machine learning on the estimation model using a learning dataset in which the color information image and the depth information are input data and the correct label information is correct data. The learning system of claim 1 .

9. a placement unit that places the object at an arbitrary position in the virtual space and places a photographing device at the photographing position; an acquisition unit that acquires the color information image in which the photographing area is photographed from the photographing position by the photographing device, and acquires a depth image in which a distance from the photographing position to the photographing area is measured as the depth information; Further equipped The learning system of claim 1 .

10. The computer Using the location information of an object located at an arbitrary position in the virtual space, correct answer label information is generated, which includes information regarding whether or not a moving object can pass through a point that is on the back side of the object from a viewpoint from an arbitrary shooting position; learning an estimation model that estimates a passable area of ​​the moving body from within the photographed area using a color information image of a photographed area including the object from the photographed position, depth information from the photographed position to the photographed area, and the correct label information; In generating the correct label information, among the object areas corresponding to the object in the photographing area, an area other than a horizontal projection plane when the object is viewed from above based on the placement information is regarded as a point on the back side of the object from a viewpoint from the photographing position, and the correct label information is generated. How to learn.

11. a generation process for generating correct label information including information on whether a moving object can pass through a point on the back side of the object from a viewpoint of an arbitrary photographing position, using the position information of the object placed at an arbitrary position in the virtual space; a learning process for learning an estimation model that estimates a passable area of ​​the moving body from within the photographed area, using a color information image of a photographed area including the object from the photographed position, depth information from the photographed position to the photographed area, and the correct label information; on the computer, In the generation process, the computer generates the correct label information by regarding, among the object regions corresponding to the object in the photographed region, regions other than a horizontal projection plane when the object is viewed from above based on the arrangement information, as points that are on the back side of the object from a viewpoint from the photographed position. Learning program.

12. An area estimation system comprising an estimation unit that estimates the passable area in the second area by inputting a second color information image of a second area including a second object photographed from a second photographing position in a specified traffic environment and second depth information from the second photographing position to the second area to the estimation model trained by the learning system described in claim 1.

Images