Learning system, method, program, and region estimation system

Abstract

To accurately estimate a passable region including a region behind an object in a captured image by efficiently generating a data set for learning to be used for machine learning of an estimation model.SOLUTION: A learning system (1) includes: a generation unit (143) that uses arrangement information (41) on an object (31) arranged at an arbitrary position in a virtual space (3) to generate a correct answer label image (1513) including information relating to whether or not a moving body can pass through a site behind the object in the perspective from an arbitrary imaging position; and a learning unit (144) that uses a color information image (1511) obtained by imaging an imaging region including the object from an imaging position, a depth image (1512) from the imaging position to the imaging region, and the correct answer label image (1513) to perform learning of an estimation model (160) for estimating a passable region through which the moving body can pass from among the imaging region.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a learning system, a learning method, a learning program, and a region estimation system. [Background technology]

[0002] In recent years, there has been a demand for the use of AI (Artificial Intelligence) estimation models such as neural networks to estimate passable areas for vehicles and the like from images of a given traffic environment. In order to improve the estimation accuracy of the estimation models, machine learning of the estimation models is being carried out using teacher data such as images captured in the traffic environment.

[0003] Patent Document 1 discloses a technique related to a teacher data generating device. The teacher data generating device acquires external environment information (captured images) collected by a camera mounted on a vehicle or the like. Thereafter, if the vehicle can pass through an actual area corresponding to a road surface area in the captured image without taking an evasive action, the teacher data generating device attaches a label indicating passability to the captured image. On the other hand, if the vehicle takes an evasive action for the road surface area, the teacher data generating device does not attach a label to the captured image, or attaches a label indicating impassability to the captured image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 116423 Summary of the Invention [Problem to be solved by the invention]

[0005] Here, the technology according to Patent Document 1 determines whether a road surface area is passable or not by actually driving a vehicle. Therefore, in order to determine whether an area behind an obstacle included in a captured image is passable or not, it is necessary for the vehicle to actually drive there or to capture the area behind the obstacle. However, there is a limit to actually driving a vehicle or capturing images of all points. Therefore, there is a problem in that it is difficult to machine-train an estimation model to accurately estimate whether a road surface area is passable or not.

[0006] The present disclosure has been made to solve such problems, and accurately estimates passable areas, including areas behind objects in a captured image, by efficiently generating a learning dataset to be used for machine learning of an estimation model. [Means for solving the problem]

[0007] A learning system according to a first aspect of the present disclosure includes a generation unit that uses placement information of an object placed at an arbitrary position in a virtual space to generate correct label information including information regarding the passability of a moving body at a point that is behind the object from a viewpoint from an arbitrary shooting position, and a learning unit that uses a color information image of a shooting area including the object photographed from the shooting position, depth information from the shooting position to the shooting area, and the correct label information to learn an estimation model that estimates a passable area for the moving body from within the shooting area.

[0008] In this manner, by using the arrangement information of the object in the virtual space, the present embodiment can identify whether or not a point is hidden behind an object even from a viewpoint from any shooting position, and can generate correct answer label information that accurately labels whether or not a moving object can pass through. Therefore, a set of images taken from the same viewpoint and correct answer label information can be efficiently generated as a learning dataset. By using an estimation model machine-learned using such a learning dataset, it is possible to accurately estimate a passable area including the area behind the object in the captured image.

[0009] In addition, it is preferable that the generating unit generates the correct label information by regarding, among the object regions corresponding to the object in the shooting region, regions other than the horizontal projection plane when the object is viewed from above based on the arrangement information as points that are on the back side of the object from the viewpoint from the shooting position. In this way, even if a region appears to be an object region from the viewpoint from any shooting position, the horizontal projection plane when the object is viewed from above can be identified based on the arrangement information, so that it is possible to accurately label the passability of the moving body for at least the regions other than the horizontal projection plane.

[0010] Furthermore, the generating unit may generate the correct label information by making at least the ground surface of the object identified based on the placement information impassable, and making at least the area other than the horizontal projection surface passable. Even if the object area is an object area from a viewpoint from an arbitrary shooting position, if the ground surface is identified in the placement information, it is at least impassable for a moving body, whereas there is a high probability that the area other than the horizontal projection surface is at least passable. Thus, it is possible to generate highly accurate correct label information.

[0011] Furthermore, the generating unit may generate the correct label information by regarding the area of ​​the horizontal projection plane including the ground surface as impassable. The area below the horizontal projection plane includes the ground surface and non-ground surface of the object, both of which are highly likely to be impassable for moving objects. Therefore, it is possible to generate correct label information with higher accuracy.

[0012] The generating unit may generate the correct answer label information so that a different label regarding the passability is assigned to each point included in the object area depending on whether the point is the back side of the object from the viewpoint from the shooting position. For example, even if the point is an object area from the viewpoint from an arbitrary shooting position, the passability of a point that can be identified as the back side of the object from the placement information is often different from the point where the object actually exists. Therefore, it is possible to generate highly accurate correct answer label information.

[0013] In addition, it is preferable that the generating unit generates, as the correct label information, a passable area image in which a label regarding passability is assigned to each pixel of an image of the shooting area photographed from the shooting position. This makes it possible to clarify the pixel-by-pixel correspondence between the image photographed from the same viewpoint and the correct label information, thereby improving the accuracy of machine learning of the estimation model.

[0014] Furthermore, the generating unit may specify planar arrangement information including the object in the virtual space from the arrangement information, and generate the passable area image from the planar arrangement information as a viewpoint from the shooting position. This makes it possible to generate correct label information that is accurately approximating the coordinate system of the photographed image photographed from an arbitrary photographing position.

[0015] Alternatively, the generating unit may generate the correct label information so that a label indicating that confirmation of passability is required is added to a point on the back side of the object. This is because there are cases where it is difficult to identify passability even based on the placement information, and it is desirable to actually prompt a user to confirm passability.

[0016] The learning unit may perform machine learning on the estimation model using a learning data set in which the color information image and the depth information are input data and the correct label information is correct data. This clarifies the format of teacher data for the estimation model, and can improve the efficiency of machine learning processing.

[0017] The virtual space may further include a placement unit that places the object at an arbitrary position in the virtual space and places a photographing device at the photographing position, and an acquisition unit that acquires the color information image in which the photographing area is photographed from the photographing position by the photographing device, and acquires a depth image in which the distance from the photographing position to the photographing area is measured as the depth information. By setting the types of input data for the estimation model to a color information image and a depth image acquired from the same viewpoint, it is possible to reduce the discrepancy between the machine learning result based on the data acquired in the virtual space on the simulator and the estimation result in a real environment.

[0018] A learning method according to a second aspect of the present disclosure includes a computer using placement information of an object placed at an arbitrary position in a virtual space to generate correct label information including information regarding the passability of a moving body at a point that is behind the object from a viewpoint from an arbitrary shooting position, and learning an estimation model that estimates a passable area for the moving body within the shooting area using a color information image of a shooting area including the object photographed from the shooting position, depth information from the shooting position to the shooting area, and the correct label information.

[0019] A learning program according to a third aspect of the present disclosure causes a computer to execute a generation process that uses placement information of an object placed at an arbitrary position in a virtual space to generate correct label information including information regarding the passability of a moving body at a point that is behind the object from a viewpoint from an arbitrary shooting position, and a learning process that uses a color information image of a shooting area including the object photographed from the shooting position, depth information from the shooting position to the shooting area, and the correct label information to learn an estimation model that estimates a passable area for the moving body from within the shooting area.

[0020] In the second and third aspects as well, the same effects as in the first aspect can be expected.

[0021] An area estimation system according to a fourth aspect of the present disclosure includes an estimation unit that estimates the passable area in the second area by inputting a second color information image of a second area including a second object captured from a second shooting position in a specified traffic environment and second depth information from the second shooting position to the second area to the estimation model trained by the learning system described in the first aspect.

[0022] In this manner, in the fourth aspect, by using a machine-learned estimation model as in the first aspect, it is possible to accurately estimate the passable area, including the area behind objects in the captured image, from the captured image and depth information from the same viewpoint in the real environment. Effect of the Invention

[0023] According to the present disclosure, by efficiently generating a learning dataset to be used for machine learning of an estimation model, it is possible to accurately estimate a passable area, including the area behind an object in a captured image. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is a block diagram showing a configuration including a learning system and a region estimation system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing an internal configuration of a learning system according to a first embodiment. [Diagram 3] 1 is a block diagram showing an internal configuration of a moving body according to a first embodiment of the present invention. [Figure 4] 4 is a flowchart showing the flow of a learning process according to the first embodiment. [Diagram 5] FIG. 13 is a diagram showing an example of a color information image captured in a virtual space. [Figure 6] FIG. 1 is a diagram showing an example of a depth image captured in a virtual space. [Figure 7] FIG. 13 is a diagram showing an example of a previous correct label image in a state in which the back side of an object in a virtual space cannot be identified. [Figure 8] FIG. 2 is a diagram showing an example of a correct label image of a passable area in a virtual space according to the first embodiment. [Figure 9] FIG. 13 is a diagram showing an example in which different labels are assigned to grounded surfaces and non-grounded surfaces in an object region. [Figure 10] 5 is a flowchart showing the flow of a region estimation process according to the first embodiment. [Figure 11] FIG. 13 is a diagram showing an example of a color information image. [Figure 12] FIG. 2 is a diagram illustrating an example of a depth image. [Figure 13] FIG. 13 is a diagram showing an example of a previous region estimation image in a state where the back side of an object cannot be identified. [Figure 14] FIG. 2 is a diagram showing an example of an area estimation image of a passable area according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] In the following, specific embodiments of the present invention including the above-mentioned aspects will be described in detail with reference to the drawings. In each drawing, the same elements are given the same reference numerals, and for clarity of explanation, duplicate explanations will be omitted as necessary.

[0026] <Embodiment 1> 1 is a block diagram showing a configuration including a learning system 1 and an area estimation system 2 according to the first embodiment. The learning system 1 is an information system for generating a learning data set 151 and the like using captured images of objects 31 and the like in a virtual space 3 and arrangement information of the objects 31 and the like, and for machine learning a area estimation model 160 using the learning data set 151 and the like. Here, the area estimation model 160 is an AI model for estimating a passable area of ​​a predetermined moving body (e.g., moving body 20) from within a captured area. Details of the area estimation model 160 will be described later.

[0027] The learning system 1 includes a placement unit 141, an acquisition unit 142, a generation unit 143, and a learning unit 144. The placement unit 141 places objects 31 to 3n (n is a natural number equal to or greater than 1) at arbitrary positions in the virtual space 3, and places the camera 30 at an arbitrary shooting position. The virtual space 3 can be said to be a virtual three-dimensional environment on a predetermined simulator, for example, a virtual traffic environment. The object 31 and the like are virtual objects, and the camera 30 can be said to be a shooting device, particularly a virtual shooting device. The camera 30 images color information in a shooting area. Furthermore, the camera 30 may image depth information from the shooting position to the shooting area as a depth image. For example, the camera 30 may have a function of shooting a depth image for the same shooting area as the color information image. The depth information does not have to be a depth image. Therefore, the placement unit 141 may place a virtual depth measurement sensor or the like at the same shooting position as the camera 30. The depth measurement sensor may measure the distance from the shooting position to the shooting area as depth information, or may capture the area as a depth image. For example, the depth measurement sensor may emit a light beam to measure the distance to an object. The camera 30 and the depth measurement sensor are not limited to these.

[0028] The arrangement unit 141 may perform the following process, for example, using the functions of the simulator. First, the arrangement unit 141 randomly selects at least one of object information, such as the type of object to be arranged, the three-dimensional shape, and the texture to be attached to the surface. The arrangement unit 141 also randomly selects the number of objects to be arranged and the position in the virtual space 3 where each object is to be arranged. The arrangement unit 141 also randomly selects the position (photographing position) of the camera 30 to be arranged. Then, the arrangement unit 141 generates arrangement information 41 including the selected object information, the number and positions of the objects, and the position of the camera 30. Then, the arrangement unit 141 arranges the objects 31 to 3n and the camera 30 in the virtual space 3 based on the arrangement information 41. For example, the arrangement unit 141 may generate the arrangement information 41 by domain randomization in the simulator, and arrange each object 31, etc., and the camera 30. This can reduce the gap between the simulator and the real world. Alternatively, the placement unit 141 may generate the placement information 41 by selecting the object information, the number and positions of the objects, and the position of the camera 30, etc. based on a predetermined rule.

[0029] The acquisition unit 142 acquires a color information image 42 in which a photographing area is photographed from a photographing position by the camera 30 arranged in the virtual space 3. That is, the camera 30 photographs a photographing area including the object 31 at the photographing position, and outputs the color information image 42 to the acquisition unit 142. The acquisition unit 142 also acquires a depth image 43 in which a distance from the photographing position to the photographing area is measured by the camera 30 arranged in the virtual space 3 or a depth measurement sensor, as depth information. That is, the camera 30 measures a distance from the photographing position to the object 31, and outputs the depth image 43 to the acquisition unit 142. In other words, the acquisition unit 142 acquires depth information of each pixel in a photographing range corresponding to the color information image 42, starting from the photographing position of the camera 30. The color information image 42 and the depth image 43 are assumed to have a common photographing range (coordinate system). At this time, the acquisition unit 142 stores the acquired color information image 42 and depth image 43 in a memory or storage unit described later as a color information image 1511 and a depth image 1512. It can also be said that the acquisition unit 142 renders the color information image 42 and the depth image 43 using the function of the simulator.

[0030] The generating unit 143 uses the arrangement information 41 to generate correct label information including information on whether or not a moving object can pass through a point on the back side of the object 31, etc., from the viewpoint from the shooting position of the camera 30. Here, the "point on the back side of the object 31, etc." refers to a point where the ground, etc., is hidden (not visible) by the object 31, etc., from the viewpoint from the shooting position of the camera 30. In addition, the "information on whether or not a moving object can pass through" refers to information indicating either passable, impassable, or the need to confirm passable or not. In addition, the "information on whether or not a moving object can pass through" may be a binary value of passable or impassable. In addition, the "correct label information" refers to information in which information on whether or not a moving object can pass through is assigned as a label to each point corresponding to the shooting area in the virtual space 3. In addition, the generating unit 143 may use the color information image 42 or the depth image 43 acquired by the acquiring unit 142 together with the arrangement information 41 when generating the correct label information.

[0031] In particular, it is desirable that the generation unit 143 generates correct label information by regarding, among the object regions corresponding to the object in the shooting region, regions other than the horizontal projection plane when the object is viewed from above as points that are on the back side of the object from the viewpoint from the shooting position based on the arrangement information 41. As a result, even if a region appears to be an object region from the viewpoint from any shooting position, the horizontal projection plane when the object is viewed from above can be identified based on the arrangement information 41, so that it is possible to accurately label the passability of the moving body for at least the regions other than the horizontal projection plane.

[0032] Furthermore, the generating unit 143 may generate correct label information by setting at least the ground surface of the object specified based on the arrangement information 41 as impassable, and at least the area other than the horizontal projection surface as passable. Even if the object area is an object area from the viewpoint of an arbitrary shooting position, if it is a ground surface in the arrangement information 41, it is at least impassable for a moving body, while there is a high probability that the area other than the horizontal projection surface is at least passable. Therefore, it is possible to generate correct label information with high accuracy. Furthermore, the generating unit 143 may generate correct label information by setting at least the area of ​​the horizontal projection surface (including the ground surface) as impassable, among the object area. The area below the horizontal projection surface includes the ground surface and non-ground surface of the object, and both are highly likely to be impassable for a moving body. Therefore, it is possible to generate correct label information with higher accuracy. It is possible that even the non-ground surface of the object below the horizontal projection surface is passable for a moving body. In this case, for example, the generation unit 143 may generate correct label information indicating that the moving body can pass through the non-ground surface of the object under the horizontal projection plane from the three-dimensional shape of the object in the placement information 41. Alternatively, the generation unit 143 may generate correct label information such that a label indicating that confirmation of passability is required is added to a point on the back side of the object. This is because there are cases where it is difficult to identify passability even from the placement information 41, and it may be desirable to actually prompt the user to confirm passability.

[0033] Here, the generating unit 143 may generate correct label information by regarding the non-ground surface of the object as the back side of the object, instead of the "area other than the horizontal projection surface". In this way, even if an area appears to be an object area from a viewpoint from any shooting position, the running area of ​​the moving body and the non-ground surface of the object can be identified by the arrangement information, so that the non-ground surface can be accurately labeled as to whether the moving body can pass through it.

[0034] Furthermore, the generating unit 143 may generate correct label information by treating the ground surface of the object specified based on the arrangement information 41 as impassable and the non-ground surface as passable. Even if the object area is an object area from the viewpoint of an arbitrary shooting position, if the ground surface is in the arrangement information 41, the moving body cannot pass through it, whereas the non-ground surface is highly likely to be passable. Therefore, it is possible to generate correct label information with high accuracy.

[0035] Furthermore, the generating unit 143 may generate correct answer label information so that different labels related to passability are assigned to each point included in the object area depending on whether it is the back side of the object from the viewpoint from the shooting position. For example, even if it is an object area from the viewpoint from an arbitrary shooting position, the passability of a point that can be identified as the back side of the object from the arrangement information 41 is often different from that of a point where the object actually exists. Therefore, it is possible to generate correct answer label information with high accuracy.

[0036] In addition, it is preferable that the generating unit 143 generates, as the correct label information, a passable area image in which a label indicating passability is assigned to each pixel of an image in which the photographed area is photographed from the photographing position. For example, the generating unit 143 generates a correct label image 1513 in which a label indicating passability or non-passability is assigned to each pixel of the color information image 1511. This clarifies the pixel-by-pixel correspondence between the photographed image from the same viewpoint and the correct label information, improving the accuracy of machine learning of the area estimation model 160. Here, the passable area image can be said to be an image in which a label regarding passability is assigned to an area that is not visible behind an object when viewed from the angle of the camera 30.

[0037] Furthermore, the generating unit 143 may specify planar arrangement information including an object in the virtual space 3 from the arrangement information 41, and generate a passable area image from the viewpoint from the shooting position using the planar arrangement information. This makes it possible to generate correct label information that is accurately approximating the coordinate system of a photographed image photographed from an arbitrary shooting position.

[0038] The generating unit 143 generates the generated correct answer label image 1513 as a learning dataset 151 by associating it with the color information image 1511 and the depth image 1512. At this time, the generating unit 143 stores the learning dataset 151 in a memory or a storage unit described later. Similarly, the generating unit 143 generates a correct answer label image based on other arrangement information in the arrangement unit 141. Then, the generating unit 143 similarly generates a learning dataset 15m (m is a natural number equal to or greater than 1) for the other arrangement information and stores it in a memory or the like. Note that it is sufficient for the learning system 1 to generate at least one learning dataset.

[0039] The learning unit 144 learns the area estimation model 160 using a color information image 1511 in which a shooting area including an object is shot from a shooting position, a depth image 1512 which is depth information from the shooting position to the shooting area, and a correct label image 1513.

[0040] The area estimation model 160 is a model formula such as a function defined by parameters and variables, or a computer program. The area estimation model 160 is, for example, an estimation model that estimates a passable area of ​​a moving object from a shooting area corresponding to the color information image 1511. In particular, the area estimation model 160 may take a color information image and depth information as input data, and an image indicating the estimated passable area as output data. The area estimation model 160 is typically composed of a neural network. For example, the area estimation model 160 may be a neural network that performs semantic segmentation. In particular, the area estimation model 160 is preferably a CNN (Convolutional Neural Network) type. For example, the area estimation model 160 may set each pixel value of the color information image and the depth image to each element (neuron) of the input layer, and output the label of each pixel of the image indicating the passable area to each element of the output layer. In addition, various models and networks whose input and output are compatible with the above can be applied to the area estimation model 160.

[0041] Specifically, the learning unit 144 performs machine learning on the estimation model using a learning data set in which the color information image and the depth information are input data and the correct label information is correct data (teacher data). The learning unit 144 may apply a technique equivalent to statistical machine learning to the estimation model. For example, the learning unit 144 performs deep learning on the area estimation model 160 using the learning data sets 151 to 15m. That is, the learning unit 144 optimizes and updates the parameters 161 of the area estimation model 160 using the learning data sets 151 to 15m so as to improve the estimation accuracy of the passable area.

[0042] Fig. 2 is a block diagram showing the internal configuration of the learning system 1 according to the present embodiment 1. Fig. 2 shows an example in which the learning system 1 is realized by one information processing device, that is, a computer. However, the learning system 1 may be made redundant with multiple computers, or may be a computer system realized by multiple computers in which each functional block is different.

[0043] The learning system 1 includes a storage unit 11, a memory 12, a communication unit 13, and a control unit 14. The storage unit 11 is an example of a non-volatile storage device such as a hard disk or a flash memory. The storage unit 11 stores a simulator program 111, a learning program 112, arrangement information 113, learning data sets 151 to 15m, and a region estimation model 160. The learning program 112 may include the simulator program 111. In addition, a part or all of the arrangement information 113, the learning data sets 151 to 15m, and the region estimation model 160 may be stored in a storage device different from the storage unit 11. For example, a part or all of the arrangement information 113, the learning data sets 151 to 15m, and the region estimation model 160 may be stored in an external storage device connected to the learning system 1.

[0044] The simulator program 111 is a computer program in which a process corresponding to the simulator is implemented. For example, the simulator program 111 may be implemented with a process for arranging the object 31 and the camera 30 in the virtual space 3 and taking pictures using the camera 30. The learning program 112 is a computer program in which at least a process for generating correct answer label information and a process for learning the area estimation model 160 are implemented. The learning program 112 may further be implemented with a process for generating arrangement information using the simulator program 111, an arrangement process based on the arrangement information, and an acquisition process for a color information image and a depth image. That is, the learning program 112 may be implemented with processes corresponding to the arrangement unit 141, the acquisition unit 142, the generation unit 143, and the learning unit 144 described above.

[0045] The placement information 113 is information equivalent to the above-mentioned placement information 41. That is, the placement information 113 includes object information such as the type of object to be placed in the virtual space 3, a three-dimensional shape, and a texture to be applied to the surface, the number and positions (three-dimensional coordinates, etc.) of the objects to be placed, and the position of the camera 30, etc. (shooting position such as three-dimensional coordinates, etc.) When the placement process is performed multiple times, the placement information 113 may be a collection of multiple pieces of placement information corresponding to each time.

[0046] The learning dataset 151 is a set of a color information image 1511, a depth image 1512, and a correct answer label image 1513, and is information in which these are associated with each other. As described above, the color information image 1511 and the depth image 1512 are used as input data for the area estimation model 160. The correct answer label image 1513 is used as correct answer data in machine learning of the area estimation model 160. The correct answer label image 1513 is image data in which each pixel is assigned a label relating to passability. The learning datasets 152 to 15m have the same configuration as the learning dataset 151. The area estimation model 160 is as described above.

[0047] The memory 12 is a volatile storage device such as a RAM (Random Access Memory), and is a storage area for temporarily storing information during operation of the control unit 14. The communication unit 13 is a communication interface between the learning system 1 and an external device or a communication network. The communication unit 13 may be realized, for example, by a general-purpose or dedicated circuit realized by a semiconductor device.

[0048] The control unit 14 controls each component of the learning system 1. The control unit 14 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), or a quantum processor (quantum computer control chip). The control unit 14 loads the simulator program 111 and the learning program 112 from the storage unit 11 into the memory 12, and executes the simulator program 111 and the learning program 112. In this way, the control unit 14 realizes the functions of the arrangement unit 141, the acquisition unit 142, the generation unit 143, and the learning unit 144 described above.

[0049] Furthermore, when some or all of the components of the learning system 1 are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally or distributed. For example, the information processing devices, circuits, etc. may be realized as a client-server system, cloud computing system, etc., in a form in which each is connected via a communication network.

[0050] Returning to Fig. 1 for further explanation, the area estimation system 2 is an information system for estimating a passable area from images captured by a camera mounted on a moving object 20 in a given traffic environment 200 using an area estimation model 160 trained by the learning system 1, and for controlling the operation of the moving object 20 based on the passable area.

[0051] The traffic environment 200 is a real environment in which physical objects 201 to 20k (k is a natural number equal to or greater than 1) are arranged and in which the mobile body 20, which is a robot, travels autonomously and performs a predetermined operation. The traffic environment 200 is typically a factory where luggage, machines, etc. are arranged in various places, but it may also be an ordinary road, sidewalk, etc.

[0052] The area estimation system 2 performs area estimation using an area estimation model 160 in order to control the movement plan and operation of the moving body 20 in the traffic environment 200. In the following, a description will be given assuming that the trained area estimation model 160 is stored in the moving body 20. However, the moving body 20 may refer to the trained area estimation model 160 stored in an external storage device via wireless communication.

[0053] 3 is a block diagram showing an internal configuration of the moving body 20 according to the first embodiment. The moving body 20 includes a storage unit 21, a memory 22, a communication unit 23, a control unit 24, a camera 25, a traveling unit 26, and a grip unit 27. The storage unit 21 is an example of a non-volatile storage device such as a hard disk or a flash memory. The storage unit 21 stores a movement plan program 211, a depth image estimation model 212, and an area estimation model 213. The movement plan program 211 is a computer program in which area estimation processing, movement plan processing, and operation control processing, which will be described later, are implemented.

[0054] The depth image estimation model 212 is an AI model that estimates depth information, which is depth information, for each pixel in the same shooting area as the input color information image, and generates a depth image. The depth image estimation model 212 is a model formula such as a function defined by parameters and variables, or a computer program. In particular, the depth image estimation model 212 takes a color information image as input data, estimates depth information, which is depth information, for each pixel in the same shooting area as the color information image, and outputs a depth image generated by the estimated depth information as output data. The depth image estimation model 212 may estimate and output a depth image from a pair of color information images captured by a stereo camera of the moving body 20. Alternatively, the depth image estimation model 212 may estimate and output a depth image from a color information image captured by a monocular camera of the moving body 20. The depth image estimation model 212 is assumed to have been machine-learned in advance using a large number of learning datasets. The depth image estimation model 212 is typically configured as a neural network, but is not limited thereto.

[0055] The area estimation model 213 is the area estimation model 160 that has been trained by the above-mentioned learning system 1 and stored in the moving body 20.

[0056] The memory 22 is a volatile storage device such as a RAM, and is a storage area for temporarily holding information during operation of the control unit 24. The communication unit 23 is a communication interface between the mobile object 20 and an external device or a communication network. The communication unit 23 may be realized, for example, by a general-purpose or dedicated circuit realized by a semiconductor device.

[0057] The control unit 24 controls each component of the moving body 20. The control unit 24 is, for example, a processor such as a CPU, a GPU, an FPGA, or a quantum processor. The control unit 24 reads the movement plan program 211 from the storage unit 21 into the memory 12 and executes the movement plan program 211. In addition, the control unit 14 reads the depth image estimation model 212 and the area estimation model 213 into the memory 12 as necessary and executes the depth image estimation model 212 and the area estimation model 213. With these, the control unit 14 realizes the functions of the image acquisition unit 241, the depth acquisition unit 242, the estimation unit 243, the movement planning unit 244, and the operation control unit 245 described above. Note that the depth image estimation model 212 and the area estimation model 213 may be stored in an external storage device connected to the moving body 20. In this case, the control unit 14 transmits and receives data to and from the depth image estimation model 212 and the area estimation model 213 via the communication unit 13, thereby acquiring a depth image and acquiring an area estimation image.

[0058] The image acquisition unit 241 controls the camera 25 during a movement plan or the like to acquire a captured image captured by the camera 25. The image acquisition unit 241 acquires at least a color information image. The depth acquisition unit 242 inputs the color information image acquired by the image acquisition unit 241 to the depth image estimation model 212 and acquires a depth image from the depth image estimation model 212. The estimation unit 243 inputs the color information image and the depth image to the area estimation model 213 and acquires an estimation result of an area in which the moving body 20 can pass within the shooting area as an area estimation image. The movement plan unit 244 generates a movement plan for the moving body 20 using the area estimation image. Here, the "movement plan" may be, for example, a route for moving from the current position of the moving body 20 to a destination by passing through a passable area based on the area estimation image. The operation control unit 245 controls the operation of the moving body 20 based on the generated movement plan. Specifically, the operation control unit 245 instructs the traveling unit 26 to travel to the passable area. Furthermore, the operation control unit 245 instructs the gripping unit 27 to perform a gripping operation as necessary.

[0059] The camera 25 is a physical photographing device. The camera 25 may be a stereo camera, which is a set of a plurality of camera devices. The camera 25 photographs the photographing area in the traffic environment 200 periodically or in response to a control signal or the like from the control unit 24, and outputs the photographed image to the control unit 24. At this time, it is assumed that the photographing area may include a part or the whole of the object 201, etc.

[0060] The running unit 26 is a wheel, a leg, or the like for running on the ground, which is a running area in the traffic environment 200. The gripping unit 27 is a physical configuration for gripping the object 201, etc. The gripping unit 27 is, for example, a robot hand, but is not limited to this. The running unit 26 and the gripping unit 27 operate in response to a control signal, an instruction, a command, etc. from the control unit 24.

[0061] FIG. 4 is a flowchart showing the flow of the learning process according to the first embodiment. First, the arrangement unit 141 generates random arrangement information 41 of any objects 31 to 3n and the camera 30 in the virtual space 3 (S11). Specifically, the arrangement unit 141 randomly selects the number of objects to be arranged, and randomly selects the above-mentioned object information and position (three-dimensional coordinates) for each of the selected objects. At this time, the arrangement unit 141 may randomly select the type, three-dimensional shape (size), and texture information for each selected object. In addition, the arrangement unit 141 randomly selects the type (function) and position (three-dimensional coordinates) of the camera 30 to be arranged. Then, the arrangement unit 141 generates arrangement information 41 including the object information and position of each of the selected objects, and the type and position of the camera 30, and stores it in the storage unit 11 or the memory 12.

[0062] Next, the placement unit 141 places the object 31 and the camera 30 in the virtual space 3 based on the generated placement information 41 (S12). For ease of explanation, it is assumed below that the shooting range of the camera 30 includes a plurality of objects including the objects 31 and 32.

[0063] Then, the acquisition unit 142 acquires the color information image 42 and the depth image 43 captured by the camera 30 in the virtual space 3 (S13).

[0064] FIG. 5 is a diagram showing an example of a color information image 42 captured in the virtual space 3. The color information image 42 is data including XY coordinates and color information of each pixel. The color information image 42 is an image expressed in a plurality of colors, such as RGB, CMYK, and grayscale. For example, the color information image 42 shows that a part of the object area 310 and the object area 320 overlap from the viewpoint of the shooting position of the camera 30. Here, the object area 310 is an area corresponding to the object 31 in the shooting area of ​​the camera 30, and the object area 320 is an area corresponding to the object 32 in the shooting area of ​​the camera 30. Also, the color information image 42 shows that a point on the back side of the object 31 cannot be identified in the object area 310.

[0065] FIG. 6 is a diagram showing an example of a depth image 43 captured in the virtual space 3. The depth image 43 is an image captured from the same capturing position as the camera 30, in the same capturing area (captured range) as the color information image 42. The depth image 43 is an image in which the depth of each object, wall, and floor from the capturing position is indicated by shading. The depth image 43 shows an example in which the closer to the capturing position the darker the color is, and the farther from the capturing position the lighter the color is. Note that the depth image 43 may show the distance (depth length) from the capturing position to each point in each pixel as a numerical value or the like. However, the method of expressing the depth image 43 and the depth information is not limited to this.

[0066] Returning to FIG. 4, the description will be continued. The generating unit 143 generates a passable area image (correct answer label image 45) (distinguishing whether it is the back side of the object or not) by using the arrangement information 41 (S14). For example, the generating unit 143 specifies arrangement information on a two-dimensional plane in the virtual space 3 from the arrangement information 41. For example, the generating unit 143 specifies a plan layout diagram from the arrangement information 41. At this time, the plan layout diagram may be an image binarized with an object area on the plane and an object non-existence area. In other words, it can be said that the plan layout diagram is given a label indicating that the object area is impassable and a label indicating that the object non-existence area on the plane is passable. Then, the generating unit 143 generates a passable area image by converting the plan layout diagram into a coordinate system in which the shooting area of ​​the color information image 42 is shot from the shooting position of the camera 30. In other words, the generating unit 143 converts the plan layout diagram obtained by viewing the virtual space 3 from above into the coordinate system of the viewpoint of the camera 30 to deform the object shape on the plane. Then, the generating unit 143 may mask the transformed (converted) figure on the color information image 42 in accordance with the coordinate system of the color information image 42 to generate a passable area image. At this time, since an area where no object exists on the plane (at least an area other than the horizontal projection plane of the non-ground surface) is originally an empty area, it is an area where no object exists even after the conversion, and a passable label is assigned to it. On the other hand, an area where an object exists on the plane (at least the ground surface) is an area where an object exists even after the conversion, and a non-passable label is assigned to it. In other words, it can be said that the generating unit 143 generates an image in which the height direction of each object included in the color information image 42 is compressed as a passable area image. The generating unit 143 generates a passable area image by assigning a label indicating non-passable to an area recognized as an area where passage is impossible in the color information image 42, such as a wall area. This is because a wall area, etc. may not appear in a planar layout drawing.

[0067] Here, first, an example of a case where a correct label image is generated by the existing technology without using the generation unit 143 according to the present embodiment is shown. Fig. 7 is a diagram showing an example of a previous correct label image 44 in a state where the back side of an object in the virtual space 3 cannot be identified. The previous correct label image 44 shows that the object region 310, the object region 320, and the wall region in Fig. 5 described above are not distinguished, and all regions are labeled as impassable by a moving object.

[0068] Next, FIG. 8 is a diagram showing an example of a correct label image 45 of a passable area in the virtual space 3 according to the first embodiment. The correct label image 45 shows that the object area 310, the object area 320, and the wall area in FIG. 5 described above are distinguished, and in particular, the point on the back side of the object 31 is labeled as passable by a moving object. FIG. 9 is a diagram showing an example in which different labels are given to the ground surface and the non-ground surface in the object area 310 to clarify this point. Here, the object area 310 to which the object 31 corresponds from the viewpoint from the shooting position of the camera 30 is distinguished into a ground surface 311 and a non-ground surface 312. The ground surface 311 is an area where the object 31 is in contact with the ground. The non-ground surface 312 is an area where the object 31 is not in contact with the ground. However, the non-ground surface 312 excludes the area where the ground surface 321 of the object area 320 and the object area 310 overlap. Here, the non-ground surface 312 is originally an area that cannot be distinguished from the ground surface 311 from the viewpoint from the shooting position of the camera 30. However, the generation unit 143 can identify the non-ground surface 312 in the object area 310 by using the planar arrangement information based on the arrangement information 41. The correct labeled image 45 can be said to be an image in which passable areas and impassable areas of a moving object are classified, that is, labeled, even from the viewpoint from the shooting position of the camera 30.

[0069] Returning to FIG. 4, the explanation will be continued. The generation unit 143 stores the color information image 42 (1511) and the depth image 43 (1512) acquired in step S13, and the passable area image (correct label image 45 (1513)) generated in step S14 in the storage unit 11 as a learning dataset 151 (S15). Then, the generation unit 143 determines whether the number of sets in the learning dataset is a specified number (S16). If the number of sets does not reach the specified number in step S16, the process returns to step S11. Then, the learning system 1 executes the processes from steps S11 to S16 again.

[0070] On the other hand, in step S16, if the number of sets meets the specified number, the learning unit 144 performs machine learning of the region estimation model 160 using the stored learning data sets 151 to 15m (S17). For example, the learning unit 144 sets the color information image 1511 and the depth image 1512 included in the learning data set 151 to the input layer of the region estimation model 160, performs calculations between each layer using the parameters 161, and acquires the calculation results of the output layer. Then, the learning unit 144 obtains the difference between the label of each pixel included in the calculation result and the label of the corresponding pixel of the correct answer label image 1513, and judges whether the difference is equal to or greater than a predetermined range. If the difference is equal to or greater than the predetermined range, the learning unit 144 corrects the parameters 161 and performs the same calculation and judgment as above again. If the difference is less than the predetermined range, the learning unit 144 repeats the same process for each of the unlearned learning data sets 152 to 15m. Then, the learning unit 144 judges whether the convergence condition of the machine learning is satisfied. For example, in all of the learning data sets 152 to 15m, when the difference between the calculation result and the correct label image falls within a predetermined range or exceeds a predetermined number of repetitions, the learning unit 144 determines that the convergence condition is satisfied. The above-mentioned process is an example of machine learning. Therefore, the learning unit 144 may perform machine learning by applying various algorithms such as backpropagation. Furthermore, the generating unit 143 may use, for example, binary cross entropy loss in the loss calculation. However, the method of loss calculation is not limited to this. When the machine learning satisfies the convergence condition, the learning unit 144 updates the parameter 161 in the area estimation model 160 to a value at the time of convergence. Then, the learning system 1 ends the learning process.

[0071] Fig. 10 is a flowchart showing the flow of the area estimation process according to the present embodiment 1. Here, it is assumed that the moving body 20 has registered the area estimation model 160 learned by the learning process of Fig. 4 as the area estimation model 213. Then, the moving body 20 starts the area estimation process to perform a predetermined operation in the traffic environment 200.

[0072] First, the image acquisition unit 241 acquires a color information image 52 captured by the camera 25 of the moving object 20 in the traffic environment 200 (S21). For ease of explanation, it is assumed below that the shooting range of the camera 25 includes a plurality of objects including objects 201 to 20k. Fig. 11 is a diagram showing an example of the color information image 52 acquired in step S21.

[0073] Next, the depth acquisition unit 242 acquires a depth image 53 by inputting the color information image 52 to the depth image estimation model 212 (S22). At this time, the depth image estimation model 212 estimates depth information of each pixel from the input color information image 52, and generates and outputs a depth image 53 based on the estimated depth information. In this way, the depth acquisition unit 242 acquires the depth image 53 from the depth image estimation model 212. FIG. 12 is a diagram showing an example of the depth image 53.

[0074] Next, the estimation unit 243 inputs the color information image 52 and the depth image 53 to the area estimation model 213 to obtain an area estimation image 55 of the passable area (S23). At this time, the estimation unit 243 sets the color information image 52 and the depth image 53 to the input layer of the area estimation model 213, performs calculations between layers using learned parameters, and obtains the calculation results of the output layer. Here, the calculation results are the area estimation image 55. The area estimation image 55 can also be said to be a mask image of the passable area.

[0075] Here, first, an example of a case where a region estimation image is generated by an existing technology without using the trained region estimation model 213 according to this embodiment is shown. Fig. 13 is a diagram showing an example of a previous region estimation image 54 in a state where the back side of an object cannot be identified. The previous region estimation image 54 is an example in which a passable region 541 is superimposed on a depth image 53. In the previous region estimation image 54, the regions of the back side of a chair and the back side of a box are not identified, and it is shown that the entire object region including the regions of the back sides of the chair and the box are not passable.

[0076] Next, FIG. 14 is a diagram showing an example of an area estimation image 55 of a passable area according to the first embodiment. The area estimation image 55 is an example in which a passable area 551 is superimposed on the depth image 53. In the area estimation image 55, the area behind the chair and the box are identified, and it is shown that the area behind the chair and the box is passable. In the area estimation image 55 of FIG. 14, the passable area 551 and other areas (impassable areas) may be displayed in a different manner. For example, similar to the above-mentioned correct label image 45, the passable area 551 may be displayed in white, and the impassable area of ​​the area estimation image 55 may be displayed in black or hatched. In other words, the expression of the area estimation image 55 is not limited to these.

[0077] Returning to FIG. 10, the description will be continued. The movement planner 244 generates a movement plan using the area estimation image 55 (S24). Specifically, the movement planner 244 generates route information for the moving body 20 to reach the destination from the current position via the passable area 551 of the area estimation image 55. The destination may be inside or outside the area estimation image 55. Alternatively, the movement planner 244 can be realized in various forms based on the area estimation image 55. For example, the movement planner 244 may overlay the depth image 53 and the passable area 551, convert the unmasked pixels of the passable area 551 into the three-dimensional position (coordinate system) of the shooting position of the camera 25 based on the depth value, and generate a point set of obstacles in the passage. The point set of obstacles may be called an obstacle point cloud. Then, the movement planner 244 may generate route information to the destination so as to avoid the obstacle point cloud. Alternatively, the movement planning unit 244 may learn an AI model that directly calculates the speed of the moving object 20 from the area estimation image 55 and the direction of the destination. Then, the movement planning unit 244 may perform movement planning using the learned model.

[0078] Thereafter, the motion control unit 245 controls the motion of the moving object 20 based on the movement plan generated in step S24 (S25). For example, the motion control unit 245 instructs the traveling unit 26 to travel to the destination according to the route information included in the movement plan. In addition, the motion control unit 245 may instruct the grip unit 27 to perform a predetermined motion after reaching the destination.

[0079] In this way, in the learning system 1 of the present embodiment, by using the arrangement information of the object in the virtual space, it is possible to accurately assign a label regarding the passability of the area behind the object in the captured image. Then, it is possible to efficiently generate a large amount of learning data sets including such correct label information. Then, by using such learning data sets, it is possible to improve the learning efficiency of the estimation model and the estimation accuracy. Therefore, it is possible to accurately estimate the passable area including the area behind the object in the captured image using the trained estimation model.

[0080] Here, the learning data set of this embodiment uses images captured in a virtual space, which is an artificial environment, as input data. That is, since the images are generated by a simulator, there is little noise. On the other hand, since the traffic environment 200 is a real environment, the captured images of the real environment are affected by the brightness of lighting, sunlight, etc., unlike the above-mentioned virtual space 3. Therefore, the captured images of the real environment may contain noise. Therefore, there are cases where the accuracy of estimating the passable area for the captured images in the real environment is not sufficiently improved by simply using a color information image to input the learning data of the estimation model. Such a problem is sometimes called a Sim2Real problem.

[0081] Therefore, in this embodiment, in addition to the color information image, depth information such as a depth image of the shooting range corresponding to the color information image is used as input of learning data for the estimation model. Then, in the area estimation process, a highly accurate depth image is obtained from the color information image using the depth image estimation model 212. This reduces the influence of brightness such as lighting in the real environment, that is, the occurrence of the Sim2Real problem can be suppressed.

[0082] The following is an example of a method for acquiring a highly accurate depth image in a real environment. First, the input layer of the depth image estimation model corresponds to each pixel value of one color information image. In this case, the depth acquisition unit 242 inputs an image captured by a monocular camera of the moving body 20 to the depth image estimation model to acquire a depth image. Alternatively, the input layer of the depth image estimation model may correspond to each pixel value of a plurality of color information images. In this case, the depth acquisition unit 242 inputs an image captured by a stereo camera of the moving body 20 to the depth image estimation model to acquire a depth image. Alternatively, the moving body 20 may be equipped with an RGBD sensor. In this case, the image acquisition unit 241 and the depth acquisition unit 242 acquire a color information image and a depth image by the RGBD sensor. Alternatively, the moving body 20 may further be equipped with 3DLiDAR (Light Detection And Ranging). In this case, the image acquisition unit 241 and the depth acquisition unit 242 acquire a color information image and a depth image from a combination of a monocular camera and a 3DLiDAR.

[0083] Moreover, the area estimation process according to the present embodiment may be applied not only to a movement plan but also to a manipulation plan or task planning for a robot or the like. For example, the moving body 20 is provided with an operation planning unit. The operation planning unit may estimate a working space in which manipulation by the gripping unit 27 is possible based on the area estimation image estimated by the estimation unit 243, and generate an operation plan for controlling the operation by inserting the gripping unit 27 behind an object. Alternatively, the operation planning unit may use the area estimation image to add the area behind the object to candidates for a placement location when selecting a placement location for an object when changing the placement of the object. This allows a more flexible manipulation plan to be generated. Furthermore, the moving body 20 may be provided with a task planning unit. The task planning unit may use the area estimation image to plan various tasks for the moving body 20, explicitly considering the area behind the object.

[0084] In addition, the "information on the passability of the moving body" according to this embodiment, that is, the label, may use three values, namely, passable, impassable, or the need to confirm the passability. Specifically, the generation unit 143 may assign a label indicating that the area behind the object is passable or the need to confirm the passability. In this case, the estimation unit 243 may acquire an area estimation image including a passable area and an area where the passability needs to be confirmed from the area estimation model 213. Then, the movement plan unit 244 may generate a movement plan so that the moving body 20 moves toward the area where the passability needs to be confirmed in the area estimation image, photographs the area behind the object with a camera, and confirms the actual passability. This is because when the area behind the object is large, the uncertainty of the actual passability is high.

[0085] The moving body 20 is not limited to an autonomously moving robot as described above, but may also be applied to an automatically driven vehicle, a vehicle driven by a human, or the like.

[0086] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0087] The present disclosure is not limited to the above-described embodiment, and may be modified as appropriate without departing from the spirit and scope of the present disclosure. In addition, the present disclosure may be implemented by appropriately combining the respective embodiments. [Explanation of symbols]

[0088] 1. Learning System 11 Storage section 111 Simulator Program 112 Study Programs 113 Placement information 151 Training Dataset 1511 Color Information Image 1512 depth images 1513 correct label images 15m training dataset 160 Area Estimation Model 161 Parameters 12. Memory 13. Communications Department 14 Control section 141 Placement section 142 Acquisition Department 143 Generation part 144 Learning Department 2. Area Estimation System 200 Traffic environment 201 Object 20k objects 20 Mobile 21 Memory section 211 Mobility Planning Program 212 Depth Image Estimation Model 213 Area Estimation Model 22 Memory 23 Communications Department 24 Control Unit 241 Image Acquisition Unit 242 Depth acquisition section 243 Estimation Department 244 Mobility Planning Department 245 Motion control section 25 Camera 26 Running part 27 Gripping part 3. Virtual Space 30 Camera 31 Object 310 Object area 311 Ground plane 312 Non-contact surface 32 Object 320 Object area 321 Ground plane 3n object 41 Placement information 42 Color Information Image 43 Depth image 44 Old correct label image 45 Correct label images 52 Color Information Image 53 Depth image 54 Old area estimation image 541 Passable area 55 Region Estimation Image 551 Passable area

Claims

1. a generation unit that uses location information of an object placed at an arbitrary position in a virtual space to generate correct label information including information regarding whether a moving object can pass through a point that is a rear side of the object from a viewpoint from an arbitrary shooting position; a learning unit that learns an estimation model for estimating a passable area of ​​the moving body from within the photographed area, using a color information image of a photographed area including the object photographed from the photographing position, depth information from the photographing position to the photographed area, and the correct label information; A learning system comprising:

2. The generating unit generates the correct label information by regarding, among the object regions corresponding to the object in the photographing region, a region other than a horizontal projection plane when the object is viewed from above based on the arrangement information, as a point on the back side of the object from a viewpoint from the photographing position. The learning system according to claim 1 .

3. The generating unit generates the correct label information by making at least a ground surface of the object specified based on the arrangement information impassable and by making at least a region other than the horizontal projection plane passable. The learning system according to claim 2 .

4. The generating unit generates the correct label information by designating an area of ​​the horizontal projection plane including the ground surface as impassable within the object area. The learning system according to claim 3 .

5. The generation unit generates the correct label information such that a different label regarding passability is assigned to each point included in the object region depending on whether the point is on the back side of the object from a viewpoint from the shooting position. The learning system according to claim 2 .

6. The generating unit generates, as the correct label information, a passable area image in which a label regarding passability is assigned to each pixel of an image of the shooting area photographed from the shooting position. A learning system according to claim 1 or 2.

7. The generation unit identifies planar arrangement information including the object in the virtual space from the arrangement information, and generates the passable area image from the viewpoint from the shooting position using the planar arrangement information. The learning system according to claim 6.

8. The generation unit generates the correct label information so that a label indicating that confirmation of passability is required is added to a point on the back side of the object. A learning system according to claim 1 or 2.

9. The learning unit performs machine learning on the estimation model using a learning data set in which the color information image and the depth information are input data and the correct label information is correct data. A learning system according to claim 1 or 2.

10. a placement unit that places the object at an arbitrary position in the virtual space and places an image capture device at the image capture position; an acquisition unit that acquires the color information image in which the photographing area is photographed from the photographing position by the photographing device, and acquires, as the depth information, a depth image in which a distance from the photographing position to the photographing area is measured; Further equipped A learning system according to claim 1 or 2.

11. The computer Using location information of an object placed at an arbitrary position in the virtual space, generate correct answer label information including information regarding whether or not a moving object can pass through a point that is a rear side of the object from a viewpoint from an arbitrary shooting position; learning an estimation model for estimating a passable area of ​​the moving body from within the photographed area using a color information image of a photographed area including the object photographed from the photographing position, depth information from the photographing position to the photographed area, and the correct label information; How to learn.

12. A generation process for generating correct answer label information including information on whether a moving object can pass through a point on the back side of the object from a viewpoint from an arbitrary shooting position, using the position information of the object placed at an arbitrary position in the virtual space; a learning process for learning an estimation model for estimating a passable area of ​​the moving body from within the photographed area, using a color information image of a photographed area including the object photographed from the photographing position, depth information from the photographing position to the photographed area, and the correct label information; A learning program that causes a computer to execute the following:

13. A region estimation system comprising an estimation unit that estimates the passable region in a second region by inputting a second color information image of a second region including a second object photographed from a second shooting position in a specified traffic environment and second depth information from the second shooting position to the second region to the estimation model trained by the learning system described in claim 1.

Images