Learning device

Abstract

Provided is a system capable of searching for an appropriate region around a destination location in order to enable a moving body to achieve a designated state in accordance with an instruction, in view of the intention of an instructor who designates an instruction hidden in ambiguity in a space based on the destination location. A learned model is constructed using, as input data, an instruction from a user and scene maps (SG1-SG3) created on the basis of an environment image corresponding to the position of a moving body (20) and a direction facing a designated location. The feature amount of the primary node constituting the state scene graph (SG1) is defined on the basis of the relative arrangement relationship (distance and angle) with respect to each target with the position of the moving body (20) as a reference. Feature quantities of primary nodes constituting the state scene graph (SG1) are defined on the basis of the space occupancy of each target.

Description

Technical Field

[0001] The present invention relates to a learning device that constructs a learned model that helps achieve a specified state of an object in a specified space around a specified place. Background Art

[0002] A method for generating a scene graph from an image is proposed (for example, refer to non-patent documents 1 and 2). According to the method, the following steps are performed: input an image; detect an object from the image using a deep learning-based object detection method; use PLSI to detect context in the image; use a deep learning-based relationship detection and ontology method to detect the relationship between objects; and generate a scene graph for the input image.

[0003] Prior art literature

[0004] Non-patent literature

[0005] Non-patent literature 1: Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions, CVPR2020 (https: / / arxiv.org / pdf / 2004.03967v1.pdf)

[0006] Non-patent document 2: Multi-Layer Semantic and Geometric Modeling with NeuralMessage Passing in 3D Scene Graphs for Hierarchical Mechanical Search, ICRA2020 (https: / / arxiv.org / pdf / 2012.04060.pdf) Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, according to the prior art, even if a user instructs a mobile body such as a robot to "stop on the right side of ○○ (for example, the name of a store or facility, etc.)", it is difficult to make the mobile body stop in the area corresponding to the "right side of ○○" intended by the user. This is because, although the coordinates of a point are required to stop the mobile body, the expression "right side" included in the user's instruction does not uniquely express this point. Originally, the user did not refer to the coordinates of a uniquely determined point when using expressions such as "right side", but mostly referred to "space" such as the right side. Therefore, it is necessary to associate the words contained in the user's instructions with the space. In addition, in a space such as "right side", there are spaces where the mobile body can stop and spaces where it cannot stop. For example, if the "right side of ○○" is an open space, it will stop, but if it is a crosswalk, it cannot stop.

[0009] Therefore, the purpose of the present invention is to provide a device for generating a learned model in view of the intention of the indicator hidden in the vague instructions of the spatial designation based on the destination, which can search for an appropriate area around the destination in order to make the object achieve the specified state according to the instructions.

[0010] Means for solving problems

[0011] The learning device of the present invention generates a learned model in the following manner: an indication of an object related to the realization of a specified state in a specified space surrounding a specified place, position information of the object, multiple scene graphs produced based on images of the surroundings of the specified place obtained based on the positional relationship between the object and the specified place, and a result of whether the specified state of the object can be realized are used as learning data, and one area candidate among multiple area candidates existing in multiple surrounding spaces based on the specified place is output. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is an explanatory diagram related to the configuration of a learning device and a moving body assisting device.

[0013] Figure 2 This is an explanatory diagram related to the generation function of the learned model.

[0014] Figure 3 This is an illustration related to an image containing multiple objects.

[0015] Figure 4 This diagram is an explanatory diagram regarding the result of projecting a three-dimensional high-definition map onto a two-dimensional map.

[0016] Figure 5 This is an illustration of a state scene graph.

[0017] Figure 6This is an illustration of a layout scene graph.

[0018] Figure 7 This is an illustrative diagram indicating an example of a scene graph.

[0019] Figure 8 This is a conceptual illustration of the successive convolutions and pooling of the scene graph.

[0020] Fig. 9 This is an illustration of graph neural network.

[0021] Fig.10 A conceptual illustration of the successive convolutions and pooling of a scene graph input to a graph neural network.

[0022] Fig.11 This is an explanatory diagram related to correct solution data in different driving scenarios.

[0023] Fig.12 This is an explanatory diagram regarding correct solution data in driving scenarios with different obstacle presence patterns.

[0024] Fig.13 This is an explanatory diagram related to the region candidate output function of the moving body assistance system. DETAILED DESCRIPTION

[0025] (structure)

[0026] Figure 1 The learning device 100 and the mobile body assisting device 200 shown as one embodiment of the present invention are each configured as a device capable of accessing a database 102 via a network in order to assist a mobile body 20 (equivalent to the "object" of the present invention) in achieving a specified state. The mobile body 20 and the mobile body assisting device 200 constitute a "mobile body system".

[0027] The database 102 stores and saves environmental images (equivalent to the "image" of the present invention) representing the surrounding conditions of the mobile body 20, three-dimensional high-definition maps (map information), graphic neural network diagrams, and learned models, etc. In the present embodiment, the database 102 is composed of a device or a database server different from the learning device 100 and the mobile body assistance device 200, but the database 102 may also be a structural element of the learning device 100 and / or the mobile body assistance device 200.

[0028] The learning device 100 has a first scene graph production element 110 and a learned model generation element 120. The first scene graph production element 110 and the learned model generation element 120 are respectively composed of arithmetic processing elements such as a CPU and / or a processor core, storage elements such as a ROM and / or a RAM, and an input / output interface circuit. The first scene graph production element 110 and the learned model generation element 120 are respectively configured to perform designated tasks such as scene graph production and learned model generation described later. The functional element is configured to perform a designated task means that the hardware constituting the functional element reads the software from the storage element and reads the data as needed, and performs arithmetic processing based on the software with the data or other data as the object, thereby performing the designated task.

[0029] The mobile body assistance device 200 includes a second scene graph creation element 210 and a region candidate output element 220. The second scene graph creation element 210 and the region candidate output element 220 are respectively composed of a CPU and / or a processor core and other computing elements, a ROM and / or a RAM and other storage elements, and an input / output interface circuit, etc. The second scene graph creation element 210 and the region candidate output element 220 are respectively configured to perform designated tasks such as scene graph creation and learning completed model generation described later.

[0030] The learning device 100 and the moving body assistance device 200 may be configured by the same device. In this case, the first scene graph creation element 110 and the second scene graph creation element 210 may be configured by a single scene graph creation element.

[0031] The mobile body 20 is composed of a vehicle or a robot having an autonomous movement function, a positioning function, and a wireless communication function. The mobile body 20 includes a mobile body control device 21 and a camera 22. The mobile body 20 may also be composed of an information processing terminal (e.g., a smartphone) that is carried by a user and moves passively with the movement of the user. The mobile body assistance device 200 may also be composed of a device (e.g., a mobile body control device 21) mounted on the mobile body 20.

[0032] The mobile body control device 21 is composed of arithmetic processing elements such as a CPU and / or a processor core, storage elements such as a ROM and / or a RAM, and an input / output interface circuit. The mobile body control device 21 is configured to control the autonomous movement function, the positioning function, and the wireless communication function of the mobile body 20. The camera 22 is mounted on the mobile body 20 to capture the moving direction of the mobile body 20 or the situation in front. The mobile body 20 may also have a function of adjusting the shooting direction (optical axis direction) of the camera 22 and / or a function of measuring the shooting direction.

[0033] (Learning completes model generation function)

[0034] Through the learned model generation function, a learned model is generated based on instructions related to the specified state of the mobile body 20 in the specified space around the specified place and the following environmental image, which shows the state of the specified place and its surroundings obtained based on the position of the mobile body 20 and the direction facing the specified place.

[0035] Specifically, the user's instruction to the mobile object 20 through the input interface of the device owned by the user is sent from the device to the learning device 100 and is recognized by the first scene graph creation element 110 ( Figure 2 / Step 100). The environment image may be stored in the database 102, or may be directly sent from the device to the learning apparatus 100.

[0036] An "instruction" is an instruction related to a designated state of the mobile body 20 in a designated space around a designated place. Thus, for example, an instruction "Please stop on the right side of X" is recognized as an instruction to realize a stopped state of the mobile body 20 as a designated state in a space on the right side of the designated space around the designated place represented by the word X. In addition, an instruction "Please slow down in front of Y" is recognized as an instruction to realize a state of starting to slow down the mobile body 20 as a designated state in a space on the front side of the designated space around the designated place represented by the word Y. Furthermore, an instruction "Please pass on the left side of Z" is recognized as an instruction to realize a passing state of the mobile body 20 as a designated state in a space on the left side of the designated space around the designated place represented by the word Z.

[0037] The user who issues the instruction may be a user who is on the moving body 20 or a user who is located at a different place from the moving body 20. The instruction from the user may be an instruction by voice or an instruction by gesture.

[0038] The camera 22 mounted on the mobile body 20 acquires an environment image, which shows the state of the designated place and its surroundings acquired based on the position of the mobile body 20 and the direction facing the designated place (the shooting direction of the camera 22). Figure 2 / Step 102). The environment image may be stored in the database 102 or may be directly sent from the mobile body 20 to the learning device 100.

[0039] Thus, for example, Figure 3 As shown in FIG. 1 , the following environment image is obtained, which includes a building X0 (building), a sidewalk grid X0 extending along the lower edges of the two side surfaces of the building X0, and a sidewalk grid X1 extending along the lower edges of the two side surfaces of the building X0. 11 and X 12, extending from building X0 to sidewalk grid X 11 , X 12 The outer lane grid X 21 ~X 26 , and erected on the sidewalk grid X 12 With Roadway GridX 24 Trees at the border of X 41 , TreesX 42 There is a shop sign X on one side of the building X0. 01 and WindowX 02 , with window X03 on the other side. Figure 3 The illustrated environment image also includes a vehicle X5 and a pedestrian X as traffic participants. 61 ~X 64 .

[0040] Based on the position of the mobile body 20 (at the time of acquiring the environment image), the environment image and the map information, the state scene graph SG1 is generated by the first scene graph generation element 110 ( Figure 2 / Step 111).

[0041] Map information is, for example, a three-dimensional high-definition map, including static information such as three-dimensional structures, road surface information, and lane information. Here, it is defined as distinguishing the types and / or attributes of targets or objects through labels. For example, labels are used to distinguish targets that are above a certain height from the ground and targets that extend along the terrain. The label is defined by the label area (the area occupied by the labeled target in the environment image) and the label ID.

[0042] As a first-level target, "targets with a certain height above the ground" are classified into second-level targets such as buildings, columnar structures, and trees. As a second-level target, "buildings" are classified into third-level targets such as side walls, store billboards, windows, and entrances and exits for people or vehicles. As a second-level target, "columnar structures" are classified into third-level targets such as traffic light poles, traffic sign poles, and communication poles. Targets after the third-level target can be classified more finely.

[0043] "Targets extending along the terrain" as first-level targets are classified into second-level targets such as roadways and sidewalks, for example. "Roadways" as second-level targets are divided into multiple roadway grids as third-level targets, and each roadway grid is defined as a separate target. "Roadway grids" as third-level targets are classified into fourth-level targets such as crosswalks, center lines, roadway boundaries, and road markings such as zebra crossings. "Sidewalks" as second-level targets are divided into multiple sidewalk grids, and each sidewalk grid is defined as a separate target. "Sidewalk grids" as third-level targets are classified into fourth-level targets such as road markings such as blind paving blocks. Targets after the fourth-level targets can be classified more finely.

[0044] Labels defined in the three-dimensional high-definition graph are assigned to the targets in the environment image. Labels are also assigned to targets that meet dynamic information, such as vehicles on the roadway, pedestrians on the sidewalk or the roadway (crosswalk). In the state scene graph SG1, each target (or its label) assigned a label is defined as a primary node.

[0045] Figure 4 The result of projecting static objects (buildings, sidewalk grids, and road grids) of a 3D HD map into a 2D map is shown. Figure 4 The 2D maps shown include: Figure 3 Among the objects included in the environment image shown, there are a building X0 (building) as a static object, a sidewalk grid X extending along the lower edges of the two side surfaces of the building X0, and a sidewalk grid X extending along the lower edges of the two side surfaces of the building X0. 11 , X 12 , and the roadway grid X 21 ~X 26 By using the two-dimensional graph, it is possible to improve the recognition accuracy of the neighboring relationship of each object and the relative arrangement relationship of each object with respect to the mobile body 20 .

[0046] In the state scene graph SG1, the neighbor relationship of each object is defined as an edge. The neighbor relationship of an object indicates in which direction (for example, front, back, left, and right) other adjacent objects exist based on one object.

[0047] The feature quantity of the primary node is defined based on the relative configuration relationship between the target and the mobile body 20 and the spatial occupancy of the target. The relative configuration relationship between the target and the mobile body 20 is defined by the center or center of gravity of the target (or tag), the relative distance between the mobile body 20 (or the camera 22) and the target, and the azimuth of the direction in which the target exists based on the direction corresponding to the moving direction or posture of the mobile body 20.

[0048] When an environment image including information capable of identifying a primary node and its feature value (for example, a ranging image having the distance from the imaging device 22 as a pixel value) is obtained, a three-dimensional high-definition image may not be used.

[0049] The spatial occupation mode of the target is defined, for example, by an occupation flag (0... non-occupied, 1... occupied), which indicates whether a static target (building, columnar structure, tree, etc.) occupies the area in a manner that does not allow the mobile body 20 to pass (whether it is a target with a height above the ground of a certain level). In addition, the spatial occupation mode of the target is defined by an interference flag (0... not present, 1... present), which indicates whether a dynamic target (vehicle, pedestrian, etc.) as a designated target exists in the area in a manner that may interfere with the mobile body 20.

[0050] For example, when the target corresponding to the primary node is a "road grid" and there are other vehicles in the road grid, although the mobile body 20 can pass through the area corresponding to the target, it may interfere with the other vehicles, so the occupancy flag is "0", but the interference flag is defined as "1". For a road grid where stopping is not allowed due to road signs (e.g., pedestrian crossing, no parking), when the specified state of the mobile body 20 corresponds to the stopped state, "1" is defined or assigned as the occupancy flag. The feature quantity of the primary node can also be defined by "label area" and "label ID".

[0051] like Figure 5 As schematically shown in FIG. 1 , in the state scene graph SG1, there are multiple first-order nodes n having feature values ​​c1(x) 1(x) (x represents each target or its label) are associated through edges. Figure 5 The illustrated scene graph SG1 includes an object o representing the state of a specified location (eg, a specified store or a building including the specified store). 01 , o 02 and 03 , an object indicating the state of the first surrounding space (e.g., the space on the south side of the building) based on the designated location 11 , o 12 and 13 , an object indicating the state of the first surrounding space (e.g., the space on the east side of the building) based on the designated location 21 , o 22 , o 23 and 24 , an object o representing the state of a region candidate (e.g., a road grid) a1 , o a2 and a3 , and an object o representing the state of a specified object (e.g., a traffic participant)b1 , o b2 , o b3 and b4 .

[0052] Next, the state scene graph SG1 is convolved and pooled by the first scene graph creation element 110, thereby creating a layout scene graph SG2 ( Figure 2 / Step 112). Thus, for example, as Figure 5 The state scene graph SG1 schematically shown in FIG is convolved to produce Figure 6 The layout scene graph SG2 is schematically shown in FIG. The granularity of the layout scene graph SG2 is lower than the granularity of the state scene graph SG1 before convolution.

[0053] Through Figure 6 The layout scene graph SG2 shown in the figure defines each secondary node n 2(o0) 、n 2(o1) 、n 2(o2) 、n 2(oa) and n 2(ob) , showing the primary node clusters corresponding to the "specified place", "first surrounding space" and "second surrounding space", "region candidates in multiple surrounding spaces" and "specified target" respectively. For example, the primary node cluster corresponding to the specified place is represented by Figure 5 The primary node n of the state of the specified place (e.g., a specified store or a building including the specified store) in the state scene graph SG1 shown in FIG. 1(o01) 、n 1(o02) and n 1(o03) By Figure 6 The layout scene graph SG2 shown in the figure defines the edges, showing the nodes n 2(o0) 、n 2(o1) 、n 2(o2) 、n 2(oa) and n 2(ob) The neighbor relationship of the target cluster corresponding to the primary node cluster is represented. For example, the secondary node n corresponding to the "specified location" 2(o0) and n which is equivalent to the "second peripheral space" 2(o0) The edge between shows that the second peripheral space is located on the east side of the designated place. 2(o0) 、n 2(o1) 、n 2(o2) 、n 2(oa) and n 2(ob) Each of them has a feature quantity specified according to the feature quantity of the primary node cluster that is the convolution object (as a result of aggregating the feature quantity of the primary node cluster).

[0054] Furthermore, the layout scene graph SG2 is convolved and pooled by the first scene graph production element 110, thereby producing an indication scene graph SG3 ( Figure 2 / Step 113). Thus, for example, as Figure 6 The result of convolution of the layout scene graph SG2 schematically shown in FIG. Figure 7 The indicated scene graph SG3 is schematically shown in FIG. The granularity of the indicated scene graph SG3 is lower than the granularity of the layout scene graph SG2 before convolution.

[0055] Through Figure 7 The third nodes n defined in the indicated scene graph SG3 are shown 3(w0) 、n 3(w1) and n 3(w2) , showing secondary node clusters corresponding to words related to "specified place", "specified space" and "specified state" respectively included in the user's instruction. For example, the secondary node cluster corresponding to the specified space is composed of Figure 6 The secondary node n representing the states of the first surrounding space and the second surrounding space in the layout scene graph SG2 shown in FIG. 2(o1) 、n 2(o2) , and the secondary nodes associated with them through edges. Figure 7 The edges defined in the indicated scene graph SG3 show the adjacency relationship of words. 3(w0) 、n 3(w1) and n 3(w2) Each of them has a feature quantity defined according to the feature quantity of the secondary node cluster that is the convolution target.

[0056] exist Figure 8 , conceptually shows the steps of generating a state scene graph SG1 (1st scene graph) by convolving and pooling the initial scene graph SG0, generating a layout scene graph SG2 (2nd scene graph) by convolving and pooling the state scene graph SG1, and generating an indication scene graph SG3 (3rd scene graph) by convolving and pooling the layout scene graph SG2. For example, as a convolution method, a general "Aggregate", "Update" or "Readout" is used, and as a pooling method, "average pooling" is used.

[0057] exist Figure 8 Each of the scene graphs SG0, SG1, SG2, and SG3 shown includes a building X0 as a destination or a designated place facing a three-way intersection (or T-road), and a parking space X0 (as a road grid) in the three-way intersection. 21 , X 22 and X 24 .like Figure 8 As shown, parking space X 22 Exists in front of building X0 (below the figure +), parking space X 24 Exists on the side of building X0 ( Figure 8 Left side), parking space X 21 There is a road that does not face the building X0. In this scenario, in the parking space X 21 There are obstacles.

[0058] exist Figure 8 The initial scene graph SG0 shown includes a plurality of initial nodes n arranged along a roadway for vehicles approaching the fork road from the left. 0(k) The building X0 as the destination is regarded as a node. The location information after discretizing the path information recorded in the three-dimensional map (high-resolution map) in an unequal interval manner is regarded as a node. The grid of a specified size defined with the node as the center has the attributes of occupied / unoccupied / prohibited parking. Regarding the attributes of the grid, places such as pedestrian crossings, intersections and / or prohibited roads are treated as prohibited parking.

[0059] exist Figure 8 In the state scene graph SG1 shown, except for the first node n corresponding to building X0 0(1) In addition, it also includes a plurality of initial nodes n corresponding to the road grid. 0(k) The result of convolution and pooling and the multiple initial nodes n 0(k) In contrast, the number of primary nodes n that are sparsely arranged k(1) . In multiple 1-node n 1(k) In the three roads, the parking space X 21 , X 22 and X 24 The corresponding primary node n 1(1) 、n 1(2) and n 1(4) .

[0060] exist Figure 8 In the layout scene graph SG2 shown, except for the secondary node n corresponding to building X0 0(2) In addition, it also includes a plurality of primary nodes n corresponding to the road grid. 1(k) The result of convolution and pooling and the parking space X in the three-way intersection 21 , X 22 and X 24 The corresponding secondary nodes n 2(1) 、n 2(2) and n 2(4) That is, the secondary node n 2(1) 、n2(2) and n 2(4) The parking spaces X of the three roads forming the three-way intersection are 21 , X 22 and X 24 and multiple 1st nodes n in its vicinity 1(k) The result of convolution and pooling.

[0061] exist Figure 8 In the indicated scene graph SG3 shown, except for the third node n corresponding to the building X0 3(0) In addition, it also includes parking space X 21 , X 22 and X 24 Parking space with obstacles in X 21 Equivalent secondary node n 2(1) The same 3rd node n 3(1) , and as a parking space X with no obstacles 22 and X 24 The corresponding secondary nodes n 2(2) and n 2(4) The 3rd node n of the convolution and pooling result 3(2) .

[0062] Next, through the learning completion model generation element 120, the state scene graph SG1, the layout scene graph SG2, and the instruction scene graph SG3 are input as input data to the graph neural network GNN together with the area where the specified state of the mobile body 20 is realized, thereby generating or constructing the learning completion model ( Figure 2 / Step 120). For example, Fig. 9 As shown, the graph neural network GNN is composed of an input layer NL0, an intermediate layer NL1, and an output layer NL2. The model is constructed by adjusting the values ​​of parameters such as weight coefficients of each node constituting the graph neural network GNN in such a way that a region candidate output from the graph neural network GNN is consistent with the correct solution region indicated by the input data (input data).

[0063] exist Fig.10 The steps of generating a state scene graph SG1 (first-order scene graph) by convolving and pooling the initial scene graph SG0, generating a layout scene graph SG2 (second-order scene graph) by convolving and pooling the state scene graph SG1, and generating an indication scene graph SG3 (third-order scene graph) by convolving and pooling the layout scene graph SG2 are conceptually shown. Fig.10 In the figure, “GCN” represents convolution processing based on graph convolutional neural network, and “Pool” represents pooling processing.

[0064] exist Fig.11 In the figure, the correct solution data for different driving scenarios of the vehicle are illustrated. Fig.11 As shown in (1) of FIG. 1 , a driving scenario in which a vehicle approaches a building X0 facing the road from the left side of the figure along a road extending to the left and right is described. In this driving scenario, for example, in response to instructions such as “stop in front of the building X0”, “stop to the side of the building X0”, and “stop next to the building X0”, the vehicle is stopped in the parking space X0 in front of the building X0 (below the figure) in the drivable lane of the road. 2i , X 2i and X 2i+1 Any one of them is defined as the correct solution.

[0065] like Fig.11 As shown in (2) of FIG. 1 , a driving scenario is described in which a vehicle approaches a building X0 facing the road from the right side of the figure along a road extending to the left and right. In this driving scenario, for the same instruction, the vehicle will be driven on the drivable lane of the road (with Fig.11 (1) The vehicle is parked in the parking space X in front of the building X0. 2j-1 , X 2j and X 2j+1 Any one of them is defined as the correct solution.

[0066] like Fig.11 As shown in (3) of FIG. 1 , a driving scenario in which a vehicle approaches a building X0 facing a three-way intersection from the left side of the figure is described. In this driving scenario, for example, in response to instructions such as “stop in front of building X0”, “stop to the side of building X0”, and “stop next to building X0”, the vehicle is stopped in a parking space X0 in front of building X0 (below the figure) in the drivable lane of the three-way intersection. 2i+1 , the parking space X on the side of the building X0 (left side of the figure) 2i and a parking space X slightly away from the building X0 2i Defined as the correct solution.

[0067] like Fig.11 As shown in (4) of FIG. 1 , a driving scenario in which a vehicle approaches a building X0 facing a three-way intersection from the upper side of the figure is described. In this driving scenario, for example, in response to instructions such as “park in front of the building X0”, “park to the side of the building X0”, and “park next to the building X0”, the vehicle is parked in a parking space X0 on the side (left side of the figure) of the building X0 in the drivable lane of the three-way intersection. 2j , the parking space X in front of the building X0 (below the figure) 2j+1 and a parking space X slightly away from the building X0 2j-1 Defined as the correct solution.

[0068] like Fig.11 As shown in (5) of FIG. 1 , a driving scenario in which a vehicle approaches a building X0 facing an intersection from the left side of the figure is described. In this driving scenario, for example, in response to instructions such as “stop in front of building X0”, “stop to the side of building X0”, and “stop next to building X0”, the vehicle is stopped in a parking space X0 in front of (below in the figure) the building X0 in the drivable lane of the intersection. 2i+1 , the parking space X on the side of the building X0 (left side in the figure) 2i and a parking space X slightly away from the building X0 2i or X 2i+2 Defined as the correct solution.

[0069] like Fig.11 As shown in (6) of FIG. 1 , a driving scenario in which a vehicle approaches a building X0 facing an intersection from the upper side of the figure is described. In this driving scenario, for example, in response to instructions such as “stop in front of building X0”, “stop to the side of building X0”, and “stop next to building X0”, the vehicle is parked in a parking space X0 to the side (left side of the figure) of building X0 in the drivable lane of the intersection. 2j , the parking space X in front of the building X0 (below the figure) 2j+1 and a parking space X slightly away from the building X0 2j or X 2j+2 Defined as the correct solution.

[0070] exist Fig.12 In the example, Fig.11 As shown in (3) of , the correct solution data for the driving scenario where the vehicle approaches the building X0 facing the three-way intersection from the left side of the figure. Fig.12 As shown in (1) to (3), the vehicle is parked in the parking space X. 2i , X 2i and X 2i+1 There is no obstacle X in 50 Any of the two parking spaces is defined as the correct solution. Fig.12 As shown in (4) to (6) respectively, the vehicle is parked in the parking space X 2i , X 2i and X 2i+1 There is no obstacle X in 51 and X 52 A parking space is defined as the correct solution. Fig.12 As shown in (7), the vehicle will be parked in a parking space X where there is no obstacle. 2i , X 2i and X2i+1 Any one of the following is defined as the correct solution. Fig.12 As shown in (8), each obstacle X 50 , X 51 and X 52 Parking space X 2i , X 2i and X 2i+1 The situation where there are no parked vehicles is defined as the correct solution.

[0071] In each of the nodes N30 , N20 , and N10 constituting the input layer NL0 , the feature quantities of the primary, secondary, and tertiary nodes constituting each of the three scene graphs SG1 to SG3 are vectorized.

[0072] In the middle layer NL1, the weight coefficient propagates from bottom to top between nodes (node ​​N110→N210→N310, node N112→N212→N312, node N114→N214→N314), and then the weight coefficient propagates from top to bottom between nodes (node ​​N310→N211→N112, node N312→N213→N114). In the middle layer NL1, the weight coefficient propagates in the order of nodes N210, N212, and N214, skipping the middle nodes N211 and N213.

[0073] The output layer NL2 includes three nodes N32, N22, and N12 that output the first judgment results corresponding to the three scene graphs SG1 to SG3, and a node N40 that outputs a region candidate as a second judgment result by integrating the first result. A graph attention network (GAN) can be used as a graph neural network GNN. In this case, for example, by introducing attention, an importance score (weight coefficient) is added to the relationship between the three nodes N32, N22, and N12, and flexible changes in the output results are achieved.

[0074] (Region candidate output function)

[0075] After the learned model is generated or constructed as described above, a region candidate is outputted according to the user's instruction. Specifically, the user issues an instruction to the mobile body 20 (which may be a mobile body different from the mobile body 20 used when the learned model is generated, or a mobile body the same as the mobile body 20) via the input interface of the device owned by the user, and the instruction is sent from the device to the learning apparatus 100, and the first scene graph creation element 110 recognizes the instruction (( Fig.13 The environment image may be stored in the database 102 or directly sent from the device to the mobile assistance device 200.

[0076] The following environment image is obtained by the imaging device 22 mounted on the mobile body 20 (see Figure 3 ), the environment image shows the state of the designated place and its surroundings obtained according to the position of the mobile body 20 and the direction facing the designated place (the shooting direction of the shooting device 22) ( Fig.13 / Step 202). The environment image may be stored in the database 102 or directly sent from the mobile body 20 to the mobile body assistance device 200.

[0077] The state scene graph SG1 is generated by the second scene graph generation element 210 based on the position of the mobile body 20 (at the time of obtaining the environment image), the environment image, and the three-dimensional high-definition map (see Figure 5 )( Fig.13 / Step 211). Next, the state scene graph SG1 is convolved by the second scene graph creation element 210, thereby creating a layout scene graph SG2 (see Figure 6 )( Fig.13 / Step 212). Further, the layout scene graph SG2 is convolved by the second scene graph creation element 210, thereby creating an indication scene graph SG3 (refer to Figure 7 )( Fig.13 / Step 213).

[0078] Next, through the region candidate output element 220, the state scene graph SG1, the layout scene graph SG2, and the instruction scene graph SG3 are input into the graph neural network GNN (refer to Figure 8 )The learned model generated by Fig.13 / 220). Then, a region candidate is output as the output of the learned model ( Fig.13 Based on the output result of the learned model, the movement of the moving body 20 is controlled by the moving body control device 21 in such a way that the moving body 20 is in a specified state in a region candidate as the output result. The output result of the learned model can also be output to the output interface of the device.

[0079] (Effect)

[0080] According to the learning device 100 that performs the above functions, the user's instructions and the scene graphs SG1 to SG3 generated based on the environment image corresponding to the position of the mobile body 20 and the direction facing the designated place are used as input data to construct a learned model (see Figure 2 ).

[0081] The feature quantity of the primary node constituting the state scene graph SG1 is defined based on the relative configuration relationship (distance and angle) with each target based on the position of the moving body 20. Therefore, the relative configuration relationship with each target based on the position of the moving body 20 is also reflected in the feature quantity of the secondary node constituting the layout scene graph SG2 as a result of convolution of the state scene graph SG1. Furthermore, the relative configuration relationship with each target based on the position of the moving body 20 is also reflected in the feature quantity of the tertiary node representing the words included in the instruction constituting the instruction scene graph SG3 as a result of convolution of the layout scene graph SG2.

[0082] As a result, even if the user's arbitrary instruction is a vague spatial designation such as "right", "near", or "left", the probability of outputting a region (e.g., a road grid) existing in the space intended by the user as a region candidate can be increased (see Fig.13 ).

[0083] In addition, the feature quantity of the primary node constituting the state scene graph SG1 is defined based on the spatial occupation mode of each target, specifically, based on the occupation flag that mainly indicates the spatial occupation state of the static target and the interference flag that mainly indicates the spatial occupation state of the dynamic target. The same is true for the feature quantity of the secondary node constituting the layout scene graph SG2 and the feature quantity of the tertiary node constituting the indication scene graph SG3.

[0084] As a result, the moving body 20 can output an appropriate region candidate for realizing a designated state from the learned model through the moving body support device 200 while avoiding interference with static objects and dynamic objects.

[0085] For example, according to the user instruction "Please stop on the right side of X0 (specified place)", Figure 4 The road grid X shown 21 ~X 26 Except for the road grid X that is equivalent to the pedestrian crossing 22 A road grid other than X 21 or X 24 The learned model outputs the region candidate for realizing the stopped state (specified state) of the moving body 20. In addition, according to the user instruction "Please decelerate before X0", the region candidate can be outputted from the learned model. Figure 4 The road grid X shown 21 ~X 26 A road grid X in 21 or X 23The learned model outputs a region candidate for realizing the deceleration start state (specified state) of the moving body 20. In addition, according to the user instruction "please pass on the left side of X0", the learned model can output Figure 4 The road grid X shown 21 To X 26 A road grid X in 22 As one area candidate for realizing the traffic state (specified state) of the moving body 20.

[0086] (Other embodiments of the present invention)

[0087] According to the above-mentioned embodiment, an environmental image is obtained by a shooting device 22 mounted on a mobile body 20, but based on the measurement results of the position and travel direction of the mobile body 20 in a global coordinate system or a map coordinate system, a three-dimensional high-precision map or a two-dimensional map (map information) can be used to obtain a virtual image obtained by a virtual shooting device mounted on the mobile body 20 as an environmental image.

[0088] Description of Reference Numerals

[0089] 20: Mobile

[0090] 22: Shooting device

[0091] 100: Learning device

[0092] 102: Database

[0093] 110: Elements of the First Scene Graph Production

[0094] 120: Learning the model generation elements

[0095] 200: Mobile body assist device

[0096] 210: Second scene graph production elements

[0097] 220: Region candidate output element.

Claims

1. A learning device, wherein: A learned model is generated in the following manner: an indication of an object related to the realization of a specified state in a specified space surrounding a specified place, position information of the object, multiple scene graphs created based on images of the surroundings of the specified place obtained based on the positional relationship between the object and the specified place, and a result of whether the specified state of the object can be realized are used as learning data, and one area candidate among multiple area candidates existing in multiple surrounding spaces based on the specified place is output.

2. The learning device according to claim 1, wherein: The plurality of scene graphs include: A state scene graph, which is produced based on the position of the object, the image and the map information, and is defined by primary nodes representing each of a plurality of targets included in the image, edges representing adjacent relationships between the plurality of targets, and feature quantities of the primary nodes, wherein the feature quantities of the primary nodes correspond to a relative configuration relationship with the target based on the object and a spatial occupation state of the target; as well as A layout scene graph is produced by convolving the state scene graph, and is defined by secondary nodes composed of one or more of the primary nodes and representing each primary node cluster, edges representing adjacent relationships of target clusters composed of one or more of the targets equivalent to the primary node cluster, and feature quantities of the secondary nodes specified according to feature quantities of the primary node cluster, wherein each of the primary node clusters corresponds to the designated place, multiple surrounding spaces based on the designated place, area candidates in the multiple surrounding spaces, and a designated target.

3. The learning device according to claim 2, wherein: The multiple scene graphs include an indication scene graph, which is produced by convolving the layout scene graph and is defined by a tertiary node composed of one or more of the secondary nodes and representing a secondary node cluster corresponding to each word, edges representing adjacent relationships of the words, and feature quantities of the tertiary nodes specified according to feature quantities of the secondary node cluster, wherein each word is related to the designated place, the designated space, and the designated state contained in the indication.

4. The learning device according to claim 1, wherein: The learned model is generated using a graph neural network, which is defined in a way that weights are propagated from top to bottom and from bottom to top among nodes constituting an intermediate layer.

5. The learning device according to claim 4, wherein: The learned model is generated using the graph neural network defined in a manner that weights are propagated from nodes constituting one intermediate layer to nodes constituting other intermediate layers, wherein the other intermediate layers exist with one or more intermediate layers between them.

6. The moving body assistance system according to claim 1, wherein: The learned model is generated using, as the learning data, an area existing around the designated location and the plurality of scene graphs accompanied by results of whether or not the designated state of the object in the area can be achieved.

7. The learning device according to claim 1, wherein: The image is an image captured by a camera mounted on the object.

8. The learning device according to claim 1, wherein: The specified state of the object includes a stopped state of the object.

Images