Mobile body support apparatus and mobile body system

Abstract

To provide a mobile body that can search for a suitable area around a target location for the mobile body to achieve a specified state according to instructions, considering the intentions of a person giving the instructions, which intentions are implicitly contained within instructions that are ambiguous in specifying a space based on the target location.SOLUTION: A model is constructed using scene graphs SG1 to SG3 as input data, which are created based on user instructions and environmental images corresponding to the position of a mobile body 20 and the direction in which it faces a specified location. The feature quantities of primary nodes constituting the state scene graph SG1 are defined according to the relative arrangement relationship (distance and angle) with each object, based on the position of the mobile body 20. The feature quantities of the primary nodes constituting the state scene graph SG1 are also defined according to the spatial occupancy pattern of each object.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a mobile support device and a mobile system that is configured by the mobile support device and a mobile object having a mobility function. [Background technology]

[0002] A method for generating a scene graph from an image has been proposed (see, for example, Non-Patent Documents 1 and 2). According to this method, the steps of inputting an image, detecting objects from the image using a deep learning-based object detection method, detecting contextual situations in the image using PLSI, detecting relationships between objects using a deep learning-based relationship detection and ontology method, and generating a scene graph for the input image are performed. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions, CVPR2020(https: / / arxiv.org / pdf / 2004.03967v1.pdf) [Non-patent document 2] Multi-Layer Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Search, ICRA2020(https: / / arxiv.org / pdf / 2012.04060.pdf) Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, even if a user instructs a mobile object, such as a robot, to "park to the right of XX (e.g., the name of a store or facility)," it has been difficult for the user to stop the mobile object in the area corresponding to the user's intended "right of XX." This is because, although the coordinates of a point are required to stop a mobile object, the expression "right" included in the user's instruction does not uniquely express a point. In fact, users often do not recognize the expression "right" as a uniquely determined coordinate of a point, but rather refer to the "space" to the right. For this reason, it is necessary to link the word included in the user's instruction with the space. Furthermore, the space "right" includes spaces in which a mobile object can stop and spaces in which it cannot. For example, if the "right of XX" is an open space, the mobile object can stop, but if it is a crosswalk, the mobile object cannot stop.

[0005] Therefore, the present invention aims to provide a mobile system that can search for an appropriate area around a destination location for a mobile body to realize a specified state in accordance with an instruction, taking into account the intention of the person giving the instruction that is ambiguous in terms of spatial designation based on the destination location. [Means for solving the problem]

[0006] The mobility support device of the present invention comprises: An instruction to a mobile object regarding the realization of a specified state in a specified space around a specified location; location information of the moving object; By inputting multiple scene graphs created based on images of the surrounding area of ​​the specified location acquired based on the positional relationship between the moving body and the specified location into a trained model, one area candidate is output from among multiple area candidates existing in multiple surrounding spaces based on the specified location. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 2 is an explanatory diagram illustrating the configuration of a learning device and a mobile object support device. [Figure 2] An explanatory diagram of the trained model generation function. [Figure 3]FIG. 1 is an explanatory diagram of an image containing multiple objects. [Figure 4] FIG. 10 is an explanatory diagram showing the result of projecting a 3D high-resolution map onto a 2D map. [Figure 5] FIG. 10 is an illustrative diagram of a state scene graph. [Figure 6] FIG. 1 is an explanatory diagram illustrating an example of a layout scene graph. [Figure 7] FIG. 10 is an explanatory diagram illustrating an example of an instruction scene graph. [Figure 8] Conceptual illustration of sequential folding and pooling of a scene graph. [Figure 9] An explanatory diagram of a graph neural network. [Figure 10] Conceptual illustration of the sequential convolution and pooling of a scene graph input to a graph neural network. [Figure 11] 10A to 10C are explanatory diagrams relating to correct answer data in different driving scenes. [Figure 12] 10A to 10C are explanatory diagrams relating to correct answer data in driving scenes in which obstacles exist in different ways. [Figure 13] FIG. 2 is an explanatory diagram of an area candidate output function of the mobile support system. DETAILED DESCRIPTION OF THE INVENTION

[0008] (composition) 1, the learning device 100 and the mobile object support device 200 according to an embodiment of the present invention are configured as devices that can access the database 102 via a network to support the realization of a specified state of the mobile object 20. The mobile object 20 and the mobile object support device 200 constitute a "mobile object system."

[0009] The database 102 stores and holds environmental images (corresponding to "images" in the present invention) showing the surroundings of the mobile object 20, 3D high-definition maps (map information), graph neural network graphs, trained models, etc. In this embodiment, the database 102 is configured as a device or database server separate from the learning device 100 and the mobile object support device 200, but may also be a component of the learning device 100 and / or the mobile object support device 200.

[0010] The learning device 100 includes a first scene graph creation element 110 and a trained model generation element 120. Each of the first scene graph creation element 110 and the trained model generation element 120 is configured to perform a designated task, such as scene graph creation and trained model generation, which will be described later. A functional element being configured to perform a designated task means that the hardware constituting the functional element performs the designated task by reading software and, if necessary, data from a storage element and performing computations on the data or other data in accordance with the software.

[0011] The mobile support device 200 includes a second scene graph creation element 210 and an area candidate output element 220. Each of the second scene graph creation element 210 and the area candidate output element 220 is configured with a processing element such as a CPU and / or a processor core, a storage element such as a ROM and / or a RAM, an input / output interface circuit, etc. Each of the second scene graph creation element 210 and the area candidate output element 220 is configured to perform a specified task such as scene graph creation and trained model generation, which will be described later.

[0012] The learning device 100 and the mobile support device 200 may be configured as the same device. In this case, the first scene graph creation element 110 and the second scene graph creation element 210 may be configured as a single scene graph creation element.

[0013] The mobile object 20 is configured as a vehicle or robot having an autonomous movement function, a positioning function, and a wireless communication function. The mobile object 20 is equipped with a mobile object control device 21 and an imaging device 22. The mobile object 20 may be configured as an information processing terminal (e.g., a smartphone) that is carried by a user and moves passively along with the user's movement. The mobility support device 200 may be configured as a device (e.g., the mobile object control device 21) mounted on the mobile object 20.

[0014] The mobile object control device 21 is composed of processing elements such as a CPU and / or a processor core, storage elements such as a ROM and / or RAM, and an input / output interface circuit. The mobile object control device 21 is configured to control the autonomous movement function, positioning function, and wireless communication function of the mobile object 20. The imaging device 22 is mounted on the mobile object 20 so as to capture an image of the state in the traveling direction or ahead of the mobile object 20. The mobile object 20 may have a function to adjust the imaging direction (optical axis direction) of the imaging device 22 and / or a function to measure the imaging direction.

[0015] (Trained model generation function) The trained model generation function generates a trained model based on instructions (corresponding to "training instructions") regarding the specified state of the moving body 20 (corresponding to "training moving body") in the specified space surrounding the specified location, and an environmental image (corresponding to "training environmental image") representing the specified location and the state of its surroundings, obtained according to the position of the moving body 20 and the direction facing the specified location.

[0016] Specifically, a user's instruction to the moving object 20 via the input interface of a device owned by the user is transmitted from the device to the learning device 100 and recognized by the first scene graph creation element 110 (FIG. 2 / STEP 100). The environment image may be stored in the database 102 or may be transmitted directly from the device to the learning device 100.

[0017] An "instruction" is an instruction regarding a specified state of the moving object 20 in a specified space around a specified location. Thus, for example, an instruction "stop to the right of X" is recognized as an instruction regarding the realization of a stopped state as a specified state of the moving object 20 in the right-hand space as a specified space around the specified location represented by the word X. Similarly, an instruction "slow down just before Y" is recognized as an instruction regarding the realization of a state in which the moving object 20 starts to decelerate as a specified state in the forward space as a specified space around the specified location represented by the word Y. Furthermore, an instruction "pass to the left of Z" is recognized as an instruction regarding the realization of a passing state as a specified state of the moving object 20 in the left-hand space as a specified space around the specified location represented by the word Z.

[0018] The user issuing the instruction may be a user on board the moving body 20, or a user in a location different from the moving body 20. The user's instruction may be a voice instruction or a gesture instruction.

[0019] An image capturing device 22 mounted on the mobile object 20 captures an environmental image representing the specified location and its surroundings, captured according to the position of the mobile object 20 and the direction of the specified location (the image capturing direction of the image capturing device 22) (FIG. 2 / STEP 102). The environmental image may be stored in a database 102 or may be directly transmitted from the mobile object 20 to the learning device 100.

[0020] For example, as shown in FIG. 3, a building X0 (a building) and a sidewalk grid X1 extending along the lower edges of two sides of the building X0 are 11 , X 12 , sidewalk grid X from building X0 11 , X 12 Roadway grid X extending outside of 21 ~X 26 , and sidewalk grid X 12 and roadway grid X 24 Tree X standing on the border 41 , X 42 An environmental image containing the building X0 is acquired. 01 and window X 02 On the other side there is a window X03. The environmental image illustrated in FIG. 3 includes a vehicle X5 and a pedestrian X as traffic participants. 61 ~X 64 It further includes:

[0021] Based on the position of the moving object 20 (at the time when the environmental image is acquired), the environmental image, and map information, the first scene graph creation element 110 creates a state scene graph SG1 (FIG. 2 / STEP 111).

[0022] Map information is, for example, a high-resolution 3D map, and includes static information such as 3D structures, road surface information, and lane information, where the types and / or attributes of objects or things are defined so that they are distinguished by labels. For example, objects that are above a certain height from the ground and objects that extend along the terrain are each distinguished by a label. A label is defined by its label area (the area that the labeled object occupies in the environmental image) and label ID.

[0023] The first-ranked object, "an object that is at a certain height above the ground," is classified into second-ranked objects such as buildings, columnar structures, and trees. The second-ranked object, "buildings," is classified into third-ranked objects such as side walls, store signs, windows, and entrances and exits for people or vehicles. The second-ranked object, "columnar structures," is classified into third-ranked objects such as traffic signal poles, traffic sign poles, and communication poles. From the third-ranked object onwards, the objects may be further classified.

[0024] The first-level object, "objects extending along the terrain," is classified into second-level objects such as roadways and sidewalks. The second-level object, "roadway," is divided into multiple roadway grids, which are third-level objects, with each roadway grid defined as an individual object. The third-level object, "roadway grid," is classified into fourth-level objects such as road markings, including crosswalks, center lines, lane boundaries, and zebra zones. The second-level object, "sidewalk," is divided into multiple sidewalk grids, which are each defined as an individual object. The third-level object, "sidewalk grid," is classified into fourth-level objects such as road markings, including tactile paving blocks. From the fourth level onward, objects may be further classified.

[0025] Each object reflected in the environmental image is assigned a label defined in the 3D high-resolution map. Labels are also assigned to objects that correspond to dynamic information, such as vehicles on the roadway, pedestrians on the sidewalk or roadway (crosswalk). In the state scene graph SG1, each object to which a label is assigned (or its label) is defined as a first-order node.

[0026] FIG. 4 shows the result of projecting static objects (buildings, sidewalk grids, and roadway grids) of a 3D high-definition map as a 2D map. The 2D map shown in FIG. 4 includes, among the objects included in the environmental image shown in FIG. 3, a building X0 (a static object), sidewalk grids X1 (a sidewalk grid extending along the lower edge of two sides of the building X0), and 11 , X 12 , and roadway grid X 21 ~X 26 By using the two-dimensional map, it is possible to improve the accuracy of recognizing the adjacency relationship between each object and the relative positional relationship between each object and the moving body 20 as a reference.

[0027] In the state scene graph SG1, the adjacency relationship of each object is defined as an edge. The adjacency relationship of an object indicates in which direction (for example, front, back, left, right) other objects exist adjacent to a given object.

[0028] The feature quantity of the primary node is defined according to the relative positional relationship between the object and the moving body 20 and the spatial occupation mode of the object. The relative positional relationship between the object and the moving body 20 is defined by the center or center of gravity of the object (or label), the relative distance between the moving body 20 (or the imaging device 22) and the object, and the azimuth angle of the direction in which the object exists based on the direction of travel or the orientation according to the posture of the moving body 20.

[0029] If an environmental image containing information that can identify the primary node and its feature amount (for example, a distance measurement image having the distance from the imaging device 22 as a pixel value) is obtained, the 3D high-resolution map does not need to be used.

[0030] The spatial occupation mode of an object is defined by an occupation flag (0: not occupied, 1: occupied) that indicates whether or not a static object (such as a building, columnar structure, or tree) occupies an area in a manner that does not allow passage of the mobile object 20 (whether or not the object is at a certain height above the ground). Furthermore, the spatial occupation mode of an object is defined by an interference flag (0: absent, 1: present) that indicates whether or not a dynamic object (such as a vehicle or pedestrian) as a designated object is present in the area in a manner that may interfere with the mobile object 20.

[0031] For example, if the object corresponding to the primary node is a "road grid" and other vehicles or the like are present in the road grid, the mobile object 20 can pass through the area corresponding to the object, but there is a possibility of interference with the other vehicles or the like, so the occupancy flag is defined as "0" but the interference flag is defined as "1." However, for roadway grids where stopping is not permitted in consideration of road markings (e.g., pedestrian crossings, no parking or stopping), the occupancy flag is defined or assigned as "1" when the designated state of the mobile object 20 corresponds to a stopped state. The feature amount of the primary node may be further defined by a "label area" and a "label ID."

[0032] As shown in FIG. 5, in the state scene graph SG1, there are multiple first-order nodes n 1(x) (x represents each object or its label) are related by edges. The scene graph SG1 shown in FIG. 5 has an object o that represents the state of a specified location (e.g., a specified store or the building in which it is located). 01 , o 02 and o 03 , an object o representing the state of the first surrounding space (e.g., the space on the south side of the building) based on the specified location 11 , o 12 and o 13 , an object o representing the state of the first surrounding space (e.g., the space on the east side of the building) based on the specified location 21 , o 22 , o23 and o 24 , an object o representing the state of an area candidate (e.g., road grid) a1 , o a2 and o a3 , and an object o representing the state of a specified object (e.g., a traffic participant). b1 , o b2 , o b3 and o b4 Contains:

[0033] Next, the state scene graph SG1 is folded and pooled by the first scene graph creation component 110 to create a layout scene graph SG2 (FIG. 2 / STEP 112). As a result, for example, the layout scene graph SG2 shown in FIG. 6 is created as a result of folding the state scene graph SG1 shown in FIG. 5. The granularity of the layout scene graph SG2 is lower than the granularity of the state scene graph SG1 before folding.

[0034] The second-order node n defines the layout scene graph SG2 shown in Figure 6. 2(o0) , n 2(o1) , n 2(o2) , n 2(oa) and n 2(ob) Each of these represents a primary node cluster corresponding to the "specified location," the "first surrounding space," the "second surrounding space," the "area candidate in multiple surrounding spaces," and the "specified object." For example, the primary node cluster corresponding to the specified location is a primary node n that represents the state of the specified location (e.g., the specified store or the building in which it is located) in the state scene graph SG1 shown in FIG. 1(o01) , n 1(o02) and n 1(o03) The edges that define the layout scene graph SG2 shown in Figure 6 define the second-order nodes n 2(o0) , n 2(o1) , n 2(o2) , n 2(oa) and n 2(ob)For example, the second-order node n corresponding to the "specified location" is 2(o0) and n, which corresponds to the "second surrounding space" 2(o2) The edge between indicates that the second surrounding space is to the east of the specified location. 2(o0) , n 2(o1) , n 2(o2) , n 2(oa) and n 2(ob) Each of these has a feature value (as a result of aggregating the features of the primary node cluster) that is determined according to the features of the primary node cluster that is the subject of the convolution.

[0035] Furthermore, the layout scene graph SG2 is folded and pooled by the first scene graph creation component 110 to create a directed scene graph SG3 (FIG. 2 / STEP 113). As a result, for example, the layout scene graph SG2 schematically shown in FIG. 6 is folded to create the directed scene graph SG3 schematically shown in FIG. 7. The granularity of the directed scene graph SG3 is lower than the granularity of the layout scene graph SG2 before folding.

[0036] The tertiary node n defines the directed scene graph SG3 shown in Fig. 7. 3(w0) , n 3(w1) and n 3(w2) Each of these represents a second-order node cluster corresponding to the words related to "designated location," "designated space," and "designated state" contained in the user's instruction. For example, the second-order node cluster corresponding to the designated space is the second-order node n 2(o1) and n 2(o2) The scene graph SG3 shown in Figure 7 is composed of the nodes n and n, and the nodes n are connected to the nodes n by edges. The edges defining the scene graph SG3 shown in Figure 7 represent the word adjacency relationships. 3(w0) , n 3(w1) and n 3(w2)Each of these has a feature that is determined according to the feature of the second-order node cluster that is the subject of convolution.

[0037] 8 conceptually shows a procedure in which a state scene graph SG1 (primary scene graph) is generated by folding and pooling an initial scene graph SG0, a layout scene graph SG2 (secondary scene graph) is generated by folding and pooling the state scene graph SG1, and an instruction scene graph SG3 (tertiary scene graph) is generated by folding and pooling the layout scene graph SG2. For example, the general-purpose "Aggregate," "Update," or "Readout" is adopted as the folding method, and "average pooling" is adopted as the pooling method.

[0038] Each of the scene graphs SG0, SG1, SG2, and SG3 shown in Fig. 8 includes a building X0 as a destination or designated location facing a three-way intersection (or T-junction), and a parking space X1 at the three-way intersection (as a road grid). 21 , X 22 and X 24 As shown in Figure 8, parking space X 22 is located in front of the building X0 (downward of the figure +) and is located in the parking space X 24 is located next to the building X0 (to the left in Figure 8), and parking space X 21 is located on a road that does not face the building X0. In this scene, parking space X 21 There is an obstacle in the

[0039] The initial scene graph SG0 shown in Figure 8 has several initial nodes n arranged along the lanes that a vehicle approaching the three-way intersection from the left can travel. 0(k)The goal building, X0, is considered a node. The location information, which is irregularly discretized route information described on a 3D map (high-resolution map), is considered a node. A grid of a given size defined around a node has the attributes of occupied, unoccupied, and no parking. Regarding the grid attributes, parking is considered no-parking in crosswalks, intersections, and / or places where road parking is prohibited.

[0040] The state scene graph SG1 shown in Fig. 8 has a first-order node n corresponding to the building X0. 0(1) In addition, multiple initial nodes n 0(k) The multiple initial nodes n 0(k) Multiple primary nodes n that are more sparsely spaced than k(1) Contains multiple primary nodes n 1(k) Parking Space X at the three-way intersection 21 , X 22 and X 24 The primary node n corresponding to each of 1(1) , n 1(2) and n 1(4) Contains:

[0041] The layout scene graph SG2 shown in Figure 8 has a second-order node n 0(2) In addition, multiple primary nodes n corresponding to the road grid 1(k) Parking space X at a three-way intersection as a result of convolution and pooling 21 , X 22 and X 24 The secondary node n corresponding to each of 2(1) , n 2(2) and n 2(4) That is, the secondary node n 2(1) , n 2(2) and n 2(4) Each of the three roads that make up the three-way intersection is a parking space X 21 , X 22 and X 24 Each of these nodes and its neighboring first-order nodes n1(k) is the result of convolution and pooling.

[0042] The directed scene graph SG3 shown in Fig. 8 has a third-order node n 3(0) In addition, Parking Space X 21 , X 22 and X 24 Parking spaces with obstacles X 21 The secondary node n corresponds to 2(1) The same tertiary node n 3(1) , and Parking Space X, where there are no obstacles 22 and X 24 The secondary nodes n corresponding to each of 2(2) and n 2(4) The third node n is the result of convolution and pooling. 3(2) Contains:

[0043] Next, the trained model generation element 120 inputs the state scene graph SG1, layout scene graph SG2, and instruction scene graph SG3, together with the area in which the specified state of the moving object 20 is realized, as input data to the graph neural network GNN, thereby generating or constructing a trained model (FIG. 2 / STEP 120). For example, as shown in FIG. 9, the graph neural network GNN is composed of an input layer NL0, a hidden layer NL1, and an output layer NL2. A model is constructed by adjusting the values ​​of parameters such as weight coefficients of each node constituting the graph neural network GNN so that one area candidate output from the graph neural network GNN matches the correct area indicated by the input data (input data).

[0044] 10 conceptually shows a procedure in which an initial scene graph SG0 is folded and pooled to generate a state scene graph SG1 (primary scene graph), the state scene graph SG1 is folded and pooled to generate a layout scene graph SG2 (secondary scene graph), and the layout scene graph SG2 is folded and pooled to generate an instruction scene graph SG3 (tertiary scene graph). In FIG. 10, "GCN" represents convolution processing by a graph convolutional neural network, and "Pool" represents pooling processing.

[0045] FIG. 11 illustrates correct answer data for each of different vehicle driving scenes. As shown in FIG. 11(1), a driving scene will be described in which a vehicle approaches a building X0 facing the road from the left side of the figure along a road extending from left to right. In this driving scene, for example, in response to instructions such as "park in front of building X0," "park next to building X0," and "park near building X0," the vehicle will select parking space X0 in front of building X0 (toward the bottom of the figure) in the driving lane of the road. 2i-1 , X 2i and X 2i+1 The correct answer is defined as parking the vehicle in one of the following locations.

[0046] As shown in Figure 11(2), a driving scene will be described in which a vehicle is approaching a building X0 facing the road from the right side of the figure along a road extending from left to right. In this driving scene, in response to a similar instruction, the vehicle moves to the parking space X0 in front of the building X0 in the drivable lane of the road (the lane opposite to Figure 11(1)). 2j-1 , X 2j and X 2j+1 The correct answer is defined as parking the vehicle in one of the following locations.

[0047] As shown in Figure 11(3), a driving scene will be described in which a vehicle is approaching a building X0 facing a three-way intersection from the left side of the figure. In this driving scene, for example, in response to instructions such as "park in front of building X0," "park next to building X0," and "park near building X0," the vehicle will park in the parking space X0 in front of building X0 (downward in the figure) in the travelable lane of the three-way intersection. 2i+1 , Parking space X next to building X0 (to the left of the figure) 2i and parking space X, a short distance from structure X0 2i-1 The correct answer is defined as parking the vehicle at each of the following locations.

[0048] As shown in Figure 11(4), a driving scene will be described in which a vehicle is approaching a building X0 facing a three-way intersection from the top of the figure. In this driving scene, in response to instructions such as "park in front of building X0," "park next to building X0," and "park close to building X0," the vehicle will park in the parking space X0 next to building X0 (to the left in the figure) in the travelable lane of the three-way intersection. 2j Parking space X in front of building X0 (downward in the figure) 2j+1 and parking space X, a short distance from structure X0 2j-1 The correct answer is defined as parking the vehicle at each of the following locations.

[0049] As shown in Figure 11(5), a driving scene will be described in which a vehicle is approaching a building X0 facing a crossroads from the left side of the figure. In this driving scene, for example, in response to instructions such as "park in front of building X0," "park next to building X0," and "park near building X0," the vehicle will park in the parking space X in front of building X0 (toward the bottom of the figure) in the driving lane of the crossroads. 2i+1 , Parking space X next to building X0 (to the left of the figure) 2i and parking space X, a short distance from structure X0 2i-1 or X 2i+2 The correct answer is defined as parking the vehicle at each of the following locations.

[0050] As shown in Figure 11(6), a driving scene will be described in which a vehicle is approaching a building X0 facing a crossroads from the top of the figure. In this driving scene, for example, in response to instructions such as "park in front of building X0," "park next to building X0," and "park near building X0," the vehicle will park in the parking space X0 next to building X0 (to the left of the figure) in the driving lane of the crossroads. 2j Parking space X in front of building X0 (downward in the figure) 2j+1 and parking space X, a short distance from structure X0 2j-1 or X 2j+2 The correct answer is defined as parking the vehicle at each of the following locations.

[0051] 12 shows an example of correct data for a driving scene in which a vehicle is approaching a building X0 facing a three-way intersection from the left side of the figure, as shown in FIG. 11(3). As shown in each of FIGS. 12(1) to (3), parking space X 2i-1 , X 2i and X 2i+1 Among them, obstacle X 50 The correct answer is to park the vehicle in one of the two parking spaces where there is no parking space X. As shown in each of Figures 12(4) to (6), 2i-1 , X 2i and X 2i+1 Among them, obstacle X 51 and X 52 The correct answer is to park the vehicle in one parking space that does not contain any of the obstacles. As shown in Figure 12(7), the correct answer is to park the vehicle in the parking space X 2i-1 , X 2i and X 2i+1 The correct answer is to park the vehicle at one of the obstacles X. 50 , X 51 and X 52 Parking Space X where each of these exist 2i-1 , X 2i and X 2i+1The correct answer is defined as not parking the vehicle in any of the above locations.

[0052] At each of the nodes N30, N20, and N10 that make up the input layer NL0, the feature amounts of the first, second, and third order nodes that make up each of the three scene graphs SG1 to SG3 are vectorized.

[0053] In the hidden layer NL1, weight coefficients are propagated from bottom to top between nodes (nodes N110 → N210 → N310, nodes N112 → N212 → N312, nodes N114 → N214 → N314), and then weight coefficients are propagated from top to bottom between nodes (nodes N310 → N211 → N112, nodes N312 → N213 → N114). In the hidden layer NL1, weight coefficients are propagated in the order of nodes N210, N212, and N214, skipping intermediate nodes N211 and N213.

[0054] The output layer NL2 includes three nodes N32, N22, and N12 that output primary determination results corresponding to the three scene graphs SG1 to SG3, respectively, and a node N40 that integrates the primary results to output one area candidate as a secondary determination result. A graph attention network (GAN) may be employed as the graph neural network GNN. In this case, for example, by introducing attention, importance scores (weighting coefficients) are assigned to the relationships between the three nodes N32, N22, and N12, allowing for flexible changes in the output results.

[0055] (Area candidate output function) After the trained model is generated or constructed as described above, one area candidate is output in response to a user instruction. Specifically, a user instruction for a moving object 20 (which may be a moving object different from the moving object 20 used to generate the trained model or may be the same moving object 20) is transmitted from the device to the learning device 100 via an input interface of the device, and is recognized by the first scene graph creation element 110 (FIG. 13 / STEP 200). The environmental image may be stored in the database 102 or may be transmitted directly from the device to the mobile object support device 200.

[0056] The imaging device 22 mounted on the moving object 20 acquires an environmental image (see FIG. 3) representing the specified location and its surroundings, acquired according to the position of the moving object 20 and the direction of the specified location (the imaging direction of the imaging device 22) (FIG. 13 / STEP 202). The environmental image may be stored in the database 102, or may be directly transmitted from the moving object 20 to the mobile object support device 200.

[0057] Based on the position of the moving object 20 (at the time the environmental image is acquired), the environmental image, and the 3D high-resolution map, the second scene graph creation element 210 creates a state scene graph SG1 (see FIG. 5) (FIG. 13 / STEP 211). Next, the second scene graph creation element 210 convolves the state scene graph SG1 to create a layout scene graph SG2 (see FIG. 6) (FIG. 13 / STEP 212). Furthermore, the second scene graph creation element 210 convolves the layout scene graph SG2 to create an instruction scene graph SG3 (see FIG. 7) (FIG. 13 / STEP 213).

[0058] Next, the area candidate output element 220 inputs the state scene graph SG1, layout scene graph SG2, and instruction scene graph SG3 into a trained model generated based on the graph neural network GNN (see FIG. 8) (FIG. 13 / STEP 220). Then, one area candidate is output as the output of the trained model (FIG. 13 / STEP 230). Based on the output result of the trained model, the mobile object control device 21 controls the operation of the mobile object 20 so that the specified state of the mobile object 20 in the one area candidate as the output result is realized. The output result of the trained model may be output to an output interface constituting the device.

[0059] (effect) According to the learning device 100 that performs the above functions, scene graphs SG1 to SG3 created based on user instructions and environmental images corresponding to the position of the moving body 20 and the direction facing the specified location are used as input data to construct a learned model (see Figure 2).

[0060] The feature amounts of the first-order nodes constituting the state scene graph SG1 are defined according to the relative positional relationships (distances and angles) between each object and the position of the moving object 20. Therefore, the feature amounts of the second-order nodes constituting the layout scene graph SG2, which is the result of convolving the state scene graph SG1, also reflect the relative positional relationships between each object and the position of the moving object 20. Furthermore, the feature amounts of the third-order nodes representing words included in instructions, which constitute the instruction scene graph SG3, which is the result of convolving the layout scene graph SG2, also reflect the relative positional relationships between each object and the position of the moving object 20.

[0061] As a result, even if the user's arbitrary instruction is an ambiguous spatial designation such as "right," "forward," or "left," the probability that an area (e.g., a roadway grid) existing in the space intended by the user will be output as a candidate area is improved (see Figure 13).

[0062] Furthermore, the feature quantities of the first-order nodes constituting the state scene graph SG1 are defined according to the spatial occupation mode of each object, specifically, the occupation flags that mainly indicate the spatial occupation state of static objects and the interference flags that mainly indicate the spatial occupation state of dynamic objects. This is also true for the feature quantities of the second-order nodes constituting the layout scene graph SG2 and the feature quantities of the third-order nodes constituting the instruction scene graph SG3.

[0063] As a result, the mobile object support device 200 can output from the trained model one suitable area candidate for realizing the specified state while avoiding interference with static and dynamic objects.

[0064] For example, in response to a user instruction to "stop to the right of X0 (specified location)", the system will move to the roadway grid X shown in Figure 4. 21 ~X 26 Among them, roadway grid X corresponds to the crosswalk 22 Any one of the roadway grids except X 21 or X 24 can be output from the trained model as one area candidate for realizing the stopped state (designated state) of the moving object 20. In addition, in response to a user instruction to "please slow down before X0", 21 ~X 26 Any one of the roadway grids X 21 or X 23 can be output from the trained model as one area candidate for realizing the deceleration start state (designated state) of the moving object 20. Furthermore, in response to the user's instruction to "pass to the left of X0", 21 ~X 26 Any one of the roadway grids X 22 can be output from the trained model as one area candidate for realizing the traffic state (specified state) of the moving object 20.

[0065] (Another embodiment of the present invention) According to the above embodiment, an environmental image is acquired through an imaging device 22 mounted on the moving body 20, but a three-dimensional high-resolution map or a two-dimensional map (map information) may be used based on the measurement results of the position and direction of travel of the moving body 20 in a global coordinate system or a map coordinate system, and a virtual image acquired through a virtual imaging device mounted on the moving body 20 may be acquired as the environmental image. [Explanation of symbols]

[0066] 20. Mobile 22. Imaging device 100. Learning device 102. Database 110...First scene graph creation element 120... Trained model generation element 200. Mobility support device 210...Second scene graph creation element 220...Area candidate output element.

Claims

[Claim 1] An instruction to a mobile object regarding the realization of a specified state in a specified space around a specified location; location information of the moving object; and a plurality of scene graphs created based on images of the surroundings of the designated location acquired based on the positional relationship between the moving body and the designated location, and by inputting the scene graphs into a trained model, one area candidate is output from among a plurality of area candidates existing in a plurality of surrounding spaces based on the designated location. Mobility support devices.

Images