Controller, mobile body, route planning device, program, and method for generating learned model
The control device uses image-based relative indexing and weighted graph navigation to enhance autonomous mobile object navigation, addressing GPS and map construction challenges by leveraging image similarity for efficient route planning.
Patent Information
- Application Number
- JP2023082369
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-18
- Publication Date
- 2024-11-28
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing systems for controlling the autonomous movement of mobile objects face challenges in accurately determining position and orientation without relying on GPS signals or constructing detailed environmental maps, especially in areas with limited visibility or sensor errors, and struggle with managing routes efficiently.
A control device that uses a trained estimation model to derive a relative index between reference and current images, enabling route planning through a weighted graph based on image similarity, allowing autonomous mobile bodies to navigate within a geographical area by comparing feature amounts extracted from images captured by onboard cameras.
Enables accurate and efficient navigation of autonomous mobile objects by determining their position and orientation relative to planned routes using image-based similarity, reducing reliance on GPS and map construction efforts, and improving route planning in challenging environments.
Smart Images

Figure 2024165836000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a control device, a moving body, a path planning device, a program, and a method for generating a trained model. [Background technology]
[0002] Patent Document 1 discloses a method for determining the current position and orientation of a robot using a database of images and positions. Patent Document 2 discloses a method for estimating the self-position of a moving object using a database of images and positions. Patent Document 3 discloses an autonomous mobile system that does not include an environmental map and a self-position estimation means. (Prior art document) (Patent Documents) (Patent Document 1) Patent No. 6632173 (Patent Document 2) JP 2016-162013 A (Patent Document 3) JP 2022-187529 A Summary of the Invention [Means for solving the problem]
[0003] In a first aspect of the present invention, a control device is provided. The control device controls, for example, a moving body that moves autonomously within a predetermined geographical range. In the control device, the moving body includes, for example, a first imaging device that captures an image of the surroundings of the moving body. The control device is configured to be able to access a first storage device that stores, for example, a plurality of reference images that are images captured in advance at each of a plurality of points disposed within the geographical range in association with identification information of the plurality of points.
[0004] The control device includes, for example, a route planning unit that determines a planned route from a departure point of the moving body to a destination point. The control device includes, for example, a route information acquisition unit that acquires route information indicating one or more relay points that are one or more points constituting the planned route among a plurality of points. The control device includes, for example, a destination point determination unit that determines a destination point that is a destination of the moving body among the one or more relay points. The control device includes, for example, a destination image acquisition unit that acquires a destination image that is a reference image associated with the destination point among a plurality of reference images by referring to a first storage device. The control device includes, for example, a current image acquisition unit that acquires a current image that is an image captured by a first imaging device at a current position and attitude of the moving body. The control device includes, for example, a relative index derivation unit that derives a value of a relative index that is an index indicating a relative relationship between the current position and attitude of the moving body and the position and attitude of the moving body that is a destination at the destination point based on the target image and the current image.
[0005] In any of the above control devices, the relative index derivation unit, for example, (i) inputs the target image and the current image to a trained estimation model, and (ii) derives an output corresponding to an estimation result related to the relative index from the estimation model. In any of the above control devices, the route planning unit, for example, refers to a second storage device that stores a weighted graph including a plurality of nodes corresponding to a plurality of points and one or more edges connecting each of the plurality of nodes, and determines a planned route by deriving a node sequence that minimizes the sum of edge weights from a start node, which is a node corresponding to a starting point or a point located in the vicinity of the starting point, to an end node, which is a node corresponding to a destination or a point located in the vicinity of the destination. In the weighted graph related to the above control device, the weight of each of the one or more edges indicates, for example, the degree of similarity between the reference images of two points corresponding to each of the two nodes connected by each edge.
[0006] In any of the above control devices, the estimation model may have a feature extraction unit that receives a target image and a current image as inputs and extracts features of the target image and features of the current image.In any of the above control devices, the estimation model may have a relative index estimation unit that receives a feature of the target image and a feature of the current image as inputs and outputs an estimation result regarding the relative index.
[0007] In any of the above control devices, the feature extraction unit may include a first extraction unit that inputs a target image and extracts a feature of the target image. In any of the above control devices, the feature extraction unit may include a second extraction unit that inputs a current image and extracts a feature of the current image. In any of the above control devices, the first extraction unit may (i) input the target image to a trained first extraction model, and (ii) derive an output corresponding to an estimation result regarding the feature of the target image from the first extraction model. In any of the above control devices, the second extraction unit may (i) input the current image to a trained second extraction model, and (ii) derive an output corresponding to an estimation result regarding the feature of the current image from the second extraction model. In any of the above control devices, the first extraction model and the second extraction model may be trained models generated by distance learning in which an image is input and a feature vector of the image is output.
[0008] Any of the above control devices may include a first data set acquisition unit that acquires a first learning data set including a plurality of first learning data. Any of the above control devices may include an estimation model generation unit that generates a trained estimation model by executing machine learning using a combination of two pieces of first learning data included in the first learning data set. In any of the above control devices, each of the first learning data may be data in which an image is associated with an imaging position that is the position of a second imaging device that captured the image, and an imaging attitude that is the attitude of the second imaging device. In any of the above control devices, the estimation model generation unit may generate an estimation model by machine learning for each of a plurality of combinations, using as input a first image included in one of the two pieces of first learning data and a second image included in the other of the two pieces of first learning data, and outputting a value related to an estimation result of a relative relationship between the imaging position and imaging attitude corresponding to the first image and the imaging position and imaging attitude corresponding to the second image.
[0009] Any of the above control devices may include a movement control unit that moves the moving body based on the relative index derived by the relative index derivation unit. Any of the above control devices may include a correction unit that estimates a current position of the moving body by a particle filter or a sequential Monte Carlo method and corrects the value of the relative index based on the estimated current position. In any of the above control devices, the movement control unit may move the moving body based on the value of the relative index corrected by the correction unit.
[0010] Any of the above control devices may include a departure image acquisition unit that acquires a departure image that is an image captured by a first imaging device at a departure point. Any of the above control devices may include a similarity derivation unit that derives a degree of similarity between each of a plurality of reference images and the departure image. Any of the above control devices may include a start node determination unit that determines a node corresponding to a reference image that is most similar to the departure image as a start node. Any of the above control devices may include an end node determination unit that determines an end node based on an input from a user.
[0011] In a second aspect of the present invention, a moving body capable of moving autonomously is provided. The moving body includes, for example, any one of the control devices according to the first aspect. The moving body includes, for example, a driving unit that moves the moving body according to an instruction from the control device.
[0012] In a third aspect of the present invention, a route planning device is provided. The route planning device determines, for example, a planned route from a starting point to a destination point of a moving body that autonomously moves within a predetermined geographical range. The route planning device is configured to be accessible to a storage device that stores a weighted graph including, for example, a plurality of nodes corresponding to a plurality of points disposed within the geographical range, and one or more edges connecting each of the plurality of nodes. In the weighted graph related to the route planning device, the weight of each of the one or more edges indicates, for example, the degree of similarity between reference images that are images captured in advance at each of two points corresponding to each of two nodes connected by each edge.
[0013] The above-mentioned route planning device includes, for example, a route planning unit that determines a planned route. In the above-mentioned route planning device, the route planning unit determines the planned route by, for example, referring to the above-mentioned storage device, deriving a node sequence that minimizes the sum of edge weights from a start node, which is a node corresponding to a starting point or a point located near the starting point, to an end node, which is a node corresponding to a destination point or a point located near the destination point.
[0014] In a fourth aspect of the present invention, a method for generating a trained model is provided. The method includes, for example, acquiring a training dataset including a plurality of training data. The method includes, for example, performing distance learning using the training dataset to generate a feature output model for inputting an image and outputting a value related to an estimation result of a feature of the image. In the method, each of the plurality of training data is, for example, data in which a reference image, one or more similar images, and one or more dissimilar images are associated with each other. In the method, the relationship between the geographical position where each of the one or more similar images was captured and the geographical position where the reference image was captured meets, for example, a predetermined condition. In the method, the relationship between the geographical point where each of the one or more dissimilar images was captured and the geographical position where the reference image was captured does not meet, for example, the predetermined condition.
[0015] In any of the above methods, the predetermined condition may include a condition that the geographical distance between the locations at which each image was captured is less than or equal to a predetermined first threshold, and / or a condition that the length of the period between the times at which each image was captured is less than or equal to a predetermined second threshold.
[0016] In any of the above methods, the feature output model may include a neural network. In any of the above methods, the step of generating the feature output model by performing distance learning may include a step of generating feature vectors from each of the reference image, one or more similar images, and one or more dissimilar images included in each learning data. In any of the above methods, the step of generating the feature output model may include a step of deriving a distance between feature vectors for each of a set of the reference image and one or more similar images included in each learning data, and a set of the reference image and one or more dissimilar images included in each learning data. In any of the above methods, the step of generating the feature output model may include a step of performing distance learning of a neural network using a loss derived based on the distance and a predetermined margin value.
[0017] In any of the above methods, the step of acquiring a training dataset may include a step of generating a training dataset. In any of the above methods, the step of generating a training dataset may include a step of acquiring image data of a plurality of images captured within a predetermined geographical range in association with auxiliary data indicating the geographical position at which each of the plurality of images was captured or the time at which each of the plurality of images was captured. In any of the above methods, the step of generating a training dataset may include a step of determining, for each of the plurality of images, whether or not a relationship between the geographical position at which each image was captured and the geographical position at which at least one other image included in the plurality of images was captured meets a predetermined condition based on the geographical position or time indicated by the auxiliary data associated with the image data of each image. In any of the above methods, the step of generating a training dataset may include a step of determining, as one or more similar images corresponding to each image, other images determined to have a relationship between the geographical position at which each image was captured and the geographical position at which each image was captured meets a predetermined condition. In any of the above methods, the step of generating the training dataset may include a step of determining other images that are determined to not meet a predetermined condition in relation to the geographic location at which each image was taken as one or more dissimilar images corresponding to each image.
[0018] In a fifth aspect of the present invention, a program is provided. A computer-readable medium storing the above program may be provided. The computer-readable medium may be a non-transitory computer-readable medium or a computer-readable recording medium. The above program may be a program for causing a computer to function as a control device according to the first aspect. The above program may be a program for causing a computer to function as a path planning device according to the third aspect. The above program may be a program for causing a computer to execute a method for generating a trained model according to the fourth aspect.
[0019] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]
[0020] [Figure 1] 1 shows an example of a system configuration of a mobile object management system 100. [Diagram 2] 1 illustrates an example of information processing in the mobility management system 100. [Diagram 3] 3 illustrates a schematic diagram of an example of a transformation model 300. [Figure 4] 4 shows a schematic diagram of an example of a controlled variable estimation model 400. [Diagram 5] 10 illustrates an example of a process for generating various models. [Figure 6] 2 shows an example of the internal configuration of the autonomous moving body 110. [Figure 7] 6 shows an example of the internal configuration of a route guidance unit 650. [Figure 8] 13 shows an example of the internal configuration of the global path planning unit 720. [Figure 9] 13 shows an example of an internal configuration of the control amount determining unit 754. [Figure 10] 2 shows an example of the internal configuration of the management server 120. [Figure 11] 13 shows an example of a conversion model teacher data generating unit 1042. [Figure 12] FIG. 1 shows an example of a conceptual diagram of a topological map. [Figure 13] 1 illustrates an example of a reroute process. [Figure 14] An example of the system configuration of a computer 3000 is shown in schematic form. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0021] The present invention will be described below through the embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. In the drawings, the same reference numbers are used for the same or similar parts, and duplicate explanations may be omitted.
[0022] (Overview of the mobile management system 100) 1 illustrates an example of a system configuration of a mobile object management system 100. In this embodiment, the mobile object management system 100 includes an autonomous mobile object 110 and a management server 120. In this embodiment, the autonomous mobile object 110 includes a camera 112 and a control unit 114.
[0023] In this embodiment, the autonomous mobile body 110 and the management server 120 are configured to be able to transmit and receive information via a communication network (not shown). The communication network may be a transmission path for wired communication, a transmission path for wireless communication, or a combination thereof. The communication network may be the Internet, a dedicated line, a wireless communication network, or a combination thereof. The communication network may include a P2P network such as an ad-hoc network.
[0024] The communication method in the wireless communication network may be a mobile communication method or a wireless data communication method. Examples of the mobile communication method include the 3G method, the LTE method, the 4G method, and the 5G method. Examples of the wireless data communication method include the wireless PAN method, the wireless LAN method, the wireless MAN method, and the wireless WAN method. The communication method in the wireless communication network may be various short-distance wireless communication methods.
[0025] 1, xw and yw indicate a world coordinate system. Also, xm and ym indicate an orthogonal coordinate system with the center of gravity Mo of the autonomous mobile body 110 as the origin. In this embodiment, for the purpose of simplifying the explanation, the details of the mobile body management system 100 will be explained by taking as an example a case where the orthogonal coordinate system with the center of gravity of the autonomous mobile body 110 as the origin coincides with the orthogonal coordinate system with the center of gravity of the camera 112 as the origin.
[0026] In this embodiment, in order to facilitate understanding of the mobile object management system 100, the details of the mobile object management system 100 are described using as an example a case in which the autonomous mobile object 110 and the management server 120 cooperate to control the autonomous movement of the autonomous mobile object 110. Similarly, in this embodiment, in order to facilitate understanding of the mobile object management system 100, the details of the mobile object management system 100 are described using as an example a case in which the autonomous mobile object 110 travels two-dimensionally within the management area 10. However, it should be noted that the mobile object management system 100 and the autonomous mobile object 110 are not limited to this embodiment.
[0027] The mobile object management system 100 according to this embodiment controls the autonomous traveling of the autonomous moving object 110 by using a plurality of images (referred to as reference images 22) captured in advance at a plurality of points (referred to as reference points 20) located within the management area 10. Specifically, the mobile object management system 100 controls the autonomous traveling of the autonomous moving object 110 based on a similarity, which is an index showing the degree of similarity between two images. Examples of the similarity include cosine similarity, Euclidean distance, and Manhattan distance. Note that the smaller the Euclidean distance and Manhattan distance, the greater the similarity.
[0028] For example, the mobile body management system 100 estimates or recognizes the relative position and attitude (sometimes referred to as position and attitude) of the autonomous mobile body 110 with respect to a point (sometimes referred to as a target point) that is located on the global route (sometimes referred to as a planned route) of the autonomous mobile body 110 and serves as a local target point for the autonomous mobile body 110, based on the similarity between a reference image 22 (sometimes referred to as a target image) of the point, and an image (sometimes referred to as a current image) captured by a camera 112 mounted on the autonomous mobile body 110 during the period when the autonomous mobile body 110 is moving along the planned route.
[0029] The mobile object management system 100 determines the control amount of the autonomous mobile object 110 based on the above-mentioned relative position and attitude. The above-mentioned control amount is, for example, the amount of movement and / or the amount of rotation to the destination point. In this way, image-based route guidance is realized.
[0030] As another system for controlling the autonomous movement of the autonomous moving body 110, for example, a system that controls the autonomous traveling of the autonomous moving body 110 based on the position coordinates of the autonomous moving body 110 can be used. However, when the above system is used, it is difficult to accurately control the movement of the autonomous moving body 110 in, for example, an area where a GPS signal cannot be received or an area where an error of an internal sensor becomes large. In addition, the above system has problems such as (i) it takes a huge amount of effort to construct an accurate map over a wide area and update the map, (ii) it is necessary to manage route information independently of the map, and (iii) there are restrictions and / or a high degree of dependency on the sensor configuration of the autonomous moving body 110.
[0031] As another system for controlling the autonomous movement of the autonomous moving body 110, for example, a system that uses a plurality of images given in advance as a movement route of the autonomous moving body 110 and controls the autonomous traveling of the autonomous moving body 110 based on feature points of the images can be used. Visual Simultaneous Localization and Mapping (SLAM) is known as such a system. However, when the above system is used, it is difficult to precisely control the movement of the autonomous moving body 110 in, for example, a section with relatively little visual change or a section with relatively low illuminance. Also, in the above system, it takes a huge amount of time and effort to construct and manage a map and / or a route.
[0032] In contrast, according to the mobile object management system 100 of this embodiment, for example, it is possible to estimate or recognize the relative position and orientation of the autonomous mobile object 110 with respect to the planned route of the autonomous mobile object 110 without using the position coordinates of the autonomous mobile object 110. Furthermore, according to the mobile object management system 100 of this embodiment, even if it is difficult to extract feature points from at least one of the images, the similarity of the images can be derived. In addition, according to one embodiment of the mobile object management system 100 of this embodiment, if an image of the destination point of the autonomous mobile object 110 is specified according to some procedure, a global route can be automatically planned.
[0033] (Overview of each part related to the mobile management system 100) In this embodiment, the managed area 10 has, for example, a predetermined geographical range. Within the managed area 10, a travel-permitted area 12 may be set, which is an area in which the autonomous moving body 110 is permitted to travel. Within the managed area 10, a travel-prohibited area 14 may be set, which is an area in which the autonomous moving body 110 is prohibited from traveling.
[0034] In this embodiment, the autonomous moving body 110 only needs to have an autonomous traveling function, and the type and structure of the autonomous moving body 110 are not particularly limited. For example, various vehicles that travel using wheels or caterpillar tracks may be used as the autonomous moving body 110. The number of wheels or caterpillar tracks is not particularly limited. Details of the autonomous moving body 110 will be described later.
[0035] In this embodiment, the camera 112 captures an image of the surroundings of the autonomous moving body 110. The camera 112 may be a spherical camera and may include a plurality of image capturing elements. The camera 112 may include a visible light camera or an infrared camera.
[0036] In this embodiment, the control unit 114 controls various operations of the autonomous moving body 110. For example, the control unit 114 controls the autonomous traveling of the autonomous moving body 110. Details of the control unit 114 will be described later.
[0037] In this embodiment, the management server 120 manages various types of information related to the autonomous moving body 110. The management server 120 may cooperate with the autonomous moving body 110 to manage the autonomous traveling of the autonomous moving body 110. Details of the management server 120 will be described later.
[0038] (Specific configuration of each part of the mobile object management system 100) Each unit of the mobile object management system 100 may be realized by hardware, software, or both hardware and software. At least a part of each unit of the mobile object management system 100 may be realized by a personal computer or a mobile terminal. For example, a personal computer or a mobile terminal may be used as a user interface of the mobile object management system 100. Examples of the mobile terminal include a mobile phone, a smartphone, a PDA (registered trademark), a tablet, a notebook computer or laptop computer, and a wearable computer.
[0039] When at least some of the components constituting the mobile object management system 100 are realized by software, the components realized by the software may be realized by starting a program that defines the operation of the components in an information processing device of a general configuration. The information processing device includes, for example, (i) a data processing device having various processors (examples of the processor include a CPU, a GPU, etc.), a ROM, a RAM, a communication interface, etc., and (ii) a storage device (including an external storage device) such as a memory and a HDD. The information processing device may include (iii) input devices such as a keyboard, a touch panel, a camera, a microphone, various sensors, a GPS receiver, etc., and (iv) output devices such as a display device, a speaker, a vibration device, etc.
[0040] In the information processing device, the data processing device or storage device may store a program. For example, when the program is read into a computer, the information processing described in the program functions as a specific means in which software related to the program and various hardware resources constituting at least a part of the mobile object management system 100 work together. The specific means then realizes calculation or processing of information according to the intended use of the computer in this embodiment, thereby constructing the mobile object management system 100 or a part thereof according to the intended use.
[0041] The above program may be stored in a computer-readable medium. The above program may be stored in a non-transitory computer-readable recording medium. The above program may be stored in a computer-readable medium such as a CD-ROM, a DVD-ROM, a memory, or a hard disk, or may be stored in a storage device connected to a network. The above program may be installed in a computer constituting at least a part of the mobile management system 100 from the computer-readable medium or the storage device connected to the network.
[0042] By executing the above program, the computer may function as at least a part of each unit of the mobile object management system 100. By executing the above program, the computer may execute an information processing method in at least a part of the mobile object management system 100. The above computer may include a storage unit for storing data learned by machine learning, and an execution unit for executing the above program.
[0043] The program for causing a computer to function as at least a part of the mobile object management system 100 includes, for example, a module that defines at least a part of the operation of the mobile object management system 100. When the above program or module is executed, the above program or module acts on a data processing device, an input device, an output device, a storage device, etc. to cause the computer to function as at least a part of the mobile object management system 100 or to execute an information processing method in at least a part of the mobile object management system 100.
[0044] In one embodiment, the information processing method may be a control method for controlling a moving body. In the control method, the moving body moves autonomously within a predetermined geographical range, for example. The moving body includes, for example, a first imaging device that captures an image of the surroundings of the moving body. The subject of operation at each stage of the control method may be a computer. The computer is configured to be able to access, for example, a first storage device. The first storage device stores, for example, a plurality of reference images, which are images captured in advance at each of a plurality of points located within the geographical range, in association with identification information of the plurality of points.
[0045] The above control method includes, for example, a route planning step of determining a planned route from a departure point of the moving body to a destination point. The above control method includes, for example, a route information acquisition step of acquiring route information indicating one or more relay points, which are one or more points constituting the planned route, among a plurality of points. The above control method includes, for example, a destination point determination step of determining a destination point, which is a destination of the moving body, among the one or more relay points. The above control method includes, for example, a target image acquisition step of acquiring a target image, which is a reference image associated with the destination point, among a plurality of reference images, by referring to a first storage device. The above control method includes, for example, a current image acquisition step of acquiring a current image, which is an image captured by a first imaging device at the current position and attitude of the moving body. The above control method includes, for example, a relative index derivation step of deriving a value of a relative index, which is an index indicating a relative relationship between the current position and attitude of the moving body and the position and attitude of the moving body that is a destination at the destination point, based on the target image and the current image.
[0046] In the above control method, the relative index derivation step includes, for example, (i) inputting the target image and the current image into a trained estimation model, and (ii) deriving an output corresponding to an estimation result regarding the relative index from the estimation model. The route planning step includes, for example, a step of determining a planned route by deriving a node sequence that minimizes the sum of edge weights from a start node, which is a node corresponding to a starting point or a point located in the vicinity of the starting point, to an end node, which is a node corresponding to a destination point or a point located in the vicinity of the destination point, with reference to a second storage device that stores a weighted graph including a plurality of nodes corresponding to each of a plurality of points and one or more edges connecting each of the plurality of nodes. In the weighted graph, the weight of each of the one or more edges indicates, for example, the degree of similarity between the reference images of two points corresponding to each of the two nodes connected by each edge.
[0047] In another embodiment, the information processing method may be a route planning method for determining a planned route from a starting point of a moving object to a destination point. In the route planning method, the moving object moves autonomously within a predetermined geographical range, for example. The subject of operation of each step of the route planning method may be a computer. The computer is configured to be able to access a storage device, for example. The storage device stores a weighted graph, for example. The weighted graph includes, for example, a plurality of nodes corresponding to a plurality of points arranged within the geographical range, and one or more edges connecting the plurality of nodes. In the weighted graph, the weight of each of the one or more edges indicates, for example, the degree of similarity between reference images, which are images captured in advance at two points corresponding to two nodes connected by each edge.
[0048] The above-mentioned route planning method includes, for example, a route planning step of determining a planned route. In the above-mentioned route planning method, the route planning step includes, for example, a step of determining a planned route by referring to the above-mentioned storage device and deriving a node sequence that minimizes the sum of edge weights from a start node, which is a node corresponding to a starting point or a point located near the starting point, to an end node, which is a node corresponding to a destination point or a point located near the destination point.
[0049] In still another embodiment, the information processing method may be a method for generating a trained model. An operating subject of each step of the method may be a computer. The method includes, for example, a step of acquiring a training data set including a plurality of training data. The method includes, for example, a step of generating a feature output model for inputting an image and outputting a value related to an estimation result of a feature of the image by performing distance learning using the training data set. In the method, each of the plurality of training data is, for example, data in which a reference image, one or more similar images, and one or more dissimilar images are associated with each other. In the method, for example, a relationship between a geographical position where each of the one or more similar images is captured and a geographical position where the reference image is captured meets a predetermined condition. In the method, for example, a relationship between a geographical point where each of the one or more dissimilar images is captured and a geographical position where the reference image is captured does not meet a predetermined condition.
[0050] In yet another embodiment, the information processing method may be a method for generating a dataset used in machine learning or distance learning. An operating subject of each step of the method may be a computer. The method includes, for example, a step of acquiring image data of a plurality of images captured within a predetermined geographical range in association with auxiliary data indicating the geographical location at which each of the plurality of images was captured or the time at which each of the plurality of images was captured. The method includes, for example, a step of determining whether or not a relationship between the geographical location at which each of the plurality of images was captured and the geographical location at which at least one other image included in the plurality of images was captured meets a predetermined condition. The method includes, for example, a step of determining other images whose relationship with the geographical location at which each of the images was captured meets a predetermined condition as similar images that are positive examples corresponding to each of the images. The method includes, for example, a step of determining other images whose relationship with the geographical location at which each of the images was captured does not meet a predetermined condition as dissimilar images that are negative examples corresponding to each of the images. The method includes, for example, a step of generating a dataset by associating each of a plurality of images with each of the images, a positive example corresponding to each of the images, and a negative example corresponding to each of the images. In the method, the predetermined condition includes, for example, (i) a condition that a geographical distance between a geographical location where each of the images was captured and a geographical location where another image corresponding to each of the images was captured is equal to or smaller than a first predetermined threshold, and / or (ii) a condition that a length of a period between a time when each of the images was captured and a time when another image corresponding to each of the images was captured is equal to or smaller than a second predetermined threshold.
[0051] The management area 10 may be an example of a geographical range. The travel-permitted area 12 may be an example of a geographical range. The reference point 20 may be an example of a point located inside the geographical range. The reference image 22 may be an example of a reference image. The mobile object management system 100 may be an example of a control device or a route planning device. The autonomous mobile object 110 may be an example of a mobile object, a control device, or a route planning device. The camera 112 may be an example of an imaging device, a first imaging device, or a second imaging device. The control unit 114 may be an example of a control device or a route planning device. The control unit 114 may be an example of a storage device, a first storage device, or a second storage device. The management server 120 may be an example of a control device or a route planning device. The management server 120 may be an example of a storage device, a first storage device, or a second storage device.
[0052] (An example of another embodiment) In this embodiment, the details of the mobile object management system 100 have been described by taking as an example a case in which the autonomous moving object 110 and the management server 120 cooperate with each other to control the autonomous movement of the autonomous moving object 110. However, the mobile object management system 100 is not limited to this embodiment. In other embodiments, the autonomous moving object 110 may control the autonomous movement of the autonomous moving object 110 by itself. In this case, the autonomous moving object 110 may have at least a part of the functions and / or configurations of the management server 120. In still other embodiments, a part of the functions and / or configurations of the autonomous moving object 110 in this embodiment may be realized by the management server 120, and a part of the functions and / or configurations of the management server 120 in this embodiment may be realized by the autonomous moving object 110.
[0053] In the present embodiment, the details of the mobile object management system 100 have been described by taking as an example a case where the mobile object management system 100 controls the autonomous traveling of the autonomous mobile object 110. Also, it has been described that various vehicles that travel using wheels or caterpillars can be used as the autonomous mobile object 110. However, the control target of the mobile object management system 100 is not limited to the above-mentioned autonomous mobile object 110.
[0054] In another embodiment, the mobile object management system 100 may control the autonomous movement of any type of mobile object. In addition to vehicles, examples of the type of mobile object include ships and aircraft. Examples of ships include boats, hovercraft, personal watercraft, submarines, submersibles, and underwater scooters. Examples of aircraft include airplanes, airships or balloons, balloons, helicopters, and drones.
[0055] 2 shows an example of information processing in the mobile object management system 100. According to this embodiment, first, in step 212 (step may be abbreviated as S), a process of generating various models used in the mobile object management system 100 is executed. S212 is executed by the management server 120, for example.
[0056] According to one embodiment, in S212, a model (sometimes referred to as a conversion model) for extracting feature amounts of each image is generated. As described above, the mobile object management system 100 according to this embodiment executes a process for deriving the similarity between two images. At this time, the mobile object management system 100 extracts feature amounts of each image, and derives the similarity between the two images by comparing the two extracted feature amounts.
[0057] The conversion model (i) receives an image and (ii) outputs a value related to an estimation result of the feature of each image. Specifically, the conversion model (i) receives a two-dimensional RGB image and (ii) outputs a feature vector of the RGB image. The feature vector is, for example, a one-dimensional tensor. The conversion model is generated by, for example, distance learning using appropriate training data. The distance learning may be deep distance learning. This enables the conversion model to extract highly discriminative features.
[0058] According to another embodiment, in S212, a model (sometimes referred to as a control amount estimation model) for estimating a control amount for moving the autonomous moving body 110 to a target point is generated. As described above, the mobile body management system 100 according to this embodiment estimates a relative relationship between the current position and attitude of the autonomous moving body 110 and the position and attitude of the autonomous moving body 110 that is a target at a target point, and determines the control amount of the autonomous moving body 110 based on the estimation result.
[0059] The control amount estimation model (i) receives a target image and a current image, and (ii) outputs a value related to an estimation result of an index (sometimes referred to as a relative index) indicating a relative relationship between the current position and attitude of the autonomous moving body 110 and the position and attitude of the autonomous moving body 110 that is a target at a target point. The relative index may be the above-mentioned control amount. The control amount estimation model is generated, for example, by machine learning using appropriate training data. The control amount estimation model may include the above-mentioned conversion model.
[0060] Next, in S214, a topological map is created. The topological map is expressed, for example, as a weighted graph including a plurality of nodes corresponding to the plurality of reference points 20, and one or more edges connecting the plurality of nodes. In the weighted graph, the weight of each of the one or more edges indicates, for example, the similarity of the reference images 22 of the two reference points 20 corresponding to the two nodes connected by each edge. This makes it possible to express the relationship between the plurality of reference points 20 without using position coordinates. S214 is executed, for example, by the management server 120.
[0061] Next, in S222, a global route (as described above, may be referred to as a planned route) from the starting point to the destination point of the autonomous moving body 110 is planned. The global route is planned based on the topological map described above.
[0062] Specifically, the planned route is determined by deriving a sequence of nodes that minimizes the sum of edge weights from a node (sometimes referred to as a start node) corresponding to a reference point 20 located at or near the starting point to a node (sometimes referred to as an end node) corresponding to a reference point 20 located at or near the destination.
[0063] The start node may be a node corresponding to a reference point 20 located closest to the departure point. The end node may be a node corresponding to a reference point 20 located closest to the destination point. The node string includes one or more nodes. Each of the one or more nodes included in the node string corresponds to a reference point 20 (sometimes referred to as a relay point) on the planned route. This creates information (sometimes referred to as route information) indicating one or more relay points that constitute the planned route. S222 is executed by, for example, the autonomous mobile body 110.
[0064] Next, in S224, a navigation process is executed. Specifically, first, one or more relay points arranged on the planned route are determined as the target point. Next, a control amount of the autonomous mobile body 110 is determined based on the reference image 22 of the target point and the current image captured by the camera 112. Next, the position and attitude of the autonomous mobile body 110 are controlled based on the above control amount. Thereby, the autonomous mobile body 110 travels toward the target point. Thereafter, for example, when the autonomous mobile body 110 reaches the target point, the next target point is determined. The next target point may be determined at an appropriate timing during the period when the autonomous mobile body 110 is traveling toward the target point. The above process is repeated, thereby performing route guidance to the destination point. S224 is executed, for example, by the autonomous mobile body 110.
[0065] The conversion model may be an example of a feature output model. The controlled variable estimation model may be an example of an estimation model. The controlled variable may be an example of a relative index.
[0066] (An example of another embodiment) In this embodiment, the details of the information processing in the mobile object management system 100 will be described by taking as an example a case where S212 and S214 are executed by the management server 120, and S222 and S224 are executed by the autonomous mobile object 110. However, the information processing in the mobile object management system 100 is not limited to this embodiment. The autonomous mobile object 110 and the management server 120 may cooperate in any manner to execute the above information processing, or the above information processing may be executed by one of the autonomous mobile object 110 and the management server 120. For example, according to another embodiment, all steps are executed by the autonomous mobile object 110. In still another embodiment, S222 may be executed by the management server 120, and at least a part of S224 may be executed by the management server 120.
[0067] 3 shows an example of the conversion model 300. In this embodiment, the conversion model 300 (i) receives an image 32 and (ii) outputs a value related to an estimation result of a feature quantity 34 of the image 32. Specifically, the conversion model 300 (i) receives a two-dimensional RGB image and (ii) outputs a feature vector of the RGB image. The feature vector is, for example, a one-dimensional tensor. The conversion model 300 may include a neural network.
[0068] 4 is a schematic diagram of an example of the control amount estimation model 400. As described above, the control amount estimation model 400, for example, (i) receives the target image 42 and the current image 44, and (ii) outputs a value related to the estimation result 46 of an index indicating a relative relationship between the current position and attitude of the autonomous moving body 110 and the position and attitude of the autonomous moving body 110 that is a target at a target point.
[0069] In this embodiment, the controlled variable estimation model 400 includes, for example, a feature quantity extraction unit 420 and a relative index estimation unit 460. In this embodiment, the feature quantity extraction unit 420 includes, for example, a first extraction unit 430 and a second extraction unit 440. In this embodiment, the first extraction unit 430 includes a first extraction model 432. The second extraction unit 440 includes a second extraction model 442. In this embodiment, the relative index estimation unit 460 includes, for example, a fully connected layer 462, a fully connected layer 464, and a fully connected layer 466.
[0070] In this embodiment, the feature extraction unit 420 receives the target image 42 and the current image 44 as input, and extracts the feature of the target image 42 and the feature of the current image 44. The feature extraction unit 420 receives, for example, the target image 42 as input, and extracts the feature of the target image 42. The feature extraction unit 420 receives, for example, the current image 44 as input, and extracts the feature of the current image 44. The above feature may be a feature vector. The feature vector is, for example, a one-dimensional tensor.
[0071] In this embodiment, the first extraction unit 430 inputs the target image 42 and extracts the feature amount 52 of the target image 42. For example, the first extraction unit 430 (i) inputs the target image 42 to a trained first extraction model 432, and (ii) derives an output corresponding to an estimation result regarding the feature amount 52 of the target image 42 from the first extraction model 432.
[0072] In this embodiment, the first extraction model 432 is, for example, a trained model generated by distance learning in which an image is input and a feature vector of the image is output. The first extraction model 432 may have a similar configuration to the conversion model 300. For example, the first extraction model 432 includes a neural network.
[0073] In this embodiment, the second extraction unit 440 inputs the current image 44 and extracts feature quantities 54 of the current image 44. The second extraction unit 440, for example, (i) inputs the current image 44 to a trained second extraction model 442, and (ii) derives an output corresponding to an estimation result regarding the feature quantities 54 of the current image 44 from the second extraction model 442.
[0074] In this embodiment, the second extraction model 442 is, for example, a trained model generated by distance learning in which an image is input and a feature vector of the image is output. The second extraction model 442 may have a similar configuration to the conversion model 300. For example, the second extraction model 442 includes a neural network.
[0075] In this embodiment, the relative index estimation unit 460 receives the feature amount 52 of the target image 42 and the feature amount 54 of the current image 44 as input, and outputs an estimation result 46 related to the relative index. The relative index estimation unit 460 receives, for example, a vector obtained by concatenating or combining (sometimes called concatenating) the feature amount 52, which is the feature vector of the target image 42, and the feature amount 54, which is the feature vector of the current image 44, and outputs a value related to the estimation result 46 of the relative index described above.
[0076] The relative index estimation result 46 includes, for example, an estimation result of a relative position and an estimation result of a relative orientation. Examples of values related to the relative index estimation result 46 include a value indicating a difference in x coordinates, a value indicating a difference in y coordinates, a value of the z component qz of a quaternion, and a value of the w component qw of a quaternion.
[0077] In this embodiment, the fully connected layer 462 receives the output of the feature extraction unit 420 as an input, and outputs a vector having a dimension equal to or smaller than the dimension of the input. In this embodiment, the fully connected layer 464 receives the output of the fully connected layer 462 as an input, and outputs a vector having a dimension equal to or smaller than the dimension of the input. In this embodiment, the fully connected layer 466 receives the output of the fully connected layer 464 as an input, and outputs a value related to the relative index estimation result 46 described above.
[0078] 5 shows an example of a process for generating various models. According to this embodiment, first, in S510, imaging data of a plurality of reference points 20 is acquired. The imaging data may be data in which image data of a plurality of reference images 22 captured at each of the plurality of reference points 20 is associated with auxiliary data indicating the geographical position of each of the plurality of reference points 20 or the time when each of the plurality of reference images 22 was captured.
[0079] For example, first, the autonomous moving body 110 is manually or remotely operated to travel within the managed area 10 while the camera 112 is used to capture an image of the surroundings of the autonomous moving body 110. At this time, based on the output of various sensors mounted on the autonomous moving body 110, (i) the image captured by the camera 112 is associated with (ii) the geographical location where the image was captured (sometimes referred to as the imaging location) and / or the time when the image was captured (sometimes referred to as the imaging time).
[0080] As a result, the autonomous moving body 110 can acquire the reference images 22 of the multiple reference points 20. Furthermore, the autonomous moving body 110 can acquire imaging data of the multiple reference points 20. Thereafter, the autonomous moving body 110 transmits the imaging data to the management server 120. As a result, the management server 120 can acquire the imaging data of the multiple reference points 20.
[0081] Next, in S522, a teacher data set is generated for generating the conversion model 300. The procedure for generating the teacher data will be described in detail later. After that, in S524, deep metric learning of the conversion model 300 is executed.
[0082] In addition, in S532, a teacher data set is generated for generating the controlled variable estimation model 400. The procedure for generating the teacher data will be described later in detail. After that, in S534, machine learning of the controlled variable estimation model 400 is executed.
[0083] 6 shows an example of the internal configuration of the autonomous moving body 110. In this embodiment, the autonomous moving body 110 includes a camera 112, a sensing unit 622, a communication unit 624, an input / output unit 626, a driving unit 628, and a control unit 114. In this embodiment, the control unit 114 includes a data storage unit 630, a data collection unit 642, a communication control unit 644, an input / output control unit 646, a driving control unit 648, and a route guidance unit 650.
[0084] In this embodiment, the sense unit 622 includes various sensors. The sense unit 622 includes, for example, various internal sensors. The sense unit 622 includes, for example, various external sensors. Examples of the internal sensors include a rotary encoder, a speed sensor, an angular velocity sensor, an angular acceleration sensor, and a gravity sensor. Examples of the external sensors include a camera, a microphone, a distance sensor, an obstacle detection sensor, a LiDAR, and a GPS signal receiver.
[0085] In this embodiment, the communication unit 624 transmits and receives information to and from an external communication device. The communication unit 624 transmits and receives information to and from the management server 120, for example.
[0086] In this embodiment, the input / output unit 626 functions as an interface with the user. For example, the input / output unit 626 accepts input from the user. For example, the input / output unit 626 presents information to the user. Examples of the input / output unit 626 include various input devices and various output devices. Examples of the input device include a touch panel, a keyboard, a pointing device, a camera, and a microphone. Examples of the output device include a display, a projector, and a speaker.
[0087] In this embodiment, the driving unit 628 moves the autonomous moving body 110. The driving unit 628 moves the autonomous moving body 110, for example, in accordance with an instruction from the drive control unit 648. Examples of the driving unit 628 include a power generating device such as a motor or an engine, and a propulsion generating device such as a wheel or a caterpillar.
[0088] In this embodiment, the data storage unit 630 stores various types of information related to the autonomous mobile body 110. In one embodiment, the data storage unit 630 stores various types of information used in information processing in the autonomous mobile body 110. In another embodiment, the data storage unit 630 stores various types of information generated in information processing in the autonomous mobile body 110.
[0089] The data storage unit 630 stores, for example, identification information of each of the multiple reference points 20 and image data or imaging data of the reference images 22 of each of the multiple reference points 20 in association with each other. The data storage unit 630 stores, for example, identification information of each of the multiple reference points 20 and feature amounts of the reference images 22 of each of the multiple reference points 20 in association with each other. The above feature amounts may be estimation results of the transformation model 300 for each of the multiple reference images 22. As described above, the above feature amounts may be feature vectors.
[0090] In this embodiment, the data collection unit 642 collects various types of information related to the autonomous moving body 110. In one embodiment, the data collection unit 642 acquires image data of an image captured by the camera 112. In another embodiment, the data collection unit 642 acquires various types of data output by the sense unit 622. For example, the data collection unit 642 acquires the auxiliary data described above or data for generating the auxiliary data.
[0091] In this embodiment, the communication control unit 644 controls the operation of the communication unit 624. In this embodiment, the input / output control unit 646 controls the operation of the input / output unit 626. In this embodiment, the drive control unit 648 controls the operation of the drive unit 628. The drive control unit 648 may control the operation of the drive unit 628 based on the control amount derived by the route guidance unit 650. In this embodiment, the route guidance unit 650 executes the navigation process described above. Details of the route guidance unit 650 will be described later.
[0092] The drive control unit 648 may be an example of a movement control unit. The route guidance unit 650 may be an example of a movement control unit, a control device, or a route planning device.
[0093] 7 illustrates an example of the internal configuration of the route guidance unit 650. In this embodiment, the route guidance unit 650 includes a global route planning unit 720 and a local route planning unit 740. In this embodiment, the local route planning unit 740 includes a route information acquisition unit 742, a target point determination unit 744, a target image acquisition unit 746, a current image acquisition unit 752, and a control amount determination unit 754.
[0094] In this embodiment, the global route planning unit 720 plans a global route (as described above, it may be referred to as a planned route) which is a route from a starting point to a destination point of the autonomous moving body 110. The global route planning unit 720 may generate route information indicating one or more relay points that constitute the planned route. Details of the global route planning unit 720 will be described later.
[0095] In this embodiment, the local path planning unit 740 plans a local path, which is a path from the current position or position and attitude of the autonomous moving body 110 to the above-mentioned target point. In addition, the local path planning unit 740 determines a specific control amount of the autonomous moving body 110, and outputs information indicating the control amount to the drive control unit 648. This allows the local path planning unit 740 to control the movement of the autonomous moving body 110.
[0096] In this embodiment, the route information acquisition unit 742 acquires the above-mentioned route information from, for example, the global route planning unit 720. In this embodiment, the destination point determination unit 744 determines the above-mentioned destination point.
[0097] In one embodiment, when the route guidance unit 650 determines to start route guidance, the destination point determination unit 744 determines one of the reference images 22 arranged in the vicinity of the starting point as the next destination point. The destination point determination unit 744 may determine the reference image 22 arranged closest to the starting point as the next destination point. The above-mentioned departure point is, for example, the position or position and attitude of the autonomous moving body 110 at the time when the route guidance unit 650 determines to start route guidance.
[0098] In another embodiment, after route guidance by the route guidance unit 650 is started and the autonomous moving body 110 starts moving, the destination point determination unit 744 determines, for example, from among one or more relay points indicated by the above-mentioned route information, a relay point N points (N is an integer equal to or greater than 1) ahead of the relay point that is the current destination point, as the next destination point. The value of N is determined, for example, based on the geographical interval between the multiple reference points 20.
[0099] In this embodiment, the target image acquisition unit 746 acquires a target image. For example, the target image acquisition unit 746 refers to the data storage unit 630 and acquires, as a target image, a reference image 22 associated with a reference point 20 that is a target point, from among the multiple reference images 22.
[0100] In this embodiment, the current image acquisition unit 752 acquires a current image. Specifically, the current image acquisition unit 752 accesses the data collection unit 642, for example, to acquire image data of an image captured by the camera 112 in the current position and orientation of the autonomous moving body 110.
[0101] In this embodiment, the control amount determiner 754 derives the value of the above-mentioned relative index based on, for example, the target image and the current image. The control amount determiner 754 inputs, for example, the target image and the current image to the trained control amount estimation model 400. The control amount determiner 754 derives, for example, an output corresponding to an estimation result related to the relative index from the control amount estimation model 400. The control amount determiner 754 may determine the value of the relative index indicated by the output from the control amount estimation model 400 as the above-mentioned control amount.
[0102] In this embodiment, the controlled variable determiner 754 may correct the estimation result obtained by the controlled variable estimation model 400. The controlled variable determiner 754 corrects the estimation result obtained by the controlled variable estimation model 400 by, for example, a particle filter or a sequential Monte Carlo method.
[0103] Specifically, the control amount determination unit 754 first estimates the current position of the autonomous moving body 110 by a particle filter or a sequential Monte Carlo method. Next, the control amount determination unit 754 corrects the value of the relative index described above based on the estimated current position. The control amount determination unit 754 may determine the value of the relative index after correction as the above-mentioned control amount.
[0104] The global path planner 720 may be an example of a path planner or a path planning device. The local path planner 740 may be an example of a movement control unit, a control device, or a path planning device. The control amount determiner 754 may be an example of a relative index deriving unit or a correction unit.
[0105] Fig. 8 illustrates an example of the internal configuration of the global path planner 720. Fig. 8 illustrates an example of the internal configuration of the global path planner 720 together with an example of the internal structure of the data storage unit 630. As shown in Fig. 8, in this embodiment, the data storage unit 630 stores the conversion model 300. In this embodiment, the data storage unit 630 includes a reference information storage unit 812 and a topological map storage unit 814.
[0106] In this embodiment, the global route planning unit 720 includes a feature vector generating unit 822 and a similarity derivation unit 824. The global route planning unit 720 includes a user input acquiring unit 842, a destination image acquiring unit 844, and an end node determining unit 846. The global route planning unit 720 includes a departure point image acquiring unit 864 and a start node determining unit 866. In this embodiment, the global route planning unit 720 includes a global route determining unit 880.
[0107] In this embodiment, the reference information storage unit 812 stores various information related to the reference points 20 and / or the reference images 22. For example, the reference information storage unit 812 stores identification information of each of the multiple reference points 20 and image data or imaging data of the reference images 22 of each of the multiple reference points 20 in association with each other. The reference information storage unit 812 may store identification information of each of the multiple reference points 20 and identification information of the reference images 22 of each of the multiple reference points 20 in association with each other. This allows the identification information of each of the multiple reference points 20 to be associated with image data or imaging data of the reference images 22 of each of the multiple reference points 20. The reference information storage unit 812 may further store identification information of each of the multiple reference points 20 and position information of each of the multiple reference points 20 in association with each other.
[0108] The reference information storage unit 812 stores, for example, identification information of each of the multiple reference points 20 and feature amounts of the reference images 22 of each of the multiple reference points 20 in association with each other. The feature amounts may be estimation results of the transformation model 300 for each of the multiple reference images 22. As described above, the feature amounts may be feature vectors. The reference information storage unit 812 may further store identification information of each of the multiple reference points 20 and position information of each of the multiple reference points 20 in association with each other.
[0109] In this embodiment, the topological map storage unit 814 stores the above-mentioned topological map. As described above, the topological map is expressed as a weighted graph including, for example, a plurality of nodes corresponding to the plurality of reference points 20, and one or more edges connecting the plurality of nodes. In the above weighted graph, the weight of each of the one or more edges indicates, for example, the similarity of the reference images 22 of the two reference points 20 corresponding to each of the two nodes connected by each edge.
[0110] In this embodiment, the feature vector generation unit 822 receives an image and outputs a value related to an estimation result of the feature amount of the image. The feature vector generation unit 822 outputs the value related to the estimation result to, for example, the similarity derivation unit 824.
[0111] Specifically, the feature vector generation unit 822 (i) inputs an image (sometimes referred to as a target image) that is to be the subject of information processing in the feature vector generation unit 822 to the trained conversion model 300, and (ii) derives an output corresponding to the feature amount of the target image from the conversion model 300. As described above, the target image is, for example, a two-dimensional RGB image, and the feature amount of the target image is, for example, a feature vector.
[0112] The feature vector generation unit 822, for example, accesses the reference information storage unit 812 to acquire image data of the multiple reference images 22. The feature vector generation unit 822 receives each of the multiple reference images 22 as input, and outputs values related to the estimation results of the feature amounts of each of the multiple reference images 22 to the similarity derivation unit 824.
[0113] The feature vector generation unit 822 acquires, for example, image data of an image of a destination point of the autonomous moving body 110 (sometimes referred to as a destination image) from the destination image acquisition unit 844. The feature vector generation unit 822 receives the destination image as input, and outputs a value related to an estimation result of the feature amount of the destination image to the similarity derivation unit 824.
[0114] The feature vector generation unit 822 acquires image data of an image of the departure point of the autonomous moving body 110 (sometimes referred to as a departure image) from, for example, the departure image acquisition unit 864. The feature vector generation unit 822 receives the departure image as input, and outputs a value related to the estimation result of the feature amount of the departure image to the similarity derivation unit 824.
[0115] In this embodiment, the similarity derivation unit 824 derives the similarity between two images. Specifically, the similarity derivation unit 824 derives the similarity between the two images based on the feature amounts of the two images. When the feature amounts of the two images are feature vectors, the similarity derivation unit 824 derives, for example, the cosine similarity of the feature vectors of the two images as the similarity between the two images.
[0116] In one embodiment, the similarity derivation unit 824 derives a similarity between each of the reference images 22 of the multiple reference points 20 arranged in the managed area 10 and the destination image. Specifically, the similarity derivation unit 824 derives a similarity between each of the multiple reference images 22 and the destination image based on a feature amount of each of the multiple reference images 22 and a feature amount of the destination image. The similarity derivation unit 824 may acquire the feature amount of each image from the feature vector generation unit 822, or may access the reference information storage unit 812 to acquire the feature amount of each image.
[0117] The similarity derivation unit 824 may output information indicating the similarity between each of the reference images 22 of the multiple reference points 20 arranged in the managed area 10 and the destination image to the end node determination unit 846. This allows the end node determination unit 846 to determine, as the destination point, the imaging position of the reference image 22 that has the greatest similarity to the destination image among the multiple reference images 22, or the imaging position and imaging attitude of the reference image 22.
[0118] In another embodiment, the similarity derivation unit 824 derives the similarity between each of the reference images 22 of the multiple reference points 20 arranged in the managed area 10 and the departure point image. Specifically, the similarity derivation unit 824 derives the similarity between each of the multiple reference images 22 and the departure point image based on the feature amount of each of the multiple reference images 22 and the feature amount of the departure point image. The similarity derivation unit 824 may acquire the feature amount of each image from the feature vector generation unit 822, or may access the reference information storage unit 812 to acquire the feature amount of each image.
[0119] The similarity derivation unit 824 may output information indicating the similarity between each of the reference images 22 of the multiple reference points 20 arranged in the managed area 10 and the departure point image to the start node determination unit 866. This allows the start node determination unit 866 to determine, as the departure point, the imaging position of the reference image 22 having the greatest similarity to the departure point image among the multiple reference images 22, or the imaging position and imaging attitude of the reference image 22.
[0120] In this embodiment, the user input acquisition unit 842 acquires an input from a user regarding a destination point of the autonomous moving body 110. The user input acquisition unit 842 cooperates with, for example, the input / output unit 626 and the input / output control unit 646 to acquire the above-mentioned input from the user.
[0121] In one embodiment, the user can input instructions regarding the destination point by operating the input / output unit 626 to select a point on a map displayed on the user interface screen. In another embodiment, the user can input instructions regarding the destination point by operating the input / output unit 626 to select a reference image 22 of the destination point on the user interface screen. In yet another embodiment, the user can input instructions regarding the destination point by operating the input / output unit 626 to input image data of an image of the destination point that the user has acquired in advance to the mobile object management system 100.
[0122] In this embodiment, the destination image acquisition unit 844 acquires an image of the destination point of the autonomous moving body 110. Specifically, when the user inputs image data of an image of the destination point acquired in advance to the moving body management system 100, the destination image acquisition unit 844 acquires an image indicated by the image data input by the user to the moving body management system 100 as the destination image.
[0123] The destination image acquisition unit 844 may output image data of the destination image to the feature vector generation unit 822. As described above, when the feature vector generation unit 822 acquires a destination image, the feature vector generation unit 822 receives the destination image as input, and outputs a value related to the estimation result of the feature amount of the destination image to the similarity derivation unit 824. The value related to the estimation result of the feature amount of the destination image is, for example, a feature vector of the destination image.
[0124] When the similarity derivation unit 824 acquires the feature vector of the destination image, the similarity derivation unit 824 derives the similarity between the destination image and each of the reference images 22 of the multiple reference points 20 arranged in the management area 10. The similarity derivation unit 824 outputs information indicating the similarity of each combination of two images to the end node determination unit 846.
[0125] In this embodiment, the end node determination unit 846 determines the end node. The end node determination unit 846 determines the end node based on, for example, various inputs from the user.
[0126] In one embodiment, when the user input acquisition unit 842 acquires an instruction from the user regarding a destination point via a user interface screen displayed on the input / output unit 626, the end node determination unit 846 determines, as the end node, the node corresponding to the reference point 20 selected as the destination point by the instruction.
[0127] In another embodiment, when the user input acquisition unit 842 acquires image data of an image of a destination point acquired in advance by a user, the end node determination unit 846 acquires, for example, from the similarity derivation unit 824, information indicating the similarity between each of the reference images 22 of the multiple reference points 20 arranged in the managed area 10 and the destination image. The end node determination unit 846 determines, as the destination point, the imaging position of the reference image 22 having the highest similarity to the destination image among the multiple reference images 22 or the imaging position and imaging attitude of the reference image 22. The end node determination unit 846 determines, as the end node, a node corresponding to the reference point 20 at which the reference image 22 having the highest similarity to the destination image was captured. As a result, for example, a node corresponding to the destination point or a point arranged in the vicinity of the destination point is determined as the end node.
[0128] In this embodiment, the departure point image acquisition unit 864 acquires image data of the departure point image. For example, the departure point image acquisition unit 864 accesses the data collection unit 642 to acquire image data of the image captured by the camera 112 at the position and orientation of the autonomous moving body 110 at the departure point.
[0129] The departure image acquisition unit 864 may output image data of the departure image to the feature vector generation unit 822. As described above, when the feature vector generation unit 822 acquires the departure image, the feature vector generation unit 822 receives the departure image as input, and outputs a value related to the estimation result of the feature amount of the departure image to the similarity derivation unit 824. The value related to the estimation result of the feature amount of the departure image is, for example, a feature vector of the departure image.
[0130] When the similarity derivation unit 824 acquires the feature vector of the departure point image, the similarity derivation unit 824 derives the similarity between the departure point image and each of the reference images 22 of the multiple reference points 20 arranged in the management area 10. The similarity derivation unit 824 outputs information indicating the similarity of each combination of two images to the start point node determination unit 866.
[0131] In this embodiment, the start node determination unit 866 determines the start node. For example, the start node determination unit 866 determines a node corresponding to the reference point 20 located at or near the start point as the start node. If the start point is a specific reference point 20, the start node determination unit 866 determines a node corresponding to the specific reference point 20 as the start node. If the start point is not one of the multiple reference points 20, the start node determination unit 866 determines, for example, a node corresponding to the reference point 20 located closest to the start point as the start node.
[0132] The start node determination unit 866 acquires, for example, from the similarity derivation unit 824, information indicating the similarity between each of the reference images 22 of the multiple reference points 20 arranged in the managed area 10 and the departure point image. The start node determination unit 866 determines, as the departure point, the imaging position of the reference image 22 having the greatest similarity to the departure point image among the multiple reference images 22, or the imaging position and imaging attitude of the reference image 22. The start node determination unit 866 determines, as the start node, a node corresponding to the reference point 20 at which the reference image 22 having the greatest similarity to the departure point image was captured.
[0133] In this embodiment, the global route determination unit 880 determines the planned route described above. The global route determination unit 880 uses a weighted graph constituting the topological map described above to derive a node sequence that minimizes the sum of edge weights from the start node to the end node. The node sequence is derived, for example, by a known shortest route planning method. In this way, the planned route is determined.
[0134] The global route determination unit 880 may execute a process (sometimes referred to as a reroute process) for changing a planned route once determined. For example, the global route determination unit 880 executes the reroute process when the occurrence of a predetermined type of event is detected during a period in which the autonomous moving body 110 is traveling according to a planned route determined by the global route determination unit 880. Examples of the above-mentioned event include (i) an obstacle being detected on the planned route, (ii) a predetermined period having elapsed since the time when the previous reroute process was performed, (iii) a travel distance of the autonomous moving body 110 from the time when the previous reroute process was performed becoming equal to or greater than a predetermined threshold, (iv) a predetermined time having arrived, and (v) the autonomous moving body 110 reaching a predetermined point. The details of the reroute process will be described later.
[0135] The reference information storage unit 812 may be an example of a storage device or a first storage device. The topological map storage unit 814 may be an example of a second storage device. The feature vector generation unit 822 may be an example of a feature output unit. The global route determination unit 880 may be an example of a route planning unit. The reference image 22 having the highest similarity to the destination image may be an example of a reference image 22 most similar to the destination image. The reference image 22 having the highest similarity to the departure image may be an example of a reference image 22 most similar to the departure image.
[0136] Fig. 9 illustrates an example of the internal configuration of the control amount determiner 754. Fig. 9 illustrates an example of the internal configuration of the control amount determiner 754 together with an example of the internal structure of the data storage unit 630. As shown in Fig. 9, in this embodiment, the data storage unit 630 stores the control amount estimation model 400.
[0137] In this embodiment, the control amount determiner 754 includes a control amount deriver 922 and a control amount corrector 924. In one embodiment, the control amount determiner 754 determines the calculation result of the control amount deriver 922 as the above-mentioned control amount. In another embodiment, the control amount determiner 754 determines the calculation result of the control amount corrector 924 as the above-mentioned control amount.
[0138] The control amount derivation unit 922 derives the value of the above-mentioned relative index based on, for example, the target image and the current image. The control amount derivation unit 922 inputs, for example, the target image and the current image to the trained control amount estimation model 400. The control amount derivation unit 922 derives, for example, an output corresponding to an estimation result regarding the relative index from the control amount estimation model 400.
[0139] In this embodiment, the control amount correcting unit 924 may correct the estimation result by the control amount estimation model 400. The control amount correcting unit 924 corrects the estimation result by the control amount estimation model 400 by, for example, a particle filter or a sequential Monte Carlo method. The control amount correcting unit 924 corrects the estimation result by the control amount estimation model 400 according to or in accordance with the procedure described in, for example, Frank Dellaert, Dieter Fox, Wolfram Burgard, and Sebastian Thrun, "Monte carlo localization for mobile robots," Proceedings 1999 IEEE international conference on robotics and automation, Vol. 2, pp. 1322-1328, IEEE, 1999.
[0140] The control amount derivation unit 922 may be an example of a relative index derivation unit. The control amount correction unit 924 may be an example of a correction unit.
[0141] 10 illustrates an example of the internal configuration of the management server 120. In this embodiment, the management server 120 includes a data storage unit 1020, a topological map generation unit 1030, an imaging data acquisition unit 1040, a conversion model teacher data generation unit 1042, an estimation model teacher data generation unit 1044, a conversion model generation unit 1052, and a control amount estimation model generation unit 1054.
[0142] In this embodiment, the data storage unit 1020 stores various types of information related to the autonomous mobile body 110. In one embodiment, the data storage unit 630 stores various types of information used in information processing in the autonomous mobile body 110 and / or the management server 120. In another embodiment, the data storage unit 630 stores various types of information generated in information processing in the autonomous mobile body 110 and / or the management server 120.
[0143] In this embodiment, the topological map generating unit 1030 generates the above-mentioned topological map. Specifically, for each combination of two reference points 20 selected from a plurality of reference points 20 arranged inside the managed area 10, the topological map generating unit 1030 derives the similarity between two reference images 22 corresponding to the two reference points 20. The similarity between the two reference images 22 is derived, for example, by a procedure similar to the similarity derivation process in the feature vector generating unit 822 and the similarity derivation unit 824.
[0144] Specifically, the topological map generating unit 1030 first, for each of the above-mentioned multiple combinations, (i) inputs each of the two reference images 22 constituting each pair to the trained conversion model 300, and (ii) derives an output corresponding to each feature amount of the two reference images 22 from the conversion model 300. As described above, the reference image 22 is, for example, a two-dimensional RGB image, and the feature amount of the reference image 22 is, for example, a feature vector. Next, the topological map generating unit 1030 derives a similarity between the above-mentioned two reference images 22 based on the feature amount of each of the two reference images 22. The above-mentioned similarity may be a cosine similarity.
[0145] In this embodiment, the imaging data acquisition unit 1040 acquires imaging data of an image captured by the camera 112 from the autonomous moving body 110. As described above, the imaging data may be data in which image data of an image captured by the camera 112 is associated with auxiliary data indicating the imaging position and / or imaging time of the image. The auxiliary data may further include the imaging attitude of the image.
[0146] The conversion model training data generation unit 1042 generates training data used in distance learning of the conversion model 300. The conversion model training data generation unit 1042 may generate a data set including a plurality of training data. The conversion model training data generation unit 1042 will be described in detail later.
[0147] Each of the multiple teacher data used in distance learning of the conversion model 300 may be data in which a reference image, one or more similar images, and one or more dissimilar images are associated with each other. Three or more different reference images 22 among the multiple reference images 22 may be used as each of the reference image, the one or more similar images, and the one or more dissimilar images.
[0148] The geographical positional relationship between the point where each of the one or more similar images was captured and the point where the reference image was captured meets a predetermined condition. On the other hand, the geographical positional relationship between the point where each of the one or more dissimilar images was captured and the point where the reference image was captured does not meet the predetermined condition. Examples of the predetermined condition include a condition that the geographical distance between the points where each image was captured is equal to or smaller than a predetermined first threshold value, and / or a condition that the length of the period between the times when each image was captured is equal to or smaller than a predetermined second threshold value.
[0149] In this embodiment, the estimation model training data generation unit 1044 generates training data used in machine learning of the controlled variable estimation model 400. The estimation model training data generation unit 1044 may generate a data set including a plurality of training data.
[0150] Each of the multiple teacher data used in the machine learning of the control amount estimation model 400 is, for example, data in which an image is associated with an imaging position, which is the position of an imaging device that captured the image, and an imaging attitude, which is the attitude of the imaging device. The imaging device may be the camera 112 of the autonomous moving body 110, may be the camera 112 of another autonomous moving body 110, or may be an imaging device different from the camera 112.
[0151] In this embodiment, the conversion model generation unit 1052 generates a trained conversion model 300. For example, the conversion model generation unit 1052 first acquires a dataset of training data for distance learning of the conversion model 300 from the conversion model training data generation unit 1042. As described above, the above dataset includes multiple pieces of training data.
[0152] Next, the conversion model generation unit 1052 executes distance learning using the above dataset to generate a trained conversion model 300. Specifically, the conversion model generation unit 1052 executes distance learning in which each image included in the dataset is input and a value related to an estimation result of the feature amount of each image is output, to generate a trained conversion model 300.
[0153] In this embodiment, the control amount estimation model generation unit 1054 generates a trained control amount estimation model 400. For example, the control amount estimation model generation unit 1054 first acquires a data set of teacher data for machine learning of the control amount estimation model 400 from the estimation model teacher data generation unit 1044. As described above, the above data set includes a plurality of teacher data.
[0154] Next, the control amount estimation model generating unit 1054 generates a trained control amount estimation model 400 by executing machine learning using a combination of two teacher data included in the above data set. As described above, in each of the two teacher data constituting each set, an image, an imaging position of the image, and an imaging attitude of the image are associated with each other. Specifically, the control amount estimation model generating unit 1054 generates a trained control amount estimation model 400 by machine learning that inputs an image (sometimes referred to as a first image) included in one of the two teacher data and an image (sometimes referred to as a second image) included in the other of the two teacher data for each of a plurality of combinations, and outputs a value related to an estimation result of a relative relationship between the imaging position and imaging attitude corresponding to the first image and the imaging position and imaging attitude corresponding to the second image.
[0155] The imaging data acquisition unit 1040 may be an example of an image data acquisition unit. The conversion model teacher data generation unit 1042 may be an example of an image data acquisition unit or a second data set generation unit. The estimation model teacher data generation unit 1044 may be an example of an image data acquisition unit or a first data set generation unit. The conversion model generation unit 1052 may be an example of a second data set acquisition unit or a feature output model generation unit. The control amount estimation model generation unit 1054 may be an example of a first data set acquisition unit or an estimation model generation unit.
[0156] The teacher data for the conversion model may be an example of second learning data. The data set of teacher data for the conversion model may be an example of the second learning data set. The teacher data for the controlled variable estimation model may be an example of the first learning data. The data set of teacher data for the controlled variable estimation model may be an example of the first learning data set.
[0157] 11 illustrates an example of the internal configuration of the conversion model teacher data generation unit 1042. In this embodiment, the conversion model teacher data generation unit 1042 includes an imaging data extraction unit 1120, a judgment unit 1130, a similar image determination unit 1142, a dissimilar image determination unit 1144, and a data set generation unit 1146.
[0158] In this embodiment, the imaging data extraction unit 1120 extracts imaging data of a plurality of images to be used as teacher data from the imaging data of a plurality of images acquired by the imaging data acquisition unit 1040. For example, the imaging data extraction unit 1120 extracts image data at a predetermined sampling interval from image data of a plurality of images captured in succession.
[0159] In this case, images sampled close to each other are images captured at relatively close geographical locations. Similarly, images sampled far from each other are images captured at relatively distant geographical locations. By appropriately setting the sampling interval, multiple images can be extracted such that images sampled close to each other are relatively similar and images sampled far from each other are relatively different.
[0160] In this embodiment, the determination unit 1130 determines whether or not a relationship between a geographical location where each image was captured and a geographical location where at least one other image included in the multiple images was captured meets a predetermined condition for each of the multiple images extracted by the imaging data extraction unit 1120. Examples of the predetermined condition include a condition that the geographical distance between the geographical location where each image was captured and the geographical location where the other image corresponding to each image was captured is equal to or smaller than a predetermined first threshold value, and / or a condition that the length of a period between the time when each image was captured and the time when the other image corresponding to each image was captured is equal to or smaller than a predetermined second threshold value.
[0161] In one embodiment, the determination unit 1130 performs the above determination process based on the geographical location at which each image indicated by the auxiliary data of each image was captured. In another embodiment, the determination unit 1130 performs the above determination process based on the time at which each image indicated by the auxiliary data of each image was captured.
[0162] For example, if the above images are captured by a camera 112 mounted on an autonomous mobile object 110 that travels inside the managed area 10 by manual or remote control, the difference in capture time between the two images is approximately proportional to the distance between the geographical positions of the two images. This makes it possible to determine whether the positional relationship between a specific image and other images meets a predetermined condition based on the capture time of each image.
[0163] In one embodiment, the determination unit 1130 performs the above-mentioned determination process between a specific image included in the multiple images extracted by the sampling process in the imaging data extraction unit 1120 and all other images included in the multiple images. In another embodiment, the determination unit 1130 performs the above-mentioned determination process between a specific image included in the multiple images extracted by the sampling process in the imaging data extraction unit 1120 and some of the other images included in the multiple images. For example, the determination unit 1130 performs the above-mentioned determination process between a specific image and other images whose imaging positions or imaging times are included within a predetermined range based on the imaging position or imaging time of the specific image.
[0164] In this embodiment, the similar image determination unit 1142 determines other images whose relationship with the geographical location where the specific image was captured meets a predetermined condition as one or more similar images corresponding to the specific image. Each of the one or more similar images corresponding to the specific image is used as a positive example corresponding to the specific image.
[0165] In this embodiment, the dissimilar image determination unit 1144 determines other images whose relationship with the geographical location where the specific image was captured does not meet a predetermined condition as one or more dissimilar images corresponding to the specific image. Each of the one or more dissimilar images corresponding to the specific image is used as a negative example corresponding to the specific image.
[0166] In this embodiment, the dataset generation unit 1146 generates the above-mentioned dataset of teacher data. Specifically, the dataset generation unit 1146 generates a dataset of teacher data by associating each image, a positive example corresponding to each image, and a negative example corresponding to each image, for each of a plurality of images extracted by the sampling process in the imaging data extraction unit 1120.
[0167] The imaging data extraction section 1120 may be an example of an image data acquisition section.
[0168] FIG. 12 is a schematic diagram showing an example of a conceptual diagram of a topological map. As described above, the topological map is expressed as a weighted graph including, for example, a plurality of nodes corresponding to a plurality of reference points 20, and one or more edges connecting the plurality of nodes. In the weighted graph, the weight of each of the one or more edges indicates, for example, the similarity of the reference images 22 of the two reference points 20 corresponding to the two nodes connected by each edge. This allows the relationship between the plurality of reference points 20 to be expressed without using position coordinates.
[0169] 8, in this embodiment, the data storage unit 630 includes a reference information storage unit 812 and a topological map storage unit 814. In this embodiment, the concept of a topological map will be described using as an example a case in which the reference information storage unit 812 stores an image database 1212 and the topological map storage unit 814 stores a topological map 1214.
[0170] In this embodiment, the image database 1212 stores, in association with each other, for example, identification information (sometimes referred to as reference point IDs) of each of the multiple reference points 20 arranged inside the managed area 10, position information of each of the multiple reference points 20, and identification information (sometimes referred to as image IDs) of each of the reference images 22 of the multiple reference points 20. This allows, for example, when a user specifies the position or reference point ID of a goal point, a reference image 22 corresponding to the goal point can be extracted.
[0171] In this embodiment, the topological map 1214 stores, for each combination of two nodes 1220 selected from the plurality of nodes 1220 corresponding to each of the plurality of reference points 20, identification information of the two nodes constituting the combination and a weight assigned to an edge 1240 corresponding to the combination, in association with each other. This represents a weighted graph including the plurality of nodes 1220 corresponding to each of the plurality of reference points 20 and one or more edges 1240 connecting each of the plurality of nodes.
[0172] As described above, in this embodiment, the magnitude of the weight assigned to each edge is determined based on the similarity of the reference images 22 of the two reference points 20 corresponding to the two nodes 1220 connected to each edge. For example, the similarity of the reference images 22 of the two reference points 20 corresponding to the two nodes 1220 connected to each edge is used as the weight assigned to each edge.
[0173] In the embodiment described in relation to FIG. 12, seven edges are set between each of the eight nodes 1220 and the other seven nodes 1220. Each of the eight nodes 1220 corresponds to eight reference points 20 arranged inside the management area 10. In FIG. 12, in order to facilitate understanding of the topological map 1214, each of the eight nodes 1220 is arranged at a position that imitates the geographical relationship of each of the eight reference points 20. However, it should be noted that in this embodiment, the topological map 1214 does not include position information of each node.
[0174] Fig. 13 shows an example of the reroute process. As described in relation to Fig. 8, the global route determination unit 880 executes the reroute process as necessary. In this embodiment, the details of the reroute process will be described by taking as an example a case where the global route determination unit 880 executes the reroute process during a period in which the autonomous moving body 110 moves from a relay point A to a relay point B that constitutes a planned route.
[0175] According to this embodiment, first, in S1312, the global route determination unit 880 detects the occurrence of an event related to rerouting. Examples of the above event include (i) an obstacle being detected on the planned route, (ii) a predetermined period of time having elapsed since the time when the previous rerouting process was performed, (iii) the travel distance of the autonomous moving body 110 from the time when the previous rerouting process was performed becoming equal to or greater than a predetermined threshold, (iv) a predetermined time having arrived, and (v) the autonomous moving body 110 reaching a predetermined point.
[0176] Next, in S1314, the global route determination unit 880 determines whether or not it is difficult to travel between the relay point A and the relay point B. For example, when the obstacle detection sensor or LiDAR arranged in the sense unit 622 detects an obstacle on the planned route, the global route determination unit 880 determines whether or not it is possible to move the autonomous moving body 110 along the planned route.
[0177] If it is determined that passage between relay point A and relay point B is difficult (Yes in S1314), in S1316, the global route determination unit 880 changes the weight of the topological map. Specifically, the global route determination unit 880 changes the weight of an edge in the topological map that corresponds to the route between relay point A and relay point B. The global route determination unit 880 increases the weight of the above edge so that the route between relay point A and relay point B is less likely to be included in the planned route. The global route determination unit 880 may set the weight of the above edge to infinity or the upper limit of a settable value.
[0178] Next, in S1318, the global route determination unit 880 plans a global route from the current position to the destination point. For example, the global route determination unit 880 determines a node corresponding to the relay point A as a new start node, and derives a node sequence that minimizes the sum of edge weights from the new start node to the end node. This allows a planned route that does not include a route between relay points A and B to be derived.
[0179] On the other hand, if it is not determined that passage between relay point A and relay point B is difficult (No in S1314), S1316 is not executed and S1318 is executed. In this case, the new planned route may include a route between relay point A and relay point B. This ends the reroute process.
[0180] 14 shows an example of a computer 3000 in which multiple aspects of the present invention may be embodied in whole or in part. At least a part of the mobile object management system 100 may be realized by the computer 3000. For example, at least a part of the autonomous mobile object 110 may be realized by the computer 3000. At least a part of the control unit 114 may be realized by the computer 3000. For example, at least a part of the management server 120 may be realized by the computer 3000.
[0181] A program installed on the computer 3000 may cause the computer 3000 to function as or perform operations associated with an apparatus according to an embodiment of the present invention or one or more "parts" of the apparatus, and / or to perform a process or steps of the process according to an embodiment of the present invention. Such a program may be executed by the CPU 3012 to cause the computer 3000 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.
[0182] The computer 3000 according to this embodiment includes a CPU 3012, a RAM 3014, a graphic controller 3016, and a display device 3018, which are connected to each other by a host controller 3010. The computer 3000 also includes input / output units such as a communication interface 3022, a hard disk drive 3024, a DVD-ROM drive 3026, and an IC card drive, which are connected to the host controller 3010 via an input / output controller 3020. The computer also includes legacy input / output units such as a ROM 3030 and a keyboard 3042, which are connected to the input / output controller 3020 via an input / output chip 3040.
[0183] The CPU 3012 operates according to a program stored in the ROM 3030 and the RAM 3014, thereby controlling each unit. The graphic controller 3016 acquires image data generated by the CPU 3012 into a frame buffer or the like provided in the RAM 3014 or into itself, and causes the image data to be displayed on the display device 3018.
[0184] The communications interface 3022 communicates with other electronic devices via a network. The hard disk drive 3024 stores programs and data used by the CPU 3012 in the computer 3000. The DVD-ROM drive 3026 reads programs or data from the DVD-ROM 3001 and provides the programs or data to the hard disk drive 3024 via the RAM 3014. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.
[0185] The ROM 3030 stores therein a boot program or the like executed by the computer 3000 upon activation, and / or a program that depends on the hardware of the computer 3000. The input / output chip 3040 may also connect various input / output units to the input / output controller 3020 via a parallel port, a serial port, a keyboard port, a mouse port, etc.
[0186] The programs are provided by a computer-readable storage medium such as a DVD-ROM 3001 or an IC card. The programs are read from the computer-readable storage medium, installed in the hard disk drive 3024, the RAM 3014, or the ROM 3030, which are also examples of computer-readable storage media, and executed by the CPU 3012. Information processing described in these programs is read by the computer 3000, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing an operation or processing of information according to the use of the computer 3000.
[0187] For example, when communication is performed between computer 3000 and an external device, CPU 3012 may execute a communication program loaded in RAM 3014 and instruct communication interface 3022 to perform communication processing based on the processing described in the communication program. Under the control of CPU 3012, communication interface 3022 reads transmission data stored in a transmission buffer area provided in RAM 3014, hard disk drive 3024, DVD-ROM 3001, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.
[0188] Furthermore, CPU 3012 may cause all or a necessary portion of a file or database stored in an external recording medium such as hard disk drive 3024, DVD-ROM drive 3026 (DVD-ROM 3001), an IC card, etc. to be read into RAM 3014, and perform various types of processing on the data on RAM 3014. CPU 3012 may then write back the processed data to the external recording medium.
[0189] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 3012 may perform various types of processing on the data read from the RAM 3014, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and writes back the results to the RAM 3014. The CPU 3012 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 3012 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.
[0190] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 3000 or in the vicinity of the computer 3000. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the above-described programs to the computer 3000 via the network.
[0191] The blocks in the flowcharts and block diagrams in the above embodiments may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer readable instructions stored on a computer readable storage medium, and / or a processor provided with computer readable instructions stored on a computer readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.
[0192] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.
[0193] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages such as the “C” programming language or similar programming languages.
[0194] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.
[0195] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications or improvements can be made to the above embodiments. Furthermore, the matters described for a specific embodiment can be applied to other embodiments to the extent that they are not technically inconsistent. Furthermore, each component may have the same features as other components having the same name and different reference numerals. It is clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.
[0196] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order.
[0197] For example, the present specification discloses the following: (Item 1) A control device for controlling a moving object that autonomously moves within a predetermined geographical area, the moving body includes a first imaging device that captures an image of the surroundings of the moving body; the control device is configured to be able to access a first storage device that stores a plurality of reference images, which are images captured in advance at a plurality of points located within the geographical range, in association with identification information of the plurality of points; The control device includes: a route information acquisition unit that acquires route information indicating one or more relay points, which are one or more points constituting a planned route from a departure point of the mobile body to a destination point, among the plurality of points; a destination point determination unit that determines a destination point that is a destination of the moving body from among the one or more relay points; a target image acquisition unit that refers to the first storage device and acquires a target image that is a reference image associated with the target location from among the plurality of reference images; a current image acquisition unit that acquires a current image that is an image captured by the first imaging device at a current position and attitude of the moving object; a relative index derivation unit that derives a value of a relative index, which is an index indicating a relative relationship between a current position and orientation of the moving body and a position and orientation of the moving body that is a target at the target point, based on the target image and the current image; Equipped with The relative index derivation unit inputting the target image and the current image into a trained estimation model; deriving an output from the estimation model corresponding to an estimation result regarding the relative indicator; Control device. (Item 2) The above estimation model is a feature extraction unit that receives the target image and the current image as inputs and extracts feature amounts of the target image and feature amounts of the current image; a relative index estimation unit that receives the feature amount of the target image and the feature amount of the current image as input and outputs the estimation result regarding the relative index; having Item 1. The control device according to item 1. (Item 3) The feature extraction unit is a first extraction unit that receives the target image and extracts a feature amount of the target image; a second extraction unit that receives the current image and extracts a feature amount of the current image; Including, The first extraction unit (i) inputs the target image into a trained first extraction model, and (ii) derives an output corresponding to an estimation result regarding a feature amount of the target image from the first extraction model; The second extraction unit (i) inputs the current image into a trained second extraction model, and (ii) derives an output corresponding to an estimation result regarding a feature amount of the current image from the second extraction model; The first extraction model and the second extraction model are trained models generated by distance learning using an image as input and a feature vector of the image as output. Item 2. The control device according to item 2. (Item 4) a first data set acquisition unit that acquires a first training data set including a plurality of first training data; an estimation model generation unit that generates the trained estimation model by performing machine learning using a combination of two pieces of first learning data included in the first learning dataset; Further equipped with each of the first learning data is data in which an image is associated with an imaging position that is a position of a second imaging device that captured the image, and an imaging orientation that is an orientation of the second imaging device; the estimation model generation unit generates the estimation model by machine learning, for each of the plurality of combinations, using as input a first image included in one of the two first learning data sets and a second image included in the other of the two first learning data sets, and outputting a value related to an estimation result of a relative relationship between an imaging position and an imaging orientation corresponding to the first image and an imaging position and an imaging orientation corresponding to the second image; Item 1. The control device according to item 1. (Item 5) a movement control unit that moves the moving body based on the relative index derived by the relative index derivation unit, Item 1. The control device according to item 1. (Item 6) a correction unit that estimates a current position of the moving object by a particle filter or a sequential Monte Carlo method, and corrects a value of the relative index based on the estimated current position; The movement control unit moves the moving object based on the value of the relative index corrected by the correction unit. Item 5. The control device according to item 5. (Item 7) A route planning unit that determines the planned route, the route planning unit refers to a second storage device that stores a weighted graph including a plurality of nodes corresponding to the plurality of points and one or more edges connecting the plurality of nodes, and determines the planned route by deriving a node sequence that minimizes a sum of edge weights from a start node that is a node corresponding to the departure point or a point located in the vicinity of the departure point to an end node that is a node corresponding to the destination point or a point located in the vicinity of the destination point; In the weighted graph, a weight of each of the one or more edges indicates a degree of similarity between the reference images of two points corresponding to two nodes connected by each edge. Item 1. The control device according to item 1. (Item 8) a departure point image acquisition unit that acquires a departure point image that is an image captured by the first imaging device at the departure point; a similarity deriving unit that derives a degree of similarity between each of the plurality of reference images and the departure point image; a start node determination unit that determines a node corresponding to a reference image that is most similar to the start image as the start node; Further comprising: Item 7. The control device according to item 7. (Item 9) a feature output unit that receives each of the plurality of reference images and the departure image as input, and outputs values related to the estimation results of the feature values of each of the plurality of reference images and the feature values of the departure image; The feature output unit (i) inputs a target image, which is an image to be subjected to information processing in the feature output unit, to a trained feature output model, and (ii) derives an output corresponding to the feature of the target image from the feature output model. The similarity derivation unit derives a degree of similarity between each of the plurality of reference images and the departure point image based on a feature amount of each of the plurality of reference images and a feature amount of the departure point image. Item 9. The control device according to item 8. (Item 10) The target image is a two-dimensional RGB image, The feature amount is a feature vector. Item 10. The control device according to item 9. (Item 11) a second data set acquisition unit that acquires a second training data set including a plurality of second training data; a feature output model generation unit that executes metric learning using the second learning dataset to generate the feature output model for receiving an image and outputting a value related to an estimation result of a feature of the image; Further equipped with each of the plurality of second learning data is data in which a reference image, one or more similar images, and one or more dissimilar images are associated with each other; a relationship between a geographical location where each of the one or more similar images was captured and a geographical location where the reference image was captured satisfies a predetermined condition; a relationship between a geographical location where each of the one or more dissimilar images was captured and a geographical location where the reference image was captured does not satisfy the predetermined condition; Item 10. The control device according to item 9. (Item 12) The predetermined conditions are: The geographical distance between the locations at which the images were taken is equal to or smaller than a first predetermined threshold value, and / or a condition that the length of the period between the times at which the images were captured is equal to or less than a second threshold value that is determined in advance; Including, Item 12. The control device according to item 11. (Item 13) Further comprising a second data set generation unit that acquires the second learning data set, The second data set generation unit Image data of a plurality of images captured within the geographic range, and an image data acquisition unit that acquires image data of a plurality of images captured within the geographical range in association with auxiliary data indicating a geographical position at which each of the plurality of images was captured or a time at which each of the plurality of images was captured; a determination unit that determines, for each of the plurality of images, whether or not a geographical relationship between a point at which each image was captured and a point at which at least one other image included in the plurality of images was captured satisfies the predetermined condition, based on the geographical position or the time indicated by the auxiliary data associated with the image data of each image; a similar image determining unit that determines the other images, the geographical relationship of which with respect to the point at which each image was captured being determined to match the predetermined condition, as the one or more similar images corresponding to each image; a dissimilar image determining unit that determines the other images, whose geographical relationship with the point at which each image was captured does not match the predetermined condition, as the one or more dissimilar images corresponding to each image; having Item 12. The control device according to item 11. (Item 14) Further comprising an end node determination unit that determines the end node based on an input from a user. Item 7. The control device according to item 7. (Item 15) A control device for controlling a moving object that autonomously moves within a predetermined geographical area, the moving body includes an imaging device that images the surroundings of the moving body, the control device is configured to be able to access a storage device that stores a plurality of reference images that are images captured in advance at a plurality of points located within the geographical range; The control device includes: a destination image acquisition unit that acquires a destination image that is an image of a destination point of the moving object; a movement control unit that moves the moving object to a point where a degree of similarity between an image captured by the imaging device and the destination image acquired by the destination image acquisition unit exceeds a predetermined degree; Equipped with the movement control unit controls the movement of the moving body such that a plurality of intermediate points, which are points located on or near a route of the moving body from a starting point of the moving body to the destination point, among the plurality of points, satisfy a predetermined condition; the predetermined condition includes a condition that, in at least a portion of the route, a degree of similarity between the reference images corresponding to two adjacent relay points is greater than a degree of similarity between the reference images corresponding to two non-adjacent relay points; Control device. (Item 16) A moving body capable of moving autonomously, A control device according to any one of items 1 to 15, A drive unit that moves the moving body in accordance with an instruction from the control device; Equipped with Mobile body. (Item 17) A program for causing a computer to function as the control device according to any one of items 1 to 15. (Item 18) A route planning device that determines a planned route from a starting point to a destination point of a moving body that autonomously moves within a predetermined geographical range, the route planning device is configured to be able to access a storage device that stores a weighted graph including a plurality of nodes corresponding to a plurality of points disposed within the geographical range, and one or more edges connecting each of the plurality of nodes; In the weighted graph, a weight of each of the one or more edges indicates a degree of similarity between reference images, which are images captured in advance at two points corresponding to two nodes connected by each edge; The route planning device includes a route planning unit that determines the planned route, the route planning unit determines the planned route by referring to the storage device and deriving a node sequence that minimizes a sum of edge weights from a start node, which is a node corresponding to the starting point or a point located in the vicinity of the starting point, to an end node, which is a node corresponding to the destination point or a point located in the vicinity of the destination point; Path planning device. (Item 19) obtaining a training dataset including a plurality of training data; generating a feature output model for inputting an image and outputting a value related to an estimation result of a feature of the image by performing metric learning using the learning dataset; having Each of the plurality of learning data is data in which a reference image, one or more similar images, and one or more dissimilar images are associated with each other; a relationship between a geographical location where each of the one or more similar images was captured and a geographical location where the reference image was captured satisfies a predetermined condition; a relationship between a geographical point where each of the one or more dissimilar images was captured and a geographical position where the reference image was captured does not match the predetermined condition; A method for generating a trained model. (Item 20) The predetermined conditions are: The geographical distance between the locations at which the images were taken is equal to or smaller than a first predetermined threshold value, and / or a condition that the length of the period between the times at which the images were captured is equal to or less than a second threshold value that is determined in advance; Including, 20. A method for generating a trained model according to item 19. (Item 21) the feature output model includes a neural network; The step of generating the feature output model by performing metric learning includes: generating feature vectors from the reference image, the one or more similar images, and the one or more dissimilar images included in each learning data; deriving a distance between feature vectors for each of a set of the reference image and the one or more similar images included in each learning data, and a set of the reference image and the one or more dissimilar images included in each learning data; performing metric training of the neural network using a loss derived based on the metric and a predetermined margin value; Including, 20. A method for generating a trained model according to item 19. (Item 22) The step of obtaining the training data set includes generating the training data set; The step of generating the learning dataset includes: acquiring image data of a plurality of images captured within a predetermined geographical range in association with auxiliary data indicating a geographical position at which each of the plurality of images was captured or a time at which each of the plurality of images was captured; determining, for each of the plurality of images, whether or not a relationship between a geographical position at which each image was captured and a geographical position at which at least one other image included in the plurality of images was captured satisfies the predetermined condition, based on the geographical position or the time indicated by the auxiliary data associated with the image data of the each image; determining the other images, the relationship of which with the geographical location at which each image was captured being determined to match the predetermined condition, as the one or more similar images corresponding to each image; determining the other images, the relationship of which to a geographical location at which each image was captured being determined not to match the predetermined condition, as the one or more dissimilar images corresponding to each image; Including, 20. A method for generating a trained model according to item 19. (Item 23) 1. A method for generating a dataset for use in machine learning or metric learning, comprising: acquiring image data of a plurality of images captured within a predetermined geographical range in association with auxiliary data indicating a geographical position at which each of the plurality of images was captured or a time at which each of the plurality of images was captured; determining whether or not a relationship between a geographical location at which each of the plurality of images was captured and a geographical location at which at least one other image included in the plurality of images was captured satisfies a predetermined condition; determining the other images, the relationship of which with the geographical location at which each image was captured being determined to match the predetermined condition, as similar images that are positive examples corresponding to each image; determining the other images, the relationship of which with respect to the geographical location at which each image was captured being determined not to match the predetermined condition, as dissimilar images that are negative examples corresponding to each image; generating the dataset by associating each of the plurality of images with a corresponding positive example and a corresponding negative example for each of the plurality of images; having The predetermined conditions are: A condition that a geographical distance between a geographical location where each of the images was captured and a geographical location where the other image corresponding to each of the images was captured is equal to or smaller than a first threshold value that is determined in advance, and / or a condition that the length of a period between a time when each of the images was captured and a time when the other image corresponding to each of the images was captured is equal to or less than a predetermined second threshold value or is smaller than the second threshold value; Including, A method for generating the dataset. [Explanation of symbols]
[0198] 10 Management area, 12 Driving permitted area, 14 Driving prohibited area, 20 Reference point, 22 Reference image, 32 Image, 34 Feature amount, 42 Target image, 44 Current image, 46 Estimation result, 52 Feature amount, 54 Feature amount, 100 Mobile object management system, 110 Autonomous mobile object, 112 Camera, 114 Control unit, 120 Management server, 300 Conversion model, 400 Control amount estimation model, 420 Feature amount extraction unit, 430 First extraction unit, 432 First extraction model, 440 Second extraction unit, 442 Second extraction model, 460 Relative index estimation unit, 462 Fully connected layer, 464 Fully connected layer, 466 Fully connected layer, 622 Sense unit, 624 Communication unit, 626 Input / output unit, 628 Drive unit, 630 Data storage unit, 642 Data collection unit, 644 Communication control unit, 646 Input / output control unit, 648 drive control unit, 650 route guidance unit, 720 global route planning unit, 740 local route planning unit, 742 route information acquisition unit, 744 destination point determination unit, 746 target image acquisition unit, 752 current image acquisition unit, 754 control amount determination unit, 812 reference information storage unit, 814 topological map storage unit, 822 feature vector generation unit, 824 similarity derivation unit, 842 user input acquisition unit, 844 destination image acquisition unit, 846 end node determination unit, 864 departure image acquisition unit, 866 start node determination unit, 880 global route determination unit, 922 control amount derivation unit, 924 control amount correction unit, 1020 data storage unit, 1030 topological map generation unit, 1040 imaging data acquisition unit, 1042 conversion model teacher data generation unit, 1044 A unit for generating teacher data for an estimation model, 1052, a unit for generating a conversion model, 1054, a unit for generating a control amount estimation model, 1120, a unit for extracting captured image data, 1130, a unit for judging, 1142, a unit for determining a similar image, 1144, a unit for determining a dissimilar image, 1146, a unit for generating a data set, 3000, a computer, 3001, a DVD-ROM, 3010, a host controller, 3012, a CPU, 3014, a RAM, 3016, a graphics controller, 3018, a display device, 3020, an input / output controller, 3022, a communication interface, 3024, a hard disk drive, 3026, a DVD-ROM drive, 3030, a ROM, 3040, an input / output chip, 3042, a keyboard
Claims
1. A control device for controlling a moving object that autonomously moves within a predetermined geographical area, the moving body includes a first imaging device that captures an image of the surroundings of the moving body; the control device is configured to be able to access a first storage device that stores a plurality of reference images, which are images captured in advance at a plurality of points disposed within the geographical range, in association with identification information of the plurality of points; The control device includes: a route planning unit that determines a planned route from a starting point of the moving object to a destination point; a route information acquisition unit that acquires route information indicating one or more relay points that are one or more points constituting the planned route among the plurality of points; a destination point determination unit that determines a destination point that is a destination of the moving body from among the one or more relay points; a target image acquisition unit that refers to the first storage device and acquires a target image that is a reference image associated with the target location from among the plurality of reference images; a current image acquisition unit that acquires a current image that is an image captured by the first imaging device at a current position and orientation of the moving object; a relative index derivation unit that derives a value of a relative index, which is an index indicating a relative relationship between a current position and orientation of the moving object and a position and orientation of the moving object that is a target at the target point, based on the target image and the current image; Equipped with The relative index derivation unit (i) inputs the target image and the current image into a trained estimation model, and (ii) derives an output corresponding to an estimation result regarding the relative index from the estimation model; the route planning unit refers to a second storage device that stores a weighted graph including a plurality of nodes corresponding to the plurality of points and one or more edges connecting the plurality of nodes, and determines the planned route by deriving a node sequence that minimizes a sum of edge weights from a start node that is a node corresponding to the departure point or a point located in the vicinity of the departure point to an end node that is a node corresponding to the destination point or a point located in the vicinity of the destination point; In the weighted graph, a weight of each of the one or more edges indicates a degree of similarity between the reference images of two points corresponding to each of the two nodes connected by each edge. Control device.
2. The estimation model is a feature extraction unit that receives the target image and the current image as inputs and extracts features of the target image and features of the current image; a relative index estimation unit that receives the feature amount of the target image and the feature amount of the current image as input and outputs the estimation result regarding the relative index; having The control device according to claim 1 .
3. The feature extraction unit is a first extraction unit that inputs the target image and extracts a feature amount of the target image; a second extraction unit that receives the current image and extracts a feature amount of the current image; Including, The first extraction unit (i) inputs the target image to a trained first extraction model, and (ii) derives an output corresponding to an estimation result regarding a feature amount of the target image from the first extraction model; The second extraction unit (i) inputs the current image to a trained second extraction model, and (ii) derives an output corresponding to an estimation result regarding a feature amount of the current image from the second extraction model. The first extraction model and the second extraction model are trained models generated by distance learning using an image as input and a feature vector of the image as output. The control device according to claim 2.
4. a first data set acquisition unit that acquires a first training data set including a plurality of first training data; an estimation model generation unit that generates the trained estimation model by performing machine learning using a combination of two pieces of first learning data included in the first learning dataset; Further equipped with each of the first learning data is data in which an image is associated with an imaging position, which is a position of a second imaging device that captured the image, and an imaging orientation, which is an orientation of the second imaging device; the estimation model generation unit generates the estimation model by machine learning, for each of the plurality of combinations, using as input a first image included in one of the two first learning data sets and a second image included in the other of the two first learning data sets, and outputting a value related to an estimation result of a relative relationship between an imaging position and an imaging orientation corresponding to the first image and an imaging position and an imaging orientation corresponding to the second image. The control device according to claim 1 .
5. a movement control unit that moves the moving object based on the relative index derived by the relative index derivation unit, The control device according to claim 1 .
6. a correction unit that estimates a current position of the moving object by a particle filter or a sequential Monte Carlo method, and corrects a value of the relative index based on the estimated current position; The movement control unit moves the moving object based on the value of the relative index corrected by the correction unit. The control device according to claim 5.
7. a departure point image acquisition unit that acquires a departure point image that is an image captured by the first imaging device at the departure point; a similarity deriving unit that derives a degree of similarity between each of the plurality of reference images and the departure location image; a start node determination unit that determines a node corresponding to a reference image that is most similar to the start image as the start node; Further comprising: The control device according to claim 1 .
8. Further, an end node determination unit is provided for determining the end node based on an input from a user. The control device according to claim 1 .
9. A moving body capable of moving autonomously, A control device according to any one of claims 1 to 8; A drive unit that moves the moving body in accordance with an instruction from the control device; Equipped with Mobile body.
10. A program for causing a computer to function as the control device according to any one of claims 1 to 8.
11. A route planning device that determines a planned route from a starting point to a destination point of a moving body that autonomously moves within a predetermined geographical range, the route planning device is configured to be able to access a storage device that stores a weighted graph including a plurality of nodes corresponding to a plurality of points located within the geographical range and one or more edges connecting each of the plurality of nodes; In the weighted graph, a weight of each of the one or more edges indicates a degree of similarity between reference images, which are images captured in advance at two points corresponding to two nodes connected by each edge; The route planning device includes a route planning unit that determines the planned route, the route planning unit determines the planned route by referring to the storage device and deriving a node sequence that minimizes a sum of edge weights from a start node, which is a node corresponding to the departure point or a point located in the vicinity of the departure point, to an end node, which is a node corresponding to the destination point or a point located in the vicinity of the destination point; Path planning device.
12. obtaining a training dataset including a plurality of training data; generating a feature output model for inputting an image and outputting a value related to an estimation result of a feature of the image by performing metric learning using the learning dataset; having Each of the plurality of learning data is data in which a reference image, one or more similar images, and one or more dissimilar images are associated with each other; a relationship between a geographical location where each of the one or more similar images was captured and a geographical location where the reference image was captured satisfies a predetermined condition; a relationship between a geographical point at which each of the one or more dissimilar images was captured and a geographical position at which the reference image was captured does not satisfy the predetermined condition; A method for generating a trained model.
13. The predetermined condition is: The geographical distance between the locations where the images were taken is equal to or smaller than a first predetermined threshold value; and / or a condition that the length of the period between the times at which the images were captured is equal to or less than a predetermined second threshold value; Including, The method of claim 12.
14. the feature output model includes a neural network; The step of generating the feature output model by performing metric learning includes: generating feature vectors from the reference image, the one or more similar images, and the one or more dissimilar images included in each learning data; deriving a distance between feature vectors for each of a set of the reference image and the one or more similar images included in each learning data, and a set of the reference image and the one or more dissimilar images included in each learning data; performing metric training of the neural network using a loss derived based on the metric and a predetermined margin value; Including, The method of claim 12.
15. The step of obtaining the training data set includes generating the training data set; The step of generating the training data set includes: acquiring image data of a plurality of images captured within a predetermined geographical range in association with auxiliary data indicating a geographical position at which each of the plurality of images was captured or a time at which each of the plurality of images was captured; determining whether or not a relationship between a geographical location at which each of the plurality of images was captured and a geographical location at which at least one other image included in the plurality of images was captured satisfies the predetermined condition based on the geographical location or the time indicated by the auxiliary data associated with the image data of each of the plurality of images; determining the other images, the relationship of which with the geographical location at which each image was captured being determined to match the predetermined condition, as the one or more similar images corresponding to each image; determining the other images, the relationship of which to a geographical location where each image was captured does not match the predetermined condition, as the one or more dissimilar images corresponding to each image; Including, The method of claim 12.
Citation Information
Cited By
Mutually exclusive node pair judgment method, processor, storage medium and automatic transportation system
CN120087725A
Additional learning method and additional learning system
WO2026181139A1
