Motion prediction device, motion prediction method, and motion prediction program

Abstract

The object is to provide a technology that can change the structure of a sensor without changing a model used in machine learning. A movement prediction device includes: an information acquisition unit that acquires moving body information including position information of a moving body; an overhead image generation unit that generates an overhead image that illustrates the positional relationship of the moving body by a moving body graph that represents the moving body based on the moving body information; and a moving body position prediction unit. The moving body position prediction unit uses machine learning on a time series of overhead images to predict the future position of the moving body.

Description

Technical Field

[0001] The present disclosure relates to a movement prediction device, a movement prediction method, and a movement prediction program. Background Art

[0002] In an environment where humans and autonomous mobile robots (hereinafter sometimes collectively referred to as mobile bodies) coexist, the autonomous mobile robots need to move safely and efficiently in a manner that avoids colliding with or tracking other mobile bodies other than the autonomous mobile robots. To achieve this, the system controlling the autonomous mobile robot needs to make the movement of the autonomous mobile robot more efficient by measuring and predicting the movement of other mobile bodies existing around the autonomous mobile robot.

[0003] In addition, machine learning is sometimes used to predict the complex movement of other moving objects such as people. For example, Patent Document 1 describes a technology for detecting other moving objects or predicting the movement of other moving objects by performing machine learning using measurement information measured by sensors mounted on an autonomous mobile robot as input.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2020-135901 Summary of the invention

[0007] However, in the technology of Patent Document 1, machine learning is performed substantially as is with respect to the measurement information provided by the sensor of the autonomous mobile robot. Therefore, there is a problem that it is difficult to change the type of sensor or change the structure of the sensor such as adding an external sensor without changing the model used in the machine learning.

[0008] Therefore, the present disclosure has been made in view of the above-mentioned problems, and an object of the present disclosure is to provide a technology that can change the structure of a sensor without changing a model used in machine learning.

[0009] The movement prediction device involved in the present disclosure comprises: an information acquisition unit, which acquires moving body information including position information of the moving body; a bird's-eye view image generation unit, which generates a bird's-eye view image that illustrates the positional relationship of the moving body by a graphic representing the moving body, i.e., a moving body graphic, based on the moving body information; and a moving body position prediction unit, which uses machine learning on the time series of the bird's-eye view image to predict the future position of the moving body.

[0010] According to the present disclosure, an overhead image showing the positional relationship of the moving object is generated based on the moving object information, and machine learning is used to predict the future position of the moving object on the time series of the overhead image. According to such a structure, the structure of the sensor can be changed without changing the model used in the machine learning.

[0011] The objects, features, aspects and advantages of the present disclosure will become more apparent from the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a system configuration diagram showing the configuration of the movement prediction system according to the first embodiment.

[0013] Figure 2 This is a block diagram showing the structure of the movement prediction device involved in Embodiment 1.

[0014] Figure 3 This is a diagram for explaining an example of a bird's-eye view image according to the first embodiment.

[0015] Figure 4 This is a diagram for explaining an example of a process of generating an overhead image according to the first embodiment.

[0016] Figure 5 This is a diagram schematically showing an example of the operation of the motion prediction model involved in Implementation Example 1.

[0017] Figure 6 This is a diagram for explaining an example of a bird's-eye view image according to the first embodiment.

[0018] Figure 7 This is a flowchart showing the operation of the movement prediction device involved in Embodiment 1.

[0019] Figure 8 This is a system configuration diagram showing the configuration of a movement prediction system according to the second embodiment.

[0020] Fig. 9 This is a block diagram showing the structure of a movement prediction device according to the second embodiment.

[0021] Fig.10 This is a diagram for explaining an example of a bird's-eye view image according to the second embodiment. DETAILED DESCRIPTION

[0022] <Implementation Method 1>

[0023] Figure 1: is a system structure diagram showing the structure of the movement prediction system involved in the present embodiment 1. The movement prediction system is a system that supports the movement of the autonomous mobile robot by measuring and predicting the movement of the mobile bodies around the autonomous mobile robot. The movement prediction system performs various processes on the mobile bodies existing in the space of the object for which the movement prediction is implemented (hereinafter referred to as the object space). For example, the movement prediction system measures the movement state of the mobile body, or uses the measurement result of the mobile body as input to predict the future position of the mobile body, or provides movement prediction information to the autonomous mobile robot and the system that controls the autonomous mobile robot based on the prediction result. In addition, in this specification, the movement prediction of the mobile body and the position prediction of the mobile body are substantially the same.

[0024] The movement prediction system comprises: ranging sensors L1 and L2 for measuring the movement state of a moving object in a target space; and a movement prediction device 1 for predicting the movement of the moving object in the target space based on the measurement results of the ranging sensors L1 and L2. Figure 1 In the object space, there are moving bodies M1 to M5 moving in the object space. The moving body is an object whose movement is predicted by the movement prediction device 1, and is an object moving in the object space. For example, the moving body includes a person, an animal, a vehicle, an autonomous mobile robot, etc. The number of the moving bodies can be one or more.

[0025] exist Figure 1 In the example, the moving bodies M1 to M5 include the autonomous mobile robots M1 and M2. In the first embodiment, the moving bodies M2 to M5 are other moving bodies of the autonomous mobile robot M1, and the moving bodies M1, M3 to M5 are other moving bodies of the autonomous mobile robot M2. The following mainly describes the autonomous mobile robot M1, but the following description is the same as for the autonomous mobile robot M2.

[0026] The autonomous mobile robot M1 is a type of mobile body, and is a mobile device that autonomously determines the surrounding conditions and moves from a starting point to a destination. The autonomous mobile robot M1 includes, for example, unmanned robots that provide services such as delivery, security, and cleaning, and personal mobility that carries people and performs autonomous driving.

[0027] The autonomous mobile robot M1 only needs to have the function of autonomously moving. For example, the autonomous mobile robot M1 includes a communication unit, a sensor unit, a control unit, and a drive unit, which are not shown in the figure. The communication unit communicates information with the movement prediction device 1, and the sensor unit measures the conditions around the autonomous mobile robot M1 or measures the position of the autonomous mobile robot M1. The control unit controls the movement of the autonomous mobile robot M1 according to the measurement results of the sensor unit, and the drive unit moves the autonomous mobile robot M1 according to the control of the control unit.

[0028] The autonomous mobile robot M1 moves to avoid collision with other mobile bodies or smoothly tracks the movement of other mobile bodies based on the movement prediction information obtained through communication with the movement prediction device 1. Such movement is achieved by an application that is mounted on the autonomous mobile robot M1 and controls the movement of the autonomous mobile robot M1. The autonomous mobile robot M1 can, for example, plan the movement path of other mobile bodies through sparse areas, or adjust the movement direction and movement speed of the autonomous mobile robot M1 to avoid collision between the autonomous mobile robot M1 and other mobile bodies.

[0029] The movement prediction device 1 and the autonomous mobile robot M1 implement information communication using a wireless communication method. The wireless communication method is a method that allows the autonomous mobile robot M1 to exchange information with the movement prediction device 1 during movement. For example, wide-area radio wave communications such as LTE (Long Term Evolution) and WiMAX (World Interoperability For Microwave Access) can be used, as well as narrow-band radio wave communications such as 5G (5 Generation), wireless LAN (Local Area Network), Bluetooth (registered trademark), and DSRC (Dedicated Short Range Communication). Communication methods that use light or sound other than radio waves can also be used. In addition, the movement prediction device 1 and the autonomous mobile robot M1 do not need to communicate directly, but can also communicate via an information communication network.

[0030] The mobile bodies M3 to M5 are mobile bodies other than the autonomous mobile robots M1 and M2 supported by the movement prediction device 1, such as humans, animals, and vehicles. Figure 1 In FIG. 1 , people are shown as the moving objects M3 to M5 , and as an example thereof, the shapes of the heads and shoulders of the people as viewed from above are shown.

[0031] <Distance measuring sensors L1, L2>

[0032] Hereinafter, the distance measuring sensor L1 will be mainly described, but the distance measuring sensor L2 is also the same as the following description.

[0033] The distance measuring sensor L1 is a measuring device that measures the object space, measures the movement state including the presence and position of the moving body according to the measurement result, and provides the movement state as moving body information to the movement prediction device 1. The distance measuring sensor L1 is set in a place where the moving body existing around the distance measuring sensor L1 can be measured, such as the ceiling of a building, the wall of a building, a telephone pole, a street lamp, etc. The distance measuring sensors L1 and L2 can be either one or three or more.

[0034] Here, the mobile body information is information indicating the measurement result including the position information of the mobile body. The position information is information indicating the location of the mobile body. The position information can be a location represented by a point or an area represented by a space. The position information can be expressed in a form that can be transformed by the movement prediction system, for example, in the form of latitude and longitude representing the absolute position coordinates on the earth, in the form of relative coordinates representing the position coordinates relative to a specific location, in the form of a code representing a location or area by encoding, etc.

[0035] The moving object information may include attribute information of the moving object detected by the ranging sensor L1. The attribute information is information indicating at least one of the type of the moving object and the moving state. The type of the moving object is information indicating the static classification of the moving object, such as adults, children, elderly people, animals, bicycles, wheelchairs, autonomous mobile robots, and others. The moving state is information indicating the dynamic classification of the moving object, such as stopping, moving, running, walking while looking at a smartphone, pulling a suitcase, pushing a stroller, etc.

[0036] The distance measuring sensor L1 only needs to have a function of providing the moving body information to the movement prediction device 1. For example, the distance measuring sensor L1 includes: a measuring unit that measures the moving body; an estimation unit that detects the moving body based on the measurement result of the measuring unit and generates the moving body information; and a communication unit that sends the moving body information to the movement prediction device 1, which are not shown in the figure.

[0037] The measuring unit of the distance measuring sensor L1 is a processing unit that obtains a measurement value for estimating the position information of the moving object, such as the distance between the distance measuring sensor L1 and the moving object. The measuring unit of the distance measuring sensor L1 may be, for example, LiDAR (Light Detection and Ranging), a stereo camera, a ToF (Time Of Flight) camera, a monocular camera with a distance measurement function, or the like.

[0038] The estimation unit of the distance measuring sensor L1 is a processing unit that generates information about a moving body by detecting the presence of a moving body or estimating the position information of a moving body based on the measurement value obtained by the measuring unit of the distance measuring sensor L1. The estimation unit of the distance measuring sensor L1 may use, for example, a method that can detect the moving body and estimate the position information of the moving body in substantially real time within a range that does not affect the system. The method for detecting a moving body may be, for example, a method for detecting a moving body by comparing a threshold value indicating the absence of a moving body with the measurement value obtained by the measuring unit, or a method for detecting a moving body by detecting a predetermined feature of the moving body based on the measurement value. In addition, the method for estimating the position information of a moving body may be, for example, a method for estimating the position information based on the relative distance between the moving body and the distance measuring sensor L1, or a method for estimating the position information using machine learning on the measurement value.

[0039] The estimation unit of the distance sensor L1 can estimate the attribute information of the moving object based on the measurement value obtained by the measuring unit of the distance sensor L1, and generate the moving object information including the attribute information. The estimation method of the attribute information can be, for example, a method of estimating the attribute information by detecting a predetermined feature point based on the measurement value, or a method of directly estimating the attribute information using a classifier that learns the correspondence between the measurement value and the attribute information.

[0040] The communication unit of the distance sensor L1 provides the moving body information estimated by the estimation unit of the distance sensor, that is, the moving body information measured by the distance sensor L1, to the processing unit of the movement prediction device 1. The distance sensor L1 transmits and receives the moving body information with the movement prediction device 1 through information communication using a wired communication method or a wireless communication method. The distance sensor L1 can communicate with the movement prediction device 1 substantially in real time within a range that does not affect the system. In addition, the distance sensor L1 does not necessarily need to communicate directly with the movement prediction device 1, and can also communicate indirectly with the movement prediction device 1 via an information communication network.

[0041] <Movement prediction device 1>

[0042] Figure 2 1 is a block diagram showing the structure of a movement prediction device 1 according to the first embodiment. The movement prediction device 1 is an information processing device for predicting the movement of a moving object. For example, the movement prediction device 1 obtains moving object information from one or more distance measuring sensors, or predicts the future position of the moving object based on the moving object information, or provides the movement prediction information to one or more autonomous mobile robots based on the prediction result. The one or more distance measuring sensors are, for example, distance measuring sensors L1 and L2, and the autonomous mobile robots are, for example, autonomous mobile robots M1 and M2.

[0043] The movement prediction device 1 includes: a moving body information acquisition unit 11 as an information acquisition unit, which acquires moving body information from ranging sensors L1 and L2; a movement prediction unit 10, which predicts the future position of the moving body based on the moving body information; and a movement prediction information providing unit 14, which provides movement prediction information to autonomous mobile robots M1 and M2.

[0044] In addition, the movement prediction device 1 is, for example, a computer including a storage device for storing each unit as a program, a CPU (Central Processing Unit) for executing each unit, a memory for temporarily holding the results calculated by the CPU and the program executed by the CPU, and an interface for communicating with an autonomous mobile robot and a ranging sensor. In addition, the components of the movement prediction device 1 can also be implemented using a processing circuit (Processing Circuitry) and the like.

[0045] The moving body information acquisition unit 11 acquires moving body information provided by the ranging sensors L1 and L2 by performing information communication with the ranging sensors L1 and L2, and provides the moving body information to the processing unit of the movement prediction unit 10. The moving body information acquisition unit 11 can analyze the communication message sent from the ranging sensors L1 and L2, or construct the moving body information based on the communication message, or convert the expression form of the moving body information, or notify the movement prediction unit 10 of the moving body information.

[0046] The movement prediction unit 10 is an information processing unit that receives the moving object information acquired by the moving object information acquisition unit 11 as input, generates a bird's-eye view image showing the positional relationship of the moving object, and predicts the future position of the moving object based on the time series of the bird's-eye view image.

[0047] The movement prediction unit 10 includes an overhead image generation unit 12 and a moving object position prediction unit 13. The overhead image generation unit 12 generates an overhead image that illustrates the positional relationship of the moving object by a moving object graph that is a graph representing the moving object, based on the moving object information acquired by the moving object information acquisition unit 11. The moving object position prediction unit 13 generates a future overhead image by using machine learning for the time series of the overhead images generated by the overhead image generation unit 12, thereby predicting the future position of the moving object.

[0048] In the first embodiment, the movement prediction unit 10 further includes a feature information storage unit 15 for storing feature information including position information of features. Features include at least one of real features that are physical in the target space and virtual features that are not physical but are virtually set in the target space. Examples of real features are pillars, walls, vending machines, trash cans, benches, fire extinguishers, etc. Examples of virtual features are shops, roads, squares, sidewalks, etc.

[0049] The feature information stored in the feature information storage unit 15 is information including the position information of the feature, for example, predetermined information that can be used to graphically represent the feature existing in the target space. The representation form of the feature position information is the same as the representation form of the position information of the moving object. In addition, the feature information may also include attribute information indicating the type of the feature. The type of the feature is information indicating the static classification of the feature, for example, a column, a wall, a store, etc.

[0050] The bird's-eye view image generation unit 12 generates a bird's-eye view image showing the positional relationship of the moving object by projecting the position of the moving object indicated by the moving object information acquired by the moving object information acquisition unit 11 onto a plane and drawing a moving object graph at the position, and supplies the bird's-eye view image to the processing unit of the moving object position prediction unit 13. When the moving prediction unit 10 includes the feature information storage unit 15, the bird's-eye view image generation unit 12 can generate a bird's-eye view image showing the positional relationship of the features by a feature graph, that is, a graph showing the features, based on the feature information stored in the feature information storage unit 15. In this case, in the bird's-eye view image, the positional relationship of the moving object is shown by the moving object graph, and the positional relationship of the features is shown by the feature graph.

[0051] The bird's-eye view image generated by the bird's-eye view image generation unit 12 is an image expressed by a collection of pixels, and is an image that illustrates a moving body figure or a ground object figure in a manner that allows the positional relationship of the moving body or the ground object to be understood. The bird's-eye view image includes, for example, an image of a top view. When the moving body figure is expressed by a location, it may be a simplified figure centered at the location. When the moving body position information is expressed by an area, it may be a detailed figure including the area. The ground object figure may be a simplified figure like the moving body figure, or a detailed figure including the area.

[0052] In the first embodiment, in the bird's-eye view image generated by the bird's-eye view image generation unit 12, a value specific to the moving object or the ground object (hereinafter referred to as a pixel specific value) is set for the pixel representing the moving object figure or the ground object figure. The pixel specific value may be expressed as "10" and the pixel specific value of the ground object may be expressed as "20" in the machine learning described later. In addition, when the moving object information includes the attribute information of the moving object, the bird's-eye view image generation unit 12 may determine the pixel specific value of the moving object for distinguishing the moving object figure in the machine learning based on the attribute information. Similarly, when the ground object information includes the attribute information of the ground object, the bird's-eye view image generation unit 12 may determine the pixel specific value of the ground object for distinguishing the ground object figure in the machine learning based on the attribute information. For example, the pixel specific value of a normal moving person may be expressed as "12", the pixel specific value of a moving person while looking at a smartphone may be expressed as "13", the pixel specific value of a wall may be expressed as "20", and the pixel specific value of a column may be expressed as "22".

[0053] In the first embodiment, the bird's-eye view image generated by the bird's-eye view image generation unit 12 not only illustrates the moving body figure and the ground object figure, but also illustrates the free area where there is no moving body or ground object. In other words, the free area is an area representing a space where the moving body can move freely. For example, a pixel intrinsic value indicating that it is a free area can be set for a pixel where there is no moving body or ground object. The pixel intrinsic value of the free area can be set to "0", for example.

[0054] For example, the rendering range, number of pixels and aspect ratio of the bird's-eye image generated by the bird's-eye image generating unit 12 are set in accordance with the rendering range, number of pixels and aspect ratio of the bird's-eye image predicted by the moving object position predicting unit 13 .

[0055] Figure 3 1 is a diagram for explaining an example of an overhead image generated by the overhead image generating unit 12 . Figure 3 (a) is a diagram showing an overhead image F1, which is generated by the overhead image generation unit 12 based on the moving body information provided by the distance measuring sensors L1 and L2, and which shows the vicinity of the point where two roads intersect in the building. The overhead image F1 shows moving bodies M1 to M5, floor features W1 to W5, and a vacant area R1. The moving bodies M1 to M5 are represented by the moving state information obtained by the moving body information acquisition unit 11, and are moving bodies existing within the range where the overhead image is generated. The floor features W1 to W5 are represented by the floor feature information stored in the floor feature information storage unit 15, and are floor features existing within the range where the overhead image is generated.

[0056] Figure 3 (b) is about Figure 3 (a) A table showing "position information", "attribute information" and "pixel intrinsic values" for each mobile body and each ground object shown in the figure. The "position information" in the figure includes a "category" indicating the form of expression of the position information and a "coordinate" indicating the location of the position information. Figure 3 In the example of (b), the "category" of the location information is set to be either a "location" indicating a point or an "area" indicating a range. In addition, the "attribute information" in the figure includes a "category" indicating the type of the moving body or the feature on the ground, and a "pattern" indicating the movement pattern of the moving body.

[0057] The "pixel inherent value" in the figure is determined by the combination of "type" and "state" of "attribute information". For example, as the pixel inherent value, "10" is set for the group of "autonomous mobile robot" and "moving", "12" is set for the group of "person" and "moving", and "13" is set for the group of "person" and "moving while looking at a smartphone". In the case where there is no moving state, such as in the ground feature information, the "pixel inherent value" is determined by the "type". For example, as the pixel inherent value, "20" is set for "wall", "21" is set for "store", and "22" is set for "pillar". Furthermore, as the pixel inherent value, "0" is set for an empty area where there is no moving object or ground feature.

[0058] Figure 4 1 is a diagram illustrating an example of a generation process of an overhead image generated by the overhead image generating unit 12 .

[0059] First, the bird's-eye view image generating unit 12 generates a bird's-eye view image in an initial state. Figure 4 (a) is a diagram showing an initial state bird's-eye view image F1a. The initial state bird's-eye view image F1a is an image in which an invalid value or a free area is set for a pixel-specific value.

[0060] Next, the bird's-eye view image generating unit 12 draws the graphics of the features superimposed on the bird's-eye view image in the initial state. Figure 4 (b) is a diagram showing an overhead image F1b on which a ground feature graphic is superimposed. Figure 4 The process of overlapping and drawing the floor feature figures in (b) will be described separately as first to third processes.

[0061] In the first step of overlapping and drawing the feature figures, the bird's-eye view image generation unit 12 extracts the feature information within the prediction target range from the feature information stored in the feature information storage unit 15. The prediction target range is a range shown in the bird's-eye view image, and is pre-set to any one of 10m×10m to 100m×100m, for example. When extracting the feature information, the bird's-eye view image generation unit 12 extracts the feature information within the prediction target range by determining whether the position information of the feature information is within the prediction target range.

[0062] In the second step of overlapping and drawing the figure of the feature, the bird's-eye view image generation unit 12 performs coordinate transformation from the coordinate system of the extracted feature information to the coordinate system of the bird's-eye view image. The coordinate system of the bird's-eye view image may be a coordinate system that can graphically show the relative positional relationship of the moving body or the feature, and is defined by the orientation, size, and pixel density of the bird's-eye view image that are set in advance. In the coordinate transformation, the bird's-eye view image generation unit 12 appropriately performs coordinate transformation processing including base transformation, enlargement and reduction, rotation, and parallel translation on the location or shape point included in the position information of the feature information, thereby transforming from the coordinate system of the feature information to the coordinate system of the bird's-eye view image.

[0063] In the third step of overlapping and drawing the feature graphics, the bird's-eye view image generating unit 12 overlaps and draws the feature graphics on the bird's-eye view image based on the feature information after coordinate transformation. For example, the bird's-eye view image generating unit 12 draws the feature graphics at the location or area where the feature exists in the bird's-eye view image. For example, if the location information of the feature information is a location, the feature graphics are drawn with a simple graphic, and if the location information of the feature information is an area, the feature graphics are drawn with a polygonal graphic.

[0064] In addition, the bird's-eye view image generating unit 12 determines the pixel-specific value of the ground feature figure. The pixel-specific value of the ground feature figure may be of one type or of a plurality of types.

[0065] When there are multiple types of pixel intrinsic values ​​of the ground object figure, the bird's-eye view image generating unit 12 determines the pixel intrinsic value of the ground object figure based on the attribute information of the ground object information. For example, the bird's-eye view image generating unit 12 determines whether the ground object is a ground object that a mobile body can pass through and whether it is a ground object that an autonomous mobile robot can pass through, and determines the pixel intrinsic value of the ground object figure based on the determination result of the ground object.

[0066] When there are multiple types of pixel eigenvalues ​​of the ground object graphics, the bird's-eye view image generating unit 12 determines the order in which the ground object graphics are drawn on the bird's-eye view image based on the result of the ground object determination. For example, the bird's-eye view image generating unit 12 draws the ground object graphics on the bird's-eye view image in the order of ground objects that can be passed by a moving body, ground objects that can be passed by an autonomous mobile robot, and ground objects that cannot be passed by a moving body.

[0067] When there are multiple types of pixel intrinsic values ​​for the ground object graphics, the bird's-eye view image generation unit 12 can determine the pixel intrinsic values ​​and the drawing order using a ground object drawing condition table that records in advance the correspondence between the attribute information of the ground object and the pixel intrinsic values ​​and the drawing order that should be set for the ground object graphics of the ground object.

[0068] The bird's-eye view image in which the graphics of the land features are drawn may be generated in advance. In such a case, the bird's-eye view image generating unit 12 may omit the step of superimposing and drawing the graphics of the land features described above.

[0069] Next, the bird's-eye view image generating unit 12 draws the moving object figure superimposed on the bird's-eye view image in which the ground object figure is drawn. Figure 4 (c) is a diagram showing an overhead image F1c on which a moving object figure is superimposed. Figure 4 The step of overlapping drawing of the moving body graphics in (c) will be described by dividing into the first to third steps.

[0070] In the first step of superimposing and drawing the moving body graphics, the bird's-eye view image generation unit 12 extracts the moving body information illustrated in the bird's-eye view image from the moving body information acquired by the moving body information acquisition unit 11. For example, the bird's-eye view image generation unit 12 extracts the moving body information within the prediction target range and the moving body information received within the prediction unit time. The prediction unit time indicates the step time for expressing one bird's-eye view image, and is pre-set to any one of about 0.1 seconds to 1.0 seconds, for example. In extracting the moving body information, the bird's-eye view image generation unit 12 determines whether the position information of the moving body information is within the prediction target range and whether it can be received within the prediction target time, and extracts the moving body information that exists within the prediction target range and can be received within the prediction target time.

[0071] In the second step of superimposing and drawing the moving body figure, the bird's-eye view image generation unit 12 performs coordinate transformation from the extracted moving body information to the coordinate system of the bird's-eye view image. In the coordinate transformation of the moving body information, the bird's-eye view image generation unit 12 performs the same coordinate transformation as the coordinate transformation of the feature information. That is, the bird's-eye view image generation unit 12 appropriately performs coordinate transformation processing including base transformation, enlargement and reduction, rotation, and parallel translation on the location or shape point of the position information included in the moving body information, thereby transforming from the coordinate system of the moving body information to the coordinate system of the bird's-eye view image.

[0072] In the third step of superimposing and drawing the moving body graph, the bird's-eye view image generation unit 12 superimposes and draws the moving body graph on the bird's-eye view image based on the moving body information after coordinate transformation. For example, the bird's-eye view image generation unit 12 draws the moving body graph at a location or region where the moving body exists in the bird's-eye view image. The moving body graph is drawn with a simple graph or a polygonal graph, similar to the ground object graph.

[0073] In addition, the bird's-eye view image generating unit 12 determines the pixel-specific value of the moving object pattern. The type of the pixel-specific value of the moving object pattern may be one or more.

[0074] When there are multiple types of pixel-specific values ​​of the moving object graphic, the bird's-eye view image generating unit 12 determines the pixel-specific value of the moving object graphic based on the attribute information of the moving object information. For example, the bird's-eye view image generating unit 12 determines the pixel-specific value corresponding to the combination of the unique attribute of the attribute information and the movement pattern as the pixel-specific value of the moving object graphic.

[0075] When there are multiple types of pixel intrinsic values ​​of the moving body graphics, the bird's-eye view image generation unit 12 can determine the pixel intrinsic value using a moving body rendering condition table that records in advance the attribute information of the moving body and the pixel intrinsic value to be set for the moving body graphics of the moving body in correspondence. In the moving body rendering condition table, for example, for the group of the type and state of "autonomous mobile robot" and "moving", "10" is associated as the pixel intrinsic value, and for the group of the type and state of "person" and "moving while looking at a smartphone", "13" is associated as the pixel intrinsic value.

[0076] The bird's-eye view image generating unit 12 provides a bird's-eye view image depicting the moving object to the Figure 2 The moving object position prediction unit 13.

[0077] The moving body position prediction unit 13 generates one or more future bird's-eye view images by inputting the time series of bird's-eye view images generated by the bird's-eye view image generation unit 12 into a moving prediction model that performs machine learning on the time series of bird's-eye view images, thereby predicting the future position of the moving body, and providing moving prediction information to the processing unit of the moving prediction information providing unit 14 based on the prediction result.

[0078] The motion prediction model is a model that generates one or more future overhead images based on the time series of overhead images by learning the time series of overhead images. The time series of overhead images can be a past time series of overhead images or a combination of a past time series of overhead images and a current overhead image. When the motion prediction model is input with a time series of overhead images of several seconds to several tens of seconds in the past, it outputs an overhead image of several seconds to several tens of seconds in the future.

[0079] The bird's-eye view image output from the motion prediction model may be a single bird's-eye view image that illustrates future positional relationships at a predetermined time, or may be a plurality of bird's-eye view images that illustrate future positional relationships in a time series. In addition, the representation form of the bird's-eye view image output from the motion prediction model may be different from the bird's-eye view image input to the motion prediction model. In the bird's-eye view image output from the motion prediction model, for example, the probability that a moving object or a ground object exists at a location represented by pixels (hereinafter sometimes referred to as the existence probability) may be set. Alternatively, in the bird's-eye view image output from the motion prediction model, for example, the pixel-specific value representing the moving object or ground object that has the highest probability of existing at a location represented by pixels may be set.

[0080] Figure 5 FIG. 1 is a diagram schematically showing an example of the operation of the movement prediction model 130. Figure 5 In the example of FIG. 1 , a time series of overhead images F11 to F19 from the past to the present is shown. Figure 3 The movement prediction model 130 outputs a future bird's-eye view image F2 when the time series of the bird's-eye view images F11 to F19 is input.

[0081] Figure 6 This is a diagram for explaining an example of a future bird's-eye view image F2 generated by the movement prediction model 130 . Figure 6 (a) is a diagram showing a future bird's-eye view image F2 generated from the time series of bird's-eye view images F11 to F19. Figure 6In the example of (a), a heat map showing the future existence probability of the moving bodies M1 to M5 is shown as a future bird's-eye view image F2, where the parts predicted to have a high existence probability are shown in a darker way, and the parts predicted to have a low existence probability are shown in a lighter way. For example, the moving body M5 moves from the bird's-eye view image F11 to the bird's-eye view image F19 toward the center of the intersection, so the existence probability of the moving body M5 becomes higher in the center of the intersection in the future bird's-eye view image F2. Figure 6 In (a), the moving objects M1 to M5 expressed by different pixel eigenvalues ​​are superimposed on the bird's-eye view image to be expressed. However, the probability of each pixel eigenvalue may be set as the eigenvalue probability for each coordinate of the pixels constituting the bird's-eye view image.

[0082] Figure 6 (b) is a table that records the probability of the intrinsic value of each pixel at each coordinate of the pixels constituting the bird's-eye view image F2 in the future. Figure 6 In the pixel with coordinates X=4.0, Y=6.0 in the overhead image of (b), the probability that nothing exists, represented by the pixel intrinsic value "0", is 0.1; the probability that a "moving autonomous mobile robot" exists, represented by the pixel intrinsic value "10", is 0.2; and the probability that a "person moving while looking at a smartphone" exists, represented by the pixel intrinsic value "13", is 0.7.

[0083] In the motion prediction model, a machine learning technique is used, such as predicting future image information based on past image information by learning the time series of image information. The machine learning used in the motion prediction model includes, for example, reinforcement learning that extracts the feature quantity of a preset bird's-eye view image to learn the weight of the feature quantity, or deep learning that also learns the extraction of the feature quantity itself. For example, a motion prediction model is realized by combining a CNN (Convoluted Neural Network) used in image recognition and an LSTM (Long-Short Term Memory) used in the processing of time series data.

[0084] The motion prediction model preferably learns the time series of the bird's-eye view images in advance. In the learning of the motion prediction model, a learning is implemented such that the time series of the bird's-eye view images generated by the bird's-eye view image generation unit 12 is used as learning data to improve the accuracy of future bird's-eye view images, that is, the accuracy of motion prediction. For example, first, the moving body position prediction unit 13 divides the time series of the bird's-eye view images before a certain time as input data and the time series of the bird's-eye view images after a certain time as training data. Next, the moving body position prediction unit 13 provides the input data to the motion prediction model and generates the future bird's-eye view images as output data. Next, the moving body position prediction unit 13 compares the training data and the output data to adjust the motion prediction model in such a way that the output data approaches the training data. The moving body position prediction unit 13 repeats this multiple times to make the motion prediction model learn the time series of the bird's-eye view images.

[0085] The moving object position prediction unit 13 provides the moving prediction information to the Figure 2 The moving object position prediction unit 13 may provide the future bird's-eye view image generated by the moving prediction model as the moving prediction information as it is, or may provide information obtained by processing or analyzing the future bird's-eye view image as the moving prediction information. For example, the moving object position prediction unit 13 may correct the image by using morphological transformation or the like for the future bird's-eye view image, or may estimate a location with a high probability of existence of the moving object in the future bird's-eye view image as the location of the moving object, or may perform a transformation opposite to the coordinate transformation of the bird's-eye view image generation unit 12, thereby generating the moving prediction information.

[0086] The movement prediction information providing unit 14 is a processing unit that provides the movement prediction information generated by the movement prediction unit 10 to the autonomous mobile robots M1 and M2 using a wireless communication method. The communication method of the movement prediction information providing unit 14 can be a method of periodically broadcasting and publishing from the movement prediction device 1 to the autonomous mobile robots M1 and M2, or a method of responding to the request when the autonomous mobile robots M1 and M2 request the movement prediction information.

[0087] <Action>

[0088] Figure 7 This is a flowchart showing the operation of the movement prediction device 1 according to the first embodiment.

[0089] First, in step S10, the moving body information acquisition unit 11 acquires moving body information. In the present embodiment 1, the moving body information acquisition unit 11 receives moving body information from a distance measuring sensor. After step S10 is completed, the process moves to step S11.

[0090] In step S11, the motion prediction unit 10 determines whether a prediction unit time, which is a unit time for expressing the bird's-eye view image, has passed. For example, the motion prediction unit 10 measures the time from the time when step S11 was last executed to the current time, and determines whether the time exceeds a preset prediction unit time. If it is determined that the prediction unit time has passed, the process moves to step S12, and if it is determined that the prediction unit time has not passed, the process moves to step S10.

[0091] In step S12, the bird's-eye view image generation unit 12 extracts the moving body information and the feature information required for generating the bird's-eye view image. For example, the bird's-eye view image generation unit 12 extracts the moving body information within the range shown in the bird's-eye view image from the moving body information acquired by the moving body information acquisition unit 11. In addition, the bird's-eye view image generation unit 12 extracts the feature information within the range shown in the bird's-eye view image from the feature information stored in the feature information storage unit 15. After step S12 is completed, the process moves to step S13.

[0092] In step S13, the bird's-eye view image generation unit 12 generates a bird's-eye view image. For example, the bird's-eye view image generation unit 12 generates a bird's-eye view image showing the positional relationship between the moving object and the feature information extracted in step S12. After step S13 is completed, the process moves to step S14.

[0093] In step S14, the moving object position prediction unit 13 determines whether the time series of the overhead image has been generated for the time required for the movement prediction. If it is determined that the time series of the overhead image has been generated for the required time, the process moves to step S15. If it is determined that the time series of the overhead image has not been generated for the required time, the process moves to step S10.

[0094] In step S15, the moving object position prediction unit 13 predicts the future position of the moving object based on the time series of the bird's-eye view image. In the first embodiment, the moving object position prediction unit 13 generates a future bird's-eye view image by inputting the time series of the bird's-eye view image into the movement prediction model, and predicts the future position of the moving object. After step S15 is completed, the process transfers to step S16.

[0095] In step S16, the movement prediction information providing unit 14 provides the autonomous mobile robot with movement prediction information according to the prediction result. After step S16 is completed, the process transfers to step S10.

[0096] In addition, step S10 and step S16 may be performed in parallel with other steps. For example, the movement prediction device 1 may perform a process of acquiring moving object information, a process of generating a bird's-eye view image to perform movement prediction, and a process of providing movement prediction information in parallel.

[0097] <Summary of Implementation Method 1>

[0098] According to the movement prediction device 1 involved in the present embodiment 1 as described above, a time series of bird's-eye view images showing the positional relationship of the moving body is generated based on the moving body information provided by the ranging sensor, and the future position of the moving body is predicted using machine learning on the time series of bird's-eye view images. According to such a structure, machine learning is performed on the bird's-eye view images whose versatility is improved by abstracting the positional relationship of the moving body, so that even if the model used in the machine learning is not changed, it is possible to change the type of sensor, add an external sensor, or change the structure of the sensor.

[0099] In addition, according to the first embodiment, the future position of the moving body is predicted using a bird's-eye view image showing the positional relationship between the moving body and the ground objects. According to such a structure, the positional relationship of the ground objects can also be abstracted using the bird's-eye view image, so that the ground objects can be easily added or changed without changing the movement prediction model. In addition, by considering the ground objects, the movement prediction can be performed for a state close to the real world, so the accuracy of the movement prediction can be improved.

[0100] In addition, according to the first embodiment, the pixel intrinsic value of the moving object used to distinguish the moving object pattern in machine learning is determined based on the attribute information of the moving object. According to such a structure, the moving object model can be learned for each type of moving object and each movement pattern, so the accuracy of movement prediction can be improved.

[0101] In addition, according to the first embodiment, the pixel intrinsic value of the moving body used to distinguish the ground object pattern in machine learning is determined based on the attribute information of the ground object. According to such a structure, the influence of the type of ground object on the moving body can be reflected in the learning of the moving body model, so the accuracy of movement prediction can be improved.

[0102] <Implementation Method 2>

[0103] Figure 8 1 is a system configuration diagram showing the configuration of the motion prediction system according to Embodiment 2. Hereinafter, among the components according to Embodiment 2, the same or similar components as the above components are denoted by the same or similar reference numerals, and mainly the different components are described.

[0104] Figure 8 The structure and Figure 1The structure is the same as that of deleting the distance measuring sensors L1 and L2 provided and adding distance measuring sensors L1B and L2B respectively mounted on the autonomous mobile robots M1B and M2B. Figure 8 In essence, as the distance measuring sensors L1 and L2 are changed to distance measuring sensors L1B and L2B, Figure 1 The movement prediction device 1 is changed to the movement prediction device 1B.

[0105] The distance measuring sensors L1B and L2B are substantially the same as the distance measuring sensors L1 and L2 of Embodiment 1 except that they are mounted on the autonomous mobile robots, and the autonomous mobile robots M1B and M2B are substantially the same as the autonomous mobile robots M1B and M2B of Embodiment 1. Hereinafter, the distance measuring sensor L1B and the autonomous mobile robot M1B will be mainly described, but the distance measuring sensor L2B and the autonomous mobile robot M2B are also the same as described below.

[0106] The autonomous mobile robot M1B is an autonomous mobile robot equipped with a distance measuring sensor L1B as a sensor unit described in Embodiment 1. The autonomous mobile robot M1B measures the target space using the distance measuring sensor L1B, thereby estimating the position information of the autonomous mobile robot M1B, or detecting a moving body existing around the autonomous mobile robot M1B, or estimating the position information of the moving body, or estimating the attribute information of the moving body. The method for estimating the position of the autonomous mobile robot M1B may be a method that can estimate the position of the autonomous mobile robot M1B in real time, for example, it may be estimated by SLAM (Simultaneous Localization And Mapping) using the distance measuring sensor L1B, or by a method using other sensor devices. The method for detecting the moving body existing around the autonomous mobile robot M1B, and the method for estimating the position information and attribute information may be the same as the detection method and the estimation method described in Embodiment 1.

[0107] The autonomous mobile robot M1B measures the target space using the distance measuring sensor L1B, thereby estimating the position information of the ground object or estimating the attribute information of the ground object. The detection method of the ground object existing around the autonomous mobile robot M1B and the estimation method of the position information and attribute information may be the same as the detection method and estimation method of the moving body existing around the autonomous mobile robot M1B.

[0108] The autonomous mobile robot M1B provides the information obtained by the distance measuring sensor L1B as the moving body information to the movement prediction device 1B. For example, the autonomous mobile robot M1B transmits the moving body information to the movement prediction device 1B using a communication unit of the autonomous mobile robot M1B.

[0109] The autonomous mobile robot M1B can generate measurement range information based on the measurement result of the ranging sensor L1B and provide it to the movement prediction device 1B. The measurement range information is information indicating the range in which the ranging sensor L1B measures the position information of the moving body and the position information of the ground object without being blocked by obstacles. Obstacles include at least one of the moving body and the ground object. The measurement range information can express the range measured by the ranging sensor L1B, for example, it can be the measurement value of the ranging sensor L1B itself, or it can be information that converts the visible range into a geometric figure.

[0110] <Movement Prediction Device 1B>

[0111] Fig. 9 1 is a block diagram showing the structure of a movement prediction device 1B according to the second embodiment. The movement prediction device 1B is an information processing device for predicting the movement of a mobile body, similarly to the movement prediction device 1 according to the first embodiment. The movement prediction device 1B includes: a mobile body information acquisition unit 11B as an information acquisition unit, which acquires mobile body information from autonomous mobile robots M1B and M2B; a movement prediction unit 10B, which predicts the future position of the mobile body based on the mobile body information; and a movement prediction information providing unit 14, which provides movement prediction information to the autonomous mobile robots M1B and M2B.

[0112] The moving body information acquisition unit 11B is a processing unit that acquires moving body information from the autonomous mobile robots M1B and M2B instead of acquiring moving body information from the distance measuring sensors L1 and L2 in the first embodiment, and provides the moving body information to the movement prediction unit 10B. The moving body information acquisition unit 11B receives moving body information of the autonomous mobile robots M1B and M2B and moving body information of moving bodies (such as the moving body M3) around the autonomous mobile robots M1B and M2B from the autonomous mobile robots M1B and M2B. In addition, the moving body information of the autonomous mobile robots M1B and M2B includes information for determining the absolute positions of the autonomous mobile robots M1B and M2B. Except for the above, the moving body information acquisition unit 11B may be the same as the moving body information acquisition unit 11 in the first embodiment.

[0113] In the second embodiment, the mobile body information acquisition unit 11B also acquires the above-mentioned measurement range information from the autonomous mobile robots M1B and M2B, and supplies the measurement range information to the movement prediction unit 10B.

[0114] The motion prediction unit 10B includes an overhead image generation unit 12B and a moving object position prediction unit 13B. The configuration of the motion prediction unit 10B other than these may be the same as that of the motion prediction unit 10 in the first embodiment.

[0115] The bird's-eye view image generation unit 12B is a processing unit that generates a bird's-eye view image based on the moving body information acquired by the moving body information acquisition unit 11B, similarly to the bird's-eye view image generation unit 12 according to Embodiment 1. The bird's-eye view image generation unit 12B generates a bird's-eye view image of the surroundings of the autonomous mobile robots M1B and M2B instead of generating a bird's-eye view image of the surroundings of the distance measuring sensors L1 and L2 according to Embodiment 1. Thus, the moving body position prediction unit 13B generates and processes not only a bird's-eye view image of a fixed location but also a bird's-eye view image of a dynamic location.

[0116] In addition, the bird's-eye view image generation unit 12B may extract the moving body information and the ground object information within the preset static prediction target range to generate the bird's-eye view image, similarly to the bird's-eye view image generation unit 12 according to Embodiment 1. Alternatively, the bird's-eye view image generation unit 12B may set a dynamic prediction target range based on the position information of the autonomous mobile robots M1B and M2B that perform movement prediction, and extract the moving body information and the ground object information within the prediction target range to generate the bird's-eye view image.

[0117] For example, when a dynamic prediction target range is set for the autonomous mobile robot M1B, the bird's-eye view image generation unit 12B sets a rectangular area centered on or approximately centered on the autonomous mobile robot M1B for movement prediction as the dynamic prediction target range. Furthermore, the bird's-eye view image generation unit 12B generates the bird's-eye view image in such a way that the prediction target range is the same in the time series of the bird's-eye view image. For example, first, the bird's-eye view image generation unit 12B sets a prediction target range centered on the latest position information of the autonomous mobile robot M1B. Next, the bird's-eye view image generation unit 12B corrects the time series of the past bird's-eye view images based on the latest prediction target range.

[0118] When the moving body information acquisition unit 11B acquires the measurement range information, the bird's-eye view image generation unit 12B determines an idle area where there is no moving body and an unmeasured area where the presence or absence of the moving body is unknown due to being blocked by an obstacle, based on the measurement range information. In addition, the idle area may include an area where there is no ground object, and the unmeasured area may include an area where the presence or absence of the ground object is unknown due to being blocked by an obstacle. Moreover, the bird's-eye view image generation unit 12B may generate an overhead image that distinguishes the idle area and the unmeasured area and then illustrates the image based on the determination results of the idle area and the unmeasured area. For example, the pixel intrinsic value of the idle area may be expressed as "0", and the pixel intrinsic value of the unmeasured area may be expressed as "1".

[0119] Fig.10 This is a diagram for explaining an example of an overhead image generated by the overhead image generating unit 12B. Fig.10 (a) is shown in Figure 3(a) A diagram of an overhead image F3 generated by the overhead image generation unit 12B based on the moving body information and the measurement range information provided by the autonomous mobile robots M1 and M2 under the same conditions.

[0120] The bird's-eye view image F3 shows the autonomous mobile robots M1B and M2B providing the moving body information and the moving bodies M3 and M5 around the autonomous mobile robots measured by the autonomous mobile robots M1B and M2B. In addition, the bird's-eye view image F3 shows the free area R1 and the unmeasured area R2 determined based on the measurement range information provided by the autonomous mobile robots M1B and M2B. Figure 3 The mobile body M4 in (a) exists outside the measurement range of the autonomous mobile robots M1B and M2B, that is, Fig.10 Since the moving object M4 is within the unmeasured region R2 of (a), the position of the moving object M4 becomes unclear in the bird's-eye view image F3.

[0121] Fig.10 (b) is about Fig.10 (a) The moving objects and objects on the ground shown in the figure are Figure 3 (b) A table that similarly records “position information”, “attribute information”, and “pixel-specific value”. Fig.10 (b) and Figure 3 The difference between (b) is that Figure 3 In (b), there is a moving body M4. Fig.10 In (b), there is no unmeasurable moving body M4; and Fig.10 In (b), an unmeasured region R2 is added.

[0122] Fig. 9 The moving object position prediction unit 13B is a processing unit including a moving prediction model selection unit 131B, a prediction model 132B for location A, and a prediction model 133B for location B. The prediction model 132B for location A and the prediction model 133B for location B are a plurality of moving prediction models for generating future bird's-eye view images when a time series of bird's-eye view images is input. The moving prediction model selection unit 131B selects one moving prediction model for generating future bird's-eye view images from the plurality of moving prediction models based on the characteristics of the bird's-eye view images.

[0123] Each of the prediction models 132B for location A and the prediction model 133B for location B, i.e., each movement prediction model, generates one or more future bird's-eye view images from the time series of past bird's-eye view images by learning the time series of bird's-eye view images, similarly to the movement prediction model of Embodiment 1. Each movement prediction model is specialized for each specific feature by learning the time series of bird's-eye view images having predetermined features. That is, a plurality of movement prediction models are defined for each predetermined feature of the bird's-eye view image.

[0124] The feature of the bird's-eye view image is a feature that shows a different tendency of movement of the moving object. For example, the feature of the bird's-eye view image may be at least one of the place shown in the bird's-eye view image (hereinafter referred to as the place of the bird's-eye view image), the density of the moving object in the bird's-eye view image, and the time of the bird's-eye view image. The time of the bird's-eye view image corresponds to the time when the moving object information that is the source of the bird's-eye view image is detected. Fig. 9 In the example of , as the feature of the bird's-eye view image, a location including the bird's-eye view image of point A and point B is used. Fig. 9 In the example, the number of the multiple mobility prediction models is 2, but is not limited thereto.

[0125] The motion prediction model selection unit 131B is a processing unit that determines a feature based on the time series of the overhead image, and selects one motion prediction model input to the time series of the overhead image from a plurality of motion prediction models based on the determination result of the feature. For example, the motion prediction model selection unit 131B can select one motion prediction model from a plurality of motion prediction models specified for each location of the overhead image by determining the location of the overhead image from the prediction target range of the overhead image. Alternatively, the motion prediction model selection unit 131B can select one motion prediction model from a plurality of motion prediction models specified for each density of the overhead image by determining the density of the moving objects from the number of moving objects illustrated in the overhead image. Alternatively, the motion prediction model selection unit 131B can select one motion prediction model from a plurality of motion prediction models specified for each time of the overhead image by determining the time of the overhead image. Alternatively, the motion prediction model selection unit 131B can select one motion prediction model from a plurality of motion prediction models specified for each combination of location, density, and time by determining the location of the overhead image, the density of the moving objects, and the time of the overhead image.

[0126] The moving object position prediction unit 13B generates future bird's-eye view images by inputting the time series of bird's-eye view images to the movement prediction model selected by the movement prediction model selection unit 131B, thereby predicting the future position of the moving object.

[0127] <Summary of Implementation Method 2>

[0128] According to the movement prediction device 1 involved in the second embodiment as described above, a time series of bird's-eye view images showing the positional relationship of the moving body is generated based on the moving body information provided by the autonomous mobile robot, and the future position of the moving body is predicted using machine learning on the time series of bird's-eye view images. According to such a structure, machine learning is performed on the bird's-eye view images whose versatility is improved by abstracting the positional relationship of the moving body, so even if the model used in the machine learning is not changed, it is possible to change the type of sensor, add an external sensor, and other changes to the structure of the sensor. In addition, even if the distance measuring sensors L1 and L2 described in the first embodiment are not provided, movement prediction can be implemented in coordination with the autonomous mobile robots M1B and M2B.

[0129] In Embodiment 2, a bird's-eye view image is generated that also shows an empty area where no moving object exists and an unmeasured area where the presence or absence of a moving object is unknown. With such a configuration, motion prediction can be performed that distinguishes between empty areas and unmeasured areas, thereby improving the accuracy of motion prediction.

[0130] In the second embodiment, a feature is determined based on the time series of the bird's-eye view image, and one motion prediction model input into the time series of the bird's-eye view image is selected from a plurality of motion prediction models based on the result of the feature determination. With such a configuration, a motion prediction model that matches the feature of the bird's-eye view image can be used for motion prediction, thereby improving the accuracy of motion prediction.

[0131] <Modification>

[0132] The movement prediction devices 1 and 1B according to Embodiments 1 and 2 may be integrated with the distance measuring sensors L1 and L2. In this case, the processing unit of the moving object information acquisition means 11 and 11B of the movement prediction devices 1 and 1B may be common to the processing unit of the distance measuring sensors L1 and L2.

[0133] Each unit of the movement prediction apparatus 1 and 1B according to the first and second embodiments can be expressed as a calculation process and a step of an information processing method for realizing position estimation.

[0134] Each unit of the motion prediction device 1 or 1B according to the first or second embodiment can be expressed as a motion prediction program executed on a computer such as an electronic computer, and can be implemented by executing the motion prediction program on the computer. In addition, each unit of the motion prediction device 1 or 1B according to the first or second embodiment can be implemented using a recording medium recording the motion prediction program.

[0135] Furthermore, the movement prediction devices 1 and 1B according to the first and second embodiments may be provided in, for example, an autonomous mobile robot, or may be implemented by a plurality of devices other than the autonomous mobile robot.

[0136] In addition, each embodiment and each modification example can be freely combined, or each embodiment and each modification example can be appropriately modified or omitted.

[0137] The above description is intended in all respects to be illustrative and not restrictive, and it should be understood that numerous modifications not shown are conceivable.

[0138] (Explanation of symbols)

[0139] 1, 1B: movement prediction device; 11, 11B: moving body information acquisition unit; 12, 12B: bird's-eye view image generation unit; 13, 13B: moving body position prediction unit; 14: movement prediction information providing unit; 15: ground object information storage unit; 130: moving body prediction model; 132B: prediction model for location A; 133B: prediction model for location B; M1~M5: moving body.

Claims

1. A movement prediction device, comprising: an information acquisition unit for acquiring mobile body information including position information of the mobile body; an overhead image generating unit for generating, based on the moving object information, an overhead image that illustrates a positional relationship of the moving object by means of a moving object graphic that represents the moving object; and The moving object position prediction unit predicts the future position of the moving object using machine learning on the time series of the bird's-eye view images.

2. The movement prediction device according to claim 1, wherein: The machine learning includes reinforcement learning or deep learning, The moving object position prediction unit generates a future bird's-eye view image by inputting the time series of the bird's-eye view images into one or more movement prediction models that perform the machine learning, thereby predicting the future position of the moving object.

3. The movement prediction device according to claim 1 or 2, wherein: The moving object information further includes attribute information indicating at least one of the type and movement pattern of the moving object. The bird's-eye view image generating unit determines a pixel-specific value of the moving object for distinguishing the moving object pattern in the machine learning based on the attribute information of the moving object.

4. The movement prediction device according to any one of claims 1 to 3, wherein: It also includes a feature information storage unit for storing feature information including position information of features. The bird's-eye view image generating unit generates the bird's-eye view image that illustrates positional relationships of the features using a feature graphic, ie, a feature graphic that represents the features, based on the feature information.

5. The movement prediction device according to claim 4, wherein: The feature information also includes attribute information indicating the type of the feature. The bird's-eye view image generating unit determines a pixel-specific value of the feature for distinguishing the feature figure in the machine learning based on the attribute information of the feature.

6. The movement prediction device according to claim 5, wherein: The mobile body includes an autonomous mobile robot, The bird's-eye view image generation unit determines, based on the attribute information of the ground object, whether the ground object is a ground object that the moving body can pass through, and whether it is a ground object that the autonomous mobile robot can pass through, and based on the determination result of the ground object, determines the pixel intrinsic value of the ground object and determines the order in which the ground object graphics are depicted in the bird's-eye view image.

7. The movement prediction device according to any one of claims 1 to 3, wherein: It also includes a feature information storage unit for storing feature information including position information of features. The information acquisition unit further acquires measurement range information indicating a range in which the sensor detecting the position information of the moving body and the position information of the ground object measures the position information of the moving body and the position information of the ground object without being blocked by an obstacle. The bird's-eye view image generation unit determines, based on the measurement range information, a vacant area where the moving object does not exist and an unmeasured area where the presence or absence of the moving object is unclear due to being blocked by the obstacle, and generates the bird's-eye view image that also illustrates the vacant area and the unmeasured area based on the determination results of the vacant area and the unmeasured area.

8. The movement prediction device according to claim 2, wherein: The one or more motion prediction models are a plurality of motion prediction models defined for each predetermined feature of the bird's-eye view image. The moving object position prediction unit determines the feature based on the time series of the bird's-eye view image, and selects one movement prediction model that is input with the time series of the bird's-eye view image from among the plurality of movement prediction models based on the result of the feature determination.

9. The movement prediction device according to claim 8, wherein: The feature includes at least one of a place shown in the bird's-eye view image, a density of the moving objects in the bird's-eye view image, and a time of the bird's-eye view image.

10. The movement prediction device according to any one of claims 1 to 5, wherein: The mobile body includes an autonomous mobile robot, The movement prediction device further includes a movement prediction information providing unit, which provides movement prediction information to the autonomous mobile robot based on the prediction result of the mobile body position prediction unit. The autonomous mobile robot moves to avoid collision with another mobile object other than the autonomous mobile robot, or moves to track the other mobile object, based on the movement prediction information.

11. A motion prediction method comprising: An information acquisition step of acquiring mobile body information including position information of the mobile body; an overhead image generating step of generating, based on the moving object information, an overhead image that illustrates a positional relationship of the moving object by means of a moving object graphic that represents the moving object; and The moving object position prediction step predicts the future position of the moving object using machine learning on the time series of the bird's-eye view images.

12. A mobile prediction program causing a computer to execute: an information acquisition unit for acquiring mobile body information including position information of the mobile body; an overhead image generating unit for generating, based on the moving object information, an overhead image that illustrates a positional relationship of the moving object by means of a moving object graphic that represents the moving object; and The moving object position prediction unit predicts the future position of the moving object using machine learning on the time series of the bird's-eye view images.

Images