Information processing device, information processing method, program, and storage medium

Abstract

To provide an information processing device capable of reducing degradation in the quality of a three-dimensional map even when an object that may affect the creation of the three-dimensional map is present.SOLUTION: An information processing device (100) has information acquisition means (101) for acquiring measurement information from a sensor (304), object recognition means (102) for recognizing an object based on the measurement information, influence determination means (103) for determining whether or not the object influences creation of a three-dimensional map, and action determination means (104) for determining the action of a moving body (200) based on the determination result of the influence determination means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, a program, and a storage medium. [Background technology]

[0002] Conventionally, there is known a technique for acquiring a three-dimensional shape of a real space based on distance information or image information measured by a sensor such as a camera (image capture device) or a light detection and ranging (LiDAR).Patent Document 1 discloses a method for detecting objects unnecessary for generating a three-dimensional model (three-dimensional map) from image data, detecting obstacle areas corresponding to the objects, and deleting point cloud information of the 3D LiDAR corresponding to the obstacle areas before generating a three-dimensional model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-192170 Summary of the Invention [Problem to be solved by the invention]

[0004] The method disclosed in Patent Document 1 deletes point cloud information of 3D LiDAR corresponding to objects, which reduces the quality of the three-dimensional map.

[0005] Therefore, an object of the present invention is to provide an information processing device that can reduce degradation in the quality of a three-dimensional map even when an object that may affect the creation of the three-dimensional map is present. [Means for solving the problem]

[0006] An information processing device as one aspect of the present invention has an information acquisition means for acquiring measurement information from a sensor, an object recognition means for recognizing an object based on the measurement information, an influence determination means for determining whether or not the object has an influence when creating a three-dimensional map, and an action determination means for determining the action of a moving body based on the determination result of the influence determination means.

[0007] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an information processing device that can reduce degradation in the quality of a three-dimensional map even when an object that may affect the creation of the three-dimensional map is present. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an application scene in the first embodiment. [Figure 2] 1 is a configuration diagram of an information processing device according to a first embodiment. [Figure 3] 1 is a block diagram of an information processing device and a mobile object according to a first embodiment. [Figure 4] 1 is a flowchart showing a process of the information processing device according to the first embodiment. [Figure 5] 10 is a flowchart showing a process of map creation influence determination in the first embodiment. [Figure 6] FIG. 10 is a block diagram of an information processing device and a mobile object according to a second embodiment. [Figure 7] 10 is a flowchart showing a process of an information processing device according to a second embodiment. [Figure 8] 10 is a flowchart showing a process of map creation influence determination in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all of the combinations of features described in the following embodiments are necessarily essential to the present invention.

[0011] In each embodiment, an example in which the present invention is applied to a moving body will be described. A three-dimensional model is generated based on sensor information acquired while the moving body is traveling and its own position. It is desirable that the generated three-dimensional model reconstructs the working environment of the moving body exactly. Therefore, it is necessary to prevent a decrease in the quality of the three-dimensional model due to an object that moves in accordance with the movement of the moving body being registered in the three-dimensional model, or an object that should be registered in the three-dimensional model being occluded by the moving object and not being registered. [Example]

[0012] First, a first embodiment of the present invention will be described. FIG. 1 is a diagram showing an application scene (usage scene) of a mobile body 200 having an information processing device 100 according to this embodiment. The mobile body 200 includes the information processing device 100 and a sensor 304 that acquires color images and stereo images. The mobile body 200 travels through a work environment and estimates its own position based on the stereo images. The mobile body 200 also acquires color images of a wall 301 and a moving object 302 using the sensor 304. The object 302 is, for example, a person walking around the mobile body 200, but is not limited to this as long as it is a movable object, and may be a movable obstacle or obstruction, etc. The mobile body 200 reconstructs the work environment of the mobile body 200 by applying a Structure from Motion (SfM) algorithm to the color images acquired by the sensor 304.

[0013] Next, the configuration of the information processing device 100 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the hardware configuration of the information processing device 100. The information processing device 100 has a CPU 150, a RAM 151, a storage unit 152 such as an HDD or SSD, an I / O 153, a communication unit 154, and a system bus 155. The CPU 150 uses the RAM 151 as a work memory to execute an operating system (OS) and various computer programs stored in the storage unit 152, and controls each unit via the system bus 155. The programs executed by the CPU 150 include, for example, a program for determining the behavior of the moving object 200, which will be described later. The I / O 153 controls a sensor 304 mounted on the moving object 200, and inputs data acquired by the sensor 304.

[0014] Next, the configurations of the information processing device 100 and the mobile body 200 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the logical configuration of the information processing device 100 and the mobile body 200. In this embodiment, the information processing device 100 is mounted on the mobile body 200. However, this embodiment is not limited to this, and at least some of the functions of the information processing device 100 may be provided in a location different from the mobile body 200 and configured to be able to communicate with the mobile body 200.

[0015] The information processing device 100 includes a measurement information acquisition unit 101 , an object recognition unit 102 , a map creation influence determination unit 103 , and an action determination unit 104 .

[0016] The measurement information acquisition unit 101 acquires a color image from a sensor 304 mounted on the moving body 200 via the I / O 153 and transfers the data to the object recognition unit 102 .

[0017] The object recognition unit 102 applies a machine learning algorithm to the color image input from the measurement information acquisition unit 101 to recognize (detect) an object in the color image and acquire image area information containing the detected object. Here, the object to be recognized is a person, a vehicle, or a movable object such as a shelf with casters or a cart. At the time of object recognition, the object may be in a moving or stationary state.

[0018] The map creation influence determination unit 103 determines the influence of the object on the three-dimensional reconstruction (three-dimensional map) of the robot work environment based on the image area information of the object detected (recognized) by the object recognition unit 102. In this embodiment, the three-dimensional map is a three-dimensional mesh model, a map used to estimate the robot's own position, or the like, but is not limited to these.

[0019] The behavior determining unit 104 determines the behavior of the moving body 200 according to the influence of the object determined by the map creation influence determining unit 103, that is, based on the determination result of the map creation influence determining unit 103. The behavior to be determined will be described later.

[0020] In addition to the components of the information processing device 100, the moving object 200 includes a self-position estimation unit 202 and a control unit 203.

[0021] The self-position estimation unit 202 estimates its own position (self-position information) by applying an algorithm such as a Visual SLAM algorithm to stereo images acquired from the sensor 304 via the I / O 153. Here, the self-position information estimated by the self-position estimation unit 202 includes information on the three-dimensional position and posture of the moving object 200.

[0022] The control unit 203 controls the operation of the moving object 200 based on the action of the moving object 200 determined by the action determination unit 104 and the self-position information of the moving object 200 estimated by the self-position estimation unit 202 .

[0023] Next, referring to Fig. 4, a process will be described in which the information processing device 100 determines the influence of a movable object and determines the behavior of the moving body 200. Fig. 4 is a flowchart showing the process in which the information processing device 100 determines the influence of a movable object and determines the behavior of the moving body 200. This flow is executed by each unit of the information processing device 100 every time a color image is input from the measurement information acquisition unit 101 to the object recognition unit 102.

[0024] First, in step S101, the information processing device 100 performs initialization processing to transition to a state where it is ready to operate (process). Subsequently, in step S102, the measurement information acquisition unit 101 acquires a color image (measurement information) from the sensor 304 via the I / O 153.

[0025] Next, in step S103, the object recognition unit 102 performs object recognition processing using a machine learning algorithm on the color image (image area that is the measurement area of ​​the sensor 304) acquired by the measurement information acquisition unit 101. The object recognition unit 102 detects (acquires) an image area recognized as a movable object from the measurement area of ​​the sensor 304 through the object recognition processing.

[0026] Next, in step S104, the map creation influence determination unit 103 determines whether or not the movable object will degrade the quality of the three-dimensional model (three-dimensional map) generated by the three-dimensional reconstruction process, based on the image area of ​​the object detected by the object recognition unit 102 (map creation influence determination). Details of the map creation influence determination will be described later with reference to FIG. 5.

[0027] Next, in step S105, the behavior determination unit 104 determines the behavior of the moving body 200 based on the result of the determination made by the map creation influence determination unit 103. Here, the behavior to be determined differs depending on whether the moving body 200 is in a moving state or a stopped state.

[0028] First, the case where the moving body 200 is moving will be described. If the map creation influence determination unit 103 determines that "the object does not degrade the quality of the three-dimensional model," the moving body 200 continues moving (maintains the moving state). On the other hand, if it determines that "the object degrades the quality of the three-dimensional model," the moving body 200 stops moving and transitions to a stopped moving state.

[0029] Next, the case where the moving body 200 is stopped will be described. If the map creation influence determination unit 103 determines that "the object degrades the quality of the three-dimensional model," the moving body 200 maintains a stopped state. On the other hand, if the determination of the map creation influence determination unit 103 changes from "the object degrades the quality of the three-dimensional model" to "the object does not degrade the quality of the three-dimensional model," the map creation influence determination unit 103 resumes the moving of the moving body 200 (transitioning from a stopped state to a moving state).

[0030] Next, the map creation influence determination process (step S104) will be described in detail with reference to Fig. 5. Fig. 5 is a flowchart showing the map creation influence determination process.

[0031] First, in step S111, the map creation influence determination unit 103 acquires information (image information of the target object) about the object (target object) recognized by the object recognition unit 102. Here, the information about the object is area information indicating the area in the image where the object exists (its position in the image).

[0032] Next, in step S112, the map creation influence determination unit 103 calculates the number of pixels that include the target object. Next, in step S113, the map creation influence determination unit 103 calculates the ratio of the number of pixels of the target object to the number of pixels of the color image.

[0033] Next, in step S114, the map creation influence determination unit 103 determines whether or not there is an influence based on the proportion of the number of pixels of the target object calculated in step S113. For example, if the proportion of the number of pixels of the target object calculated in step S113 is greater than a predetermined value, the map creation influence determination unit 103 determines that the target object degrades the quality of the three-dimensional model. On the other hand, if the proportion is smaller than the predetermined value, the map creation influence determination unit 103 determines that the target object does not degrade the quality of the three-dimensional model.

[0034] The object recognition unit 102, the map creation influence determination unit 103, and the behavior decision unit 104 may perform each process using a trained model that has been trained using machine learning. In this case, for example, multiple combinations of input data and output data for each unit are prepared as training data, knowledge is acquired using machine learning using the training data, and a trained model is generated that outputs output data for the input data as a result based on the acquired knowledge. The trained model can be configured, for example, as a neural network model. The trained model can then operate in cooperation with a CPU, GPU, or the like as a program for performing the same processing as each unit of the information processing device 100, thereby performing each process. The trained model may be updated as needed after certain processing.

[0035] According to this embodiment, it is possible to reduce the degradation of the quality of the three-dimensional model due to the influence of movable objects.

[0036] In this embodiment, the sensor 304 is an imaging sensor that acquires a color image as measurement information, but is not limited to this. The sensor 304 may acquire, for example, a monochrome image or a depth image as measurement information. Alternatively, the sensor 304 may acquire point cloud information (measurement information) using a LiDAR.

[0037] In this embodiment, the object recognition unit 102 detects (recognizes) movable objects from a color image using a machine learning algorithm, but is not limited to this. The object recognition unit 102 may detect (recognize) movable objects by, for example, pattern matching with images of objects registered in advance (registered images). Alternatively, the object recognition unit 102 may convert the depth image into a three-dimensional point cloud and perform clustering to detect (recognize) objects.

[0038] In this embodiment, the map creation influence determination unit 103 determines whether to degrade the quality of the three-dimensional model based on the number of pixels of an object in the color image (the size of the area occupied by the object in the color image), but this is not limited to this. For example, the map creation influence determination unit 103 may store the results of object recognition by the object recognition unit 102 for each frame in chronological order and calculate the amount of change in the number of pixels occupied by the same object (target object) (the amount of change in the area occupied by the object in the color image). Then, if the amount of change in the number of pixels is greater than a predetermined value, the map creation influence determination unit 103 may determine that the quality of the three-dimensional model should be degraded.

[0039] In this embodiment, the map creation influence determination unit 103 may determine whether to degrade the quality of the three-dimensional model based on the size of a spatial region occluded by a movable object within the field of view (measurement region) of the sensor 304. For example, a case will be described in which a time-of-fly (ToF) depth camera (ToF sensor) is used as the sensor 304 to acquire a depth image. The volume of the three-dimensional space occluded by the object can be calculated using the relationship between the angle of view, resolution, and observable distance of the depth camera and information on the movable object region in the depth image. In this case, the map creation influence determination unit 103 may determine that the quality of the three-dimensional model will degrade if the calculated volume exceeds a predetermined value. Alternatively, the map creation influence determination unit 103 may store the volume calculated for each depth image frame and determine that the quality of the three-dimensional model will degrade if the change in volume exceeds a predetermined value. This allows for more accurate evaluation of the degree of influence of the movable object on the three-dimensional model.

[0040] In this embodiment, when the map creation influence determination unit 103 determines that an object will degrade the quality of the three-dimensional model, the behavior determination unit 104 determines the behavior of the moving body 200 so that the moving body 200 stops on the spot. However, this is not limited to this. For example, the behavior determination unit 104 may determine the behavior of the moving body 200 so that the moving body 200 moves to a position or posture where the object does not appear in the image. Alternatively, the behavior determination unit 104 may determine the behavior of the moving body 200 so that the moving body 200 changes its travel route and revisits the affected location later. Alternatively, the behavior determination unit 104 may determine the behavior of another vehicle (another moving body) different from the moving body 200. For example, the behavior of the other vehicle may be determined so that the other vehicle heads to the location after a predetermined time. This makes it possible to reduce degradation in the quality of the three-dimensional model even when it is difficult to stop on the spot, such as when the moving body is on a road. [Example]

[0041] Next, a second embodiment of the present invention will be described. In the first embodiment, the map creation influence determination unit 103 determines the influence of an object based on the ratio of the number of pixels of the object to the number of pixels of the color image. On the other hand, in this embodiment, for an object recognized by the object recognition unit 102, the object movement direction calculation unit (calculation means) 105 calculates the movement direction of the object. Then, the map creation influence determination unit 103 determines the influence of the object based on information on the movement direction of the object. Note that in this embodiment, explanations common to the first embodiment will be omitted. Furthermore, each of the modified examples described in the first embodiment can also be applied to this embodiment.

[0042] First, the configurations of an information processing device 100a and a moving body 200a in this embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the logical configurations of the information processing device 100a and the moving body 200a. The information processing device 100a of this embodiment differs from the information processing device 100 of the first embodiment in that it has an object movement direction calculation unit 105. Note that other configurations of the information processing device 100a are common to the information processing device 100 of the first embodiment, and therefore description thereof will be omitted.

[0043] The object movement direction calculation unit 105 calculates the movement direction of each object recognized by the object recognition unit 102. The method for calculating the movement direction will be described later. The map creation influence determination unit 103 determines the influence of the object on the three-dimensional reconstruction of the robot working environment, based on the image area information of the object recognized by the object recognition unit 102 and the movement direction of the object calculated by the object movement direction calculation unit 105.

[0044] Next, a process when the information processing device 100a determines the influence of a movable object and determines the behavior of the moving object 200a will be described with reference to Fig. 7. Fig. 7 is a flowchart showing a process when the information processing device 100a determines the influence of a movable object and determines the behavior of the moving object 200a. This flow differs from the flow in Fig. 4 described in the first embodiment mainly in that a process of calculating the moving direction of the object (step S204) is inserted.

[0045] First, in step S201, the information processing device 100a performs initialization processing to transition to a state where it is ready to operate (process). In this embodiment, the initialization processing is automatically executed when the information processing device 100a is powered on. Next, in step S202, the measurement information acquisition unit 101 acquires color images and stereo images from the sensor 304 via the I / O 153.

[0046] Next, in step S203, the object recognition unit 102 performs object recognition processing using a machine learning algorithm on the color image acquired by the measurement information acquisition unit 101. The object recognition unit 102 detects (acquires) an image area recognized as a movable object through the object recognition processing, and assigns a unique ID (identification information) to the object.

[0047] Next, in step S204, the object movement direction calculation unit 105 calculates the movement direction of the object recognized by the object recognition unit 102. Here, a method for calculating the movement direction will be described. The object movement direction calculation unit 105 acquires a depth image by stereo matching from the stereo images acquired by the measurement information acquisition unit 101. Next, the object movement direction calculation unit 105 converts the depth image area corresponding to the image area recognized as the object by the object recognition unit 102 into a three-dimensional point cloud, and calculates the three-dimensional center of gravity coordinates of the converted three-dimensional point cloud. Here, the object movement direction calculation unit 105 holds ID information of the corresponding object along with the calculated three-dimensional center of gravity coordinates. Then, the movement direction of the object is calculated based on the difference between the three-dimensional center of gravity coordinates of the object a predetermined number of frames ago (for example, 10 frames) and the current three-dimensional center of gravity coordinates.

[0048] Next, in step S205, the map creation influence determination unit 103 determines whether or not the object degrades the quality of the 3D model (3D map) generated by the 3D reconstruction process, i.e., whether or not the object has an influence on the creation of the 3D map. In this embodiment, the map creation influence determination unit 103 determines whether or not the object has an influence, based on the image area of ​​the object recognized by the object recognition unit 102 and the movement direction of the object calculated by the object movement direction calculation unit 105.

[0049] Next, in step S206, the behavior determination unit 104 determines the behavior of the moving object 200 based on the result determined by the map creation influence determination unit 103. Next, in step S207, the information processing device 100a determines whether or not to end this process. If the information processing device 100a determines not to end this process, the process returns to step S202. On the other hand, if the information processing device 100a determines to end this process (for example, when the power of the information processing device 100a is turned off), the information processing device 100a ends this flow.

[0050] Next, the map creation influence determination process (step S205) will be described in detail with reference to Fig. 8. Fig. 8 is a flowchart showing the map creation influence determination process.

[0051] First, in step S211, the map creation influence determination unit 103 acquires information (image information of the target object) about the object (target object) recognized by the object recognition unit 102. Here, the information about the object is area information indicating the area in the image where the object exists (its position in the image).

[0052] Subsequently, in step S212, the map creation influence determination unit 103 acquires the moving direction of the object (target object) calculated by the object moving direction calculation unit 105.

[0053] Next, in step S213, the map creation influence determination unit 103 predicts the number of pixels of the object obtained after a predetermined time (for example, one second) based on the number of pixels of the object relative to the number of pixels of the color image and the moving direction of the object. For example, the map creation influence determination unit 103 predicts that the number of pixels will be larger as the moving direction of the object approaches the sensor 304.

[0054] Next, in step S214, the map creation influence determination unit 103 determines whether or not there is an influence of the object based on the prediction result of step S213. For example, if the proportion of the pixel count of the object predicted in step S213 is greater than a predetermined value, the map creation influence determination unit 103 determines that the quality of the 3D model will be reduced. On the other hand, if the proportion of the pixel count of the object is smaller than the predetermined value, the map creation influence determination unit 103 determines that the quality of the 3D model will not be reduced.

[0055] According to this embodiment, by calculating the direction of movement of an object that may move, it is possible to determine the action to be taken before the quality of the three-dimensional model is affected, and a high-quality three-dimensional model can be generated.

[0056] In this embodiment, the object movement direction calculation unit 105 generates a depth image of the object from stereo camera images, converts it into a three-dimensional point cloud, and then calculates the movement direction of the object in three-dimensional coordinates, but this is not limited to this. For example, the movement direction in two-dimensional image coordinates may be calculated from a color image. When calculating the movement direction, optical flow may be applied, or the brightness difference between multiple frames may be calculated and the movement direction may be calculated based on the brightness difference. This makes it possible to reduce quality degradation of the three-dimensional model with a small amount of calculation.

[0057] In this embodiment, when calculating the movement direction, a technique such as deep learning may be used to learn the relationship between the orientation of an object and the direction of movement, and the movement direction may be calculated based on the orientation of the object. For example, the movement direction may be calculated based on inferences such as "if it is a car, it is likely to move in the direction ahead of the vehicle" or "if it is a person, it is likely to move in the direction of the line of sight." In this way, even if an object that may move is not actually moving, the movement can be predicted, thereby reducing degradation in the quality of the three-dimensional model.

[0058] In this embodiment, the map creation influence determination unit 103 predicts the number of pixels of an object after a predetermined time based on the object's movement direction calculated by the object movement direction calculation unit 105 and area information of the image recognized as an object by the object recognition unit 102, and determines the influence on map creation based on the predicted number of pixels. However, this embodiment is not limited to this. For example, if the angular difference between the object's movement direction and the movement direction of the moving body 200a is equal to or less than a predetermined value, the map creation influence determination unit 103 may determine that the object will travel within the observation range of the sensor 304 for a long period of time and determine that the object will affect map creation. This reduces the possibility that objects traveling side by side will degrade the quality of the 3D model for a long period of time.

[0059] In each embodiment, the information processing device 100 (100a) has been described as being built into the mobile object 200 (200a), but this is not limited thereto. For example, the information processing device 100 (100a) may be a computer connected to the mobile object 200 (200a) via wireless communication. In this case, the mobile object 200 (200a) transmits data acquired by the sensor 304 and the position and attitude of the mobile object 200 (200a) estimated by the self-position estimation unit 202 to the information processing device 100 (100a) via wireless communication. The information processing device 100 (100a) acquires the data transmitted by the mobile object 200 (200a) via the communication unit 154.

[0060] In each embodiment, the case where the object recognition unit 102 detects one object as a movable object in step S203 has been described, but the present invention is not limited to this. The map creation influence determination unit 103 may determine the influence by comprehensively considering the influence of multiple objects.

[0061] In each embodiment, the sensor 304 is an image sensor capable of acquiring color images and stereo images, but is not limited to this. The sensor 304 may be, for example, a depth sensor capable of acquiring depth images, or a LiDAR. Alternatively, a combination of multiple different types of sensors, such as a depth sensor and a LiDAR, may be used.

[0062] In each embodiment, the sensor 304 is described as being mounted on the moving body 200 (200a), but this is not limiting. For example, the sensor 304 may be a surveillance camera attached to the ceiling of a building and connected to the information processing device 100 (100a) by wireless communication.

[0063] In each embodiment, the map creation influence determination unit 103 may determine the influence on map creation depending on the type of object recognized by the object recognition unit 102. For example, if the object recognition unit 102 detects (recognizes) a vehicle such as a forklift or an object such as a cart with casters as an object that is likely to move, the map creation influence determination unit 103 may determine that the recognized object will degrade the quality of the three-dimensional model. This makes it possible to reduce the degradation of the quality of the three-dimensional model caused by the inclusion of an object that is not currently moving but is likely to move in the future in the three-dimensional model. Furthermore, if the object recognition unit 102 detects (recognizes) an object intended for occlusion, such as a blackout curtain, the map creation influence determination unit 103 may determine that the recognized object will degrade the quality of the three-dimensional model. This makes it possible to reduce the degradation of the quality of the three-dimensional model regardless of the possibility of movement.

[0064] In each embodiment, the behavior determined by the behavior determination unit 104 may be changed based on the hardware configuration of the moving body 200 (200a) or the category of the object recognized by the object recognition unit 102. For example, if the moving body 200 (200a) is configured to move using wheels with limited directional freedom, the behavior determination unit 104 determines to stop the moving body 200 (200a). On the other hand, if the moving body 200 (200a) is configured to move using mechanisms such as omni-wheels or mecanum wheels with relatively high directional freedom, the moving body 200 (200a) continues moving while changing its position or posture so that movable objects do not enter the observation range of the sensor 304. This can further improve the efficiency of generating a three-dimensional model. If the object category is an autonomous vehicle, the behavior determination unit 104 may determine to have the moving body 200 (200a) avoid the object. If the object is a person, the behavior determination unit 104 may determine to stop the moving body 200 (200a). In this way, by stopping the moving body 200 (200a) only when an object whose moving direction is more difficult to predict is detected, it is possible to further increase the efficiency of generating a three-dimensional model.

[0065] In each embodiment, the action determined by the action determining unit 104 is not limited to controlling the travel of the moving body 200 (200a). For example, the action determining unit 104 may determine to notify the operator of the moving body 200 (200a) that the quality of the three-dimensional model is affected via wireless communication. In this case, the action determining unit 104 functions as a notification means for notifying the operator that the object is affected. Alternatively, if the object affecting the quality of the three-dimensional model is a person, the action determining unit 104 may determine to play an electronic voice via an interface such as a speaker to urge the person to evacuate from the location. Note that the action determining unit 104 is not essential in each embodiment. The information processing device 100 (100a) may have only a notification means for notifying the operator that the object is affected, instead of the action determining unit 104.

[0066] In each embodiment, the behavior determining unit 104 may determine multiple behaviors. For example, if the object affecting the quality of the 3D model is a person, the behavior determining unit 104 may determine to play an electronic voice message via an interface such as a speaker to advise the person to evacuate the area, and to notify the operator of the mobile object that the quality of the 3D model will be affected. This makes it possible to more reliably reduce the degradation of the quality of the 3D model.

[0067] In each embodiment, the 3D model is not limited to a 3D mesh model for human viewing. For example, it may be a model that stores the coordinates of feature points and image information of the feature points required for the moving object 200 (200a) to travel using the Visual SLAM algorithm.

[0068] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0069] According to each embodiment, it is possible to provide an information processing device, an information processing method, a program, and a storage medium that can reduce degradation in the quality of a three-dimensional map even when an object that may affect the creation of the three-dimensional map is present.

[0070] The disclosure of each embodiment includes the following configurations and methods. (Configuration 1) an information acquisition means for acquiring measurement information from a sensor; an object recognition means for recognizing an object based on the measurement information; an influence determination means for determining whether or not the object has an influence when creating a three-dimensional map; and behavior determination means for determining the behavior of the moving object based on the determination result of said influence determination means. (Configuration 2) 2. The information processing device according to configuration 1, wherein the influence determining means detects an area occupied by the object in the measurement area of ​​the sensor, and determines whether or not there is an influence of the object based on the area. (Configuration 3) the measurement information is an image, The influence determination means Calculating the size of the area occupied by the object in the image; 3. The information processing device according to configuration 1 or 2, wherein it is determined whether or not there is an influence of the object based on the size of the region. (Configuration 4) the measurement information is an image, The influence determination means calculating a change in an area occupied by the object in the image; 3. The information processing device according to configuration 1 or 2, wherein it is determined whether or not there is an influence of the object based on the amount of change in the area. (Configuration 5) the measurement information is a depth image, The influence determination means Calculating a size of a spatial region occluded by the object based on the depth image; 3. The information processing device according to configuration 1 or 2, wherein it is determined whether or not there is an influence of the object based on the size of the spatial region. (Configuration 6) 3. The information processing device according to configuration 1 or 2, wherein the measurement information is point cloud information. (Configuration 7) further comprising a calculation means for calculating a moving direction of the object; 7. The information processing apparatus according to any one of configurations 1 to 6, wherein the influence determining means determines whether or not there is an influence of the object based on the movement direction. (Configuration 8) 8. The information processing device according to any one of configurations 1 to 7, wherein the object is a movable object. (Configuration 9) 9. The information processing device according to any one of configurations 1 to 8, wherein the behavior decision means stops the moving object from traveling when the influence determination means determines that there is an influence. (Configuration 10) 9. The information processing device according to any one of configurations 1 to 8, wherein the behavior decision means resumes traveling of the moving object when the influence determination means determines that the influence has disappeared. (Configuration 11) The information processing device according to any one of configurations 1 to 8, wherein the action decision means changes the travel route of the moving body so as to reduce the impact when the impact determination means determines that there is an impact. (Configuration 12) The information processing device according to any one of configurations 1 to 8, wherein the action decision means changes the travel route of the moving object so as to revisit the affected location when the influence determination means determines that there is an influence. (Configuration 13) 9. The information processing device according to any one of configurations 1 to 8, wherein the behavior decision means notifies the user of the fact that the device has been affected when the effect determination means determines that the device has been affected. (Configuration 14) an information acquisition means for acquiring measurement information from a sensor; an object recognition means for recognizing an object based on the measurement information; an influence determination means for determining whether or not the object has an influence when creating a three-dimensional map; and a notification unit that notifies the determination result of the influence determination unit. (Configuration 15) 15. The information processing device according to any one of configurations 1 to 14, wherein the three-dimensional map is at least one of a three-dimensional mesh model and a map used for estimating a self-position. (Configuration 16) the object is an object present in a space surrounding the moving object, 16. The information processing device according to any one of configurations 1 to 15, wherein the three-dimensional map is a map in the space. (Method 1) acquiring measurement information from the sensor; recognizing an object based on the measurement information; determining whether the object has an influence on the creation of a three-dimensional map; and a step of determining an action of the moving object based on the determination result in the step of determining whether or not there is an influence. (Configuration 17) A program that causes a computer to execute each step of the information processing method described in Method 1. (Configuration 18) A computer-readable storage medium storing a program for causing a computer to execute each step of the information processing method according to configuration 17.

[0071] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the present invention. Furthermore, the configurations of the respective embodiments may be combined as appropriate. [Explanation of symbols]

[0072] 100 Information processing device 101 Measurement information acquisition unit (information acquisition means) 102 Object recognition unit (object recognition means) 103 Map creation impact determination unit (impact determination means) 104 Action decision unit (action decision means) 105 Object movement direction calculation unit (calculation means) 200 Mobile 304 Sensors

Claims

1. an information acquisition means for acquiring measurement information from a sensor; an object recognition means for recognizing an object based on the measurement information; an influence determination means for determining whether or not the object has an influence when creating a three-dimensional map; and behavior determination means for determining the behavior of the moving object based on the determination result of said influence determination means.

2. 2. The information processing apparatus according to claim 1, wherein the influence determining means detects an area occupied by the object in the measurement area of ​​the sensor, and determines whether or not there is an influence of the object based on the area.

3. the measurement information is an image, The influence determination means Calculating the size of the area occupied by the object in the image; The information processing apparatus according to claim 1 , wherein whether or not there is an influence of the object is determined based on the size of the region.

4. the measurement information is an image, The influence determination means calculating a change in an area occupied by the object in the image; The information processing apparatus according to claim 1 , wherein it is determined whether or not there is an influence of the object based on the amount of change in the area.

5. the measurement information is a depth image, The influence determination means Calculating a size of a spatial region occluded by the object based on the depth image; The information processing apparatus according to claim 1 , wherein whether or not there is an influence of the object is determined based on the size of the spatial region.

6. 2. The information processing apparatus according to claim 1, wherein the measurement information is point cloud information.

7. further comprising a calculation means for calculating a moving direction of the object; 2. The information processing apparatus according to claim 1, wherein the influence determining means determines whether or not there is an influence of the object based on the moving direction.

8. 8. The information processing apparatus according to claim 1, wherein the object is a movable object.

9. 8. The information processing apparatus according to claim 1, wherein the action decision means stops the movement of the moving object when the influence determination means determines that there is an influence.

10. 8. The information processing apparatus according to claim 1, wherein the action decision means resumes the movement of the mobile object when the influence determination means determines that the influence has disappeared.

11. The information processing device according to any one of claims 1 to 7, characterized in that, when the influence determination means determines that there is an influence, the action determination means changes the travel route of the moving body so as to reduce the influence.

12. The information processing device according to any one of claims 1 to 7, characterized in that, when the influence determination means determines that there is an influence, the action determination means changes the travel route of the mobile body so as to revisit the affected location.

13. 8. The information processing apparatus according to claim 1, wherein the action decision means notifies the user of the influence when the influence determination means determines that the influence exists.

14. an information acquisition means for acquiring measurement information from a sensor; an object recognition means for recognizing an object based on the measurement information; an influence determination means for determining whether or not the object has an influence when creating a three-dimensional map; and a notification unit that notifies the determination result of the influence determination unit.

15. 15. The information processing apparatus according to claim 1, wherein the three-dimensional map is at least one of a three-dimensional mesh model and a map used for estimating a self-position.

16. the object is an object present in a space surrounding the moving object, 15. The information processing apparatus according to claim 1, wherein the three-dimensional map is a map of the space.

17. acquiring measurement information from the sensor; recognizing an object based on the measurement information; determining whether the object has an influence on the creation of a three-dimensional map; and a step of determining an action of the moving object based on the determination result in the step of determining whether or not there is an influence.

18. A program causing a computer to execute each step of the information processing method according to claim 17.

19. A computer-readable storage medium storing a program for causing a computer to execute each step of the information processing method according to claim 18.

Images