Information processing device, system, method, and program

Abstract

An information processing device comprises: a first identification means for identifying, as a first region, a space in which a discretionary object is not present within a three-dimensional space, on the basis of a three-dimensional model that includes an object; a second identification means for identifying, as a second region, a space in which a measurement point is not included within a measurement target space, on the basis of three-dimensional sensor data in which the object is measured by a three-dimensional sensor; and an estimation means for estimating the orientation of the object on the basis of a first comparison result between the first region and the second region. This makes it possible to improve the accuracy of estimating the orientation of the object using the three-dimensional model of the object and the three-dimensional sensor data.

Description

Information Processing Apparatus, System, Method, and Program

[0001] The present disclosure relates to an information processing apparatus, a system, a method, and a program.

[0002] In recent years, for example, in order to autonomously control construction machinery that loads cargo onto a loading platform (vessel) such as a dump truck, a technique for estimating the posture of the vessel using a three-dimensional sensor has been demanded.

[0003] Patent Document 1 discloses a technique related to a position and orientation measurement apparatus that measures the position and orientation of an object whose three-dimensional shape is known. The position and orientation measurement apparatus according to Patent Document 1 captures a distance image in which distance information to an object is held as pixel values, and selects an invalid region that is a region where distance information between the imaging position and the object cannot be obtained from the distance image. Then, the position and orientation measurement apparatus determines the approximate position and orientation of the object using the feature amount of the invalid region.

[0004] Japanese Patent Application Laid-Open No. 2011-174891

[0005] However, when estimating the posture of an object by comparing three-dimensional sensor data measured by a three-dimensional sensor with a three-dimensional model of the object, there is a problem that the estimation error of the posture can become large. One of the reasons is that due to insufficient three-dimensional sensor data, it may be compared with a part different from the object in the three-dimensional model.

[0006] An object of the present disclosure is to provide an information processing apparatus, a system, a method, and a program for improving the estimation accuracy of the posture of an object using a three-dimensional model of the object and three-dimensional sensor data in view of the above-described problems.

[0007] The information processing apparatus according to the present disclosure includes: a first specifying unit that specifies, as a first region, a space in which no arbitrary object exists in a three-dimensional space based on a three-dimensional model including an object; a second specifying unit that specifies, as a second region, a space that does not include a measurement point in a measurement target space based on three-dimensional sensor data in which the object is measured by a three-dimensional sensor; and an estimation unit that estimates the posture of the object based on a first comparison result between the first region and the second region.

[0008] The information processing system according to this disclosure comprises a three-dimensional sensor and an information processing device, the information processing device comprising: a first identification means for identifying a space in three-dimensional space where no object exists as a first region based on a three-dimensional model including an object; a second identification means for identifying a space in a measurement target space where no object exists as a second region based on three-dimensional sensor data measured by the three-dimensional sensor of the object; and an estimation means for estimating the orientation of the object based on a first comparison result of the first region and the second region.

[0009] The information processing method relating to this disclosure involves a computer identifying a space in a three-dimensional space where no object exists as a first region, based on a three-dimensional model including the object; identifying a space in a measurement target space where no object exists as a second region, based on three-dimensional sensor data measured by the object using a three-dimensional sensor; and estimating the orientation of the object based on a first comparison result of the first region and the second region.

[0010] The information processing device program according to this disclosure causes a computer to perform the following: a first identification process that identifies a space in a three-dimensional space where no object exists as a first region, based on a three-dimensional model including an object; a second identification process that identifies a space in a measurement target space where no object exists as a second region, based on three-dimensional sensor data measured by a three-dimensional sensor of the object; and an estimation process that estimates the pose of the object based on a first comparison result of the first region and the second region.

[0011] This disclosure makes it possible to improve the accuracy of estimating the orientation of an object using a three-dimensional model of the object and three-dimensional sensor data.

[0012] This is a block diagram showing the configuration of the information processing device according to this disclosure. This is a flowchart showing the flow of the information processing method according to this disclosure. This is a block diagram showing the overall configuration of the information processing system according to this disclosure. This is a block diagram showing the configuration of the information processing device according to this disclosure. This is a flowchart showing the flow of area identification processing based on sensing simulation according to this disclosure. This is a diagram for explaining the concept of setting up multiple virtual 3D sensors in the sensing simulation according to this disclosure. This is a diagram showing an example of point cloud data measured by a virtual 3D sensor according to this disclosure. This is a diagram showing an example of a depth image generated from point cloud data according to this disclosure. This is a diagram for explaining the concepts of three types of spatial attributes according to this disclosure. This is a diagram for explaining the relationship between the measurement points of sensor data according to this disclosure and the three types of spatial attributes of each voxel. This is a diagram showing an example of a distance transformation field with the spatial attribute "occupied" according to this disclosure. This is a diagram showing an example of an image of a voxel set and distance transformation field with the spatial attribute "occupied" according to this disclosure. This is a diagram showing an example of point cloud data with the spatial attribute determined to be "free" according to this disclosure. This is a diagram showing an example of point cloud data with the spatial attribute determined to be "occupied" or "free" being superimposed. This figure shows an example of a distance transformation field with the spatial attribute "free" according to this disclosure. This figure shows an example of an image of a voxel set and distance transformation field with the spatial attribute "free" according to this disclosure. This flowchart shows the flow of region identification processing based on real-space sensing according to this disclosure. This figure shows an example of point cloud data measured by a 3D sensor according to this disclosure. This figure shows an example of point cloud data whose spatial attribute is determined to be "free" according to this disclosure. This figure shows an example of point cloud data whose spatial attribute is determined to be "occupied" or "free" being superimposed. This flowchart shows the flow of estimation processing according to this disclosure. This figure shows an example of comparing a 3D model using a distance transformation field with the spatial attribute "occupied" according to this disclosure with 3D sensor data. This figure shows an example of comparing a 3D model using a distance transformation field with the spatial attribute "free" according to this disclosure with 3D sensor data. This figure shows an example of grid search according to this disclosure.This figure shows an example of a top view of a 3D model according to this disclosure. This figure shows a comparison of point cloud data measured by a 3D sensor in related technology and data after matching the 3D model with the point cloud data. This figure shows a comparison of point cloud data measured by a 3D sensor according to this disclosure and data after matching the 3D model with the point cloud data. This figure is for explaining the concept of error in matching 3D sensor data and 3D models in related technology. This figure is for explaining the concept of matching 3D sensor data and 3D models according to this disclosure. This is a block diagram showing the hardware configuration of the information processing device according to this disclosure.

[0013] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant explanations will be omitted where necessary for clarity.

[0014] (Embodiment 1) Figure 1 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 estimates the orientation of an object using a three-dimensional model including the object and three-dimensional sensor data measured by a three-dimensional sensor (not shown). In the following description, the "position and orientation" of the object to be estimated may be simply referred to as "orientation". The information processing device 1 is implemented using one or more computer devices. The information processing device 1 comprises a first identification unit 11, a second identification unit 12, and an estimation unit 13. The first identification unit 11, the second identification unit 12, and the estimation unit 13 may each be used as means for identifying and estimating information or data.

[0015] The first identification unit 11 identifies a space in three-dimensional space where no arbitrary object exists, based on a three-dimensional model including the object, as the first region. The "object" is the object whose position and orientation are to be estimated by the estimation unit 13. The "three-dimensional model" is data that defines the three-dimensional shape of the object. The "three-dimensional model" may be, for example, a set of three-dimensional coordinates of multiple representative points of the object. The "three-dimensional space" is a space in which a certain range is defined in an arbitrary three-dimensional coordinate system. The "three-dimensional space" may correspond to a virtual space or a real space. An arbitrary "object" includes the object and "an object integrated with the object," etc. An "object integrated with the object" is, for example, an object that is not subject to position and orientation estimation. In addition, an arbitrary "object" may include points other than the object that can be measured by a three-dimensional sensor located around the object. A "point that can be measured by a three-dimensional sensor" may be, for example, the ground around the object. "A space in three dimensions where no object exists" refers to a space in a three-dimensional model where no object, including the object in question, is defined. Therefore, the "first region" is at least a part of the three-dimensional space containing the three-dimensional model where no three-dimensional shape or other elements are defined. Alternatively, "a space in three dimensions where no object exists" may also include the space of a set of points, etc., between the measurement position (measurement source) and the measured point (measurement point) of a virtual three-dimensional sensor in a virtual space.

[0016] The second identification unit 12 identifies a second region as the space within the measurement target space that does not include the measurement point, based on the 3D sensor data obtained by the 3D sensor of the object. The 3D sensor is a sensor that can measure the distance from the sensor's installation position to the measurement point. The 3D sensor may also measure the positional relationship between the installation position (measurement source) and the measurement point, as well as the 3D coordinates of a predetermined coordinate system, based on the measured distance. "Measurement target space" refers to the 3D space that is the object to be measured by the 3D sensor. "Space within the measurement target space that does not include the measurement point" refers to the space of a set of points, etc., between the measurement source and the measurement point, when an arbitrary object is measured as a measurement point by the 3D sensor within the measurement target space.

[0017] The estimation unit 13 estimates the posture of the object based on the first comparison result of the first region and the second region. For example, the estimation unit 13 may estimate the posture of the object by adjusting the first region and the second region to match based on the first comparison result.

[0018] Figure 2 is a flowchart illustrating the flow of the information processing method. First, the first identification unit 11 identifies a space in three-dimensional space where no arbitrary object exists, based on a three-dimensional model including the object, as the first region (S1). Next, the second identification unit 12 identifies a space in the measurement target space where no arbitrary object exists, based on three-dimensional sensor data measured by a three-dimensional sensor of the object, as the second region (S2). Subsequently, the estimation unit 13 estimates the pose of the object based on the first comparison result of the first region and the second region (S3). Note that the processing order of steps S1 and S2 is not limited to this.

[0019] Here, the first region is often larger than the region in which the object exists in three-dimensional space. Similarly, the second region is often larger than the region in which the object was measured within the measurement target space. Therefore, even if the amount of three-dimensional sensor data is insufficient, a sufficient amount of data can be used for comparison by using the first and second regions. Thus, this embodiment makes it possible to improve the accuracy of estimating the orientation of an object using a three-dimensional model of the object and three-dimensional sensor data.

[0020] The information processing device 1 includes a processor, memory, and storage device, although these are not shown in the diagram. The storage device stores a computer program, for example, that implements the processing of the information processing method shown in Figure 2. The processor then loads the computer program from the storage device into the memory and executes the computer program. In this way, the processor realizes the functions of the first identification unit 11, the second identification unit 12, and the estimation unit 13.

[0021] Alternatively, each component of the information processing device 1 may be implemented with dedicated hardware. Furthermore, some or all of the components of each device may be implemented by general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be configured as a single chip or as multiple chips connected via a bus. Some or all of the components of each device may be implemented by a combination of the aforementioned circuits, etc., and programs. Additionally, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), quantum processor (quantum computer control chip), etc., can be used as the processor.

[0022] Furthermore, if some or all of the components of the information processing device 1 are realized by multiple information processing devices or circuits, these multiple information processing devices or circuits may be centrally located or distributed. For example, the information processing devices or circuits may be realized in a form in which each is connected via a communication network, such as a client-server system or a cloud computing system. Also, the functions of the information processing device 1 may be provided in SaaS (Software as a Service) format.

[0023] (Embodiment 2) Figure 3 is a block diagram showing the overall configuration of the information processing system 1000. The information processing system 1000 comprises a dump truck 31, construction equipment 32, a 3D (Three-Dimensional) sensor 20, and an information processing device 100. The dump truck 31 comprises a driver's cab 311 and a vessel 312. The vessel 312 is either the movable cargo bed itself or an example of a "box for containing cargo" attached to the cargo bed. The construction equipment 32, the 3D sensor 20, and the information processing device 100 are each connected to communicate via a network N. Here, the network N is a communication network including a wireless network. The network N may be, for example, the Internet or a dedicated line network. For example, it is desirable that the construction equipment 32 and the 3D sensor 20 communicate with the information processing device 100 via wireless communication through the network N.

[0024] The construction machine 32 is an excavation machine that loads soil and other materials into the vessel 312 using a hydraulic excavator or the like. The construction machine 32 is equipped with an autonomous control device that performs autonomous control. The construction machine 32 may receive remote control instructions from the information processing device 100 or the like via the network N and perform autonomous control based on the instructions. In particular, the construction machine 32 may receive notification of estimated posture information of the vessel 312 from the information processing device 100 or the like via the network N and perform autonomous control based on the received information. Here, the notified estimated posture information includes rotation and translation values ​​indicating the position and posture of the vessel 312 estimated by the information processing device 100, as well as three-dimensional coordinates, etc. Therefore, the construction machine 32 is an example of an object that can be autonomously controlled by the information processing device 100. Furthermore, the construction machine 32 may perform autonomous control based on instructions from other information processing devices, based on the estimated posture information estimated by the information processing device 100.

[0025] The 3D sensor 20 is an example of the three-dimensional sensor described above. The 3D sensor 20 is a sensor capable of measuring the positional information of an object in three-dimensional space (measurement space). Examples of the 3D sensor 20 include, but are not limited to, LiDAR (Light Detection and Ranging), RGB (Red-Green-Blue) cameras, and RGB-D (RGB and Depth) cameras. In the following description, LiDAR will be used as the 3D sensor 20. Therefore, the 3D sensor 20 is a sensor capable of measuring distance data from the measurement source, which is the installation location, to the measurement point. Specifically, the 3D sensor 20 irradiates laser light from the measurement source, receives reflected light from the object, measures the time required from irradiation to reception, and measures the distance and direction to the object, the three-dimensional coordinates of the object in a predetermined coordinate system, etc., as three-dimensional sensor data. Furthermore, the predetermined coordinate system may be a sensor coordinate system with the 3D sensor 20 (or its installation position) as the origin, or another coordinate system with an arbitrary location in the measurement space as the origin. Also, the 3D sensor 20 may be mounted on the construction machine 32. For example, the construction machine 32 may be attached to any location on a backhoe such as a hydraulic excavator. Furthermore, the information processing system 1000 may use two or more 3D sensors 20.

[0026] The information processing device 100 is an example of the information processing device 1 described above. The information processing device 100 is a computer device for estimating the attitude of a vessel 312, which is an example of an object. Therefore, the information processing device 100 may also be called an attitude estimation device. Furthermore, the information processing device 100 may be implemented as a computer system in which the functions are distributed or made redundant by multiple computer devices.

[0027] Figure 4 is a block diagram showing the configuration of the information processing device 100. The information processing device 100 includes a storage unit 110, a sensing simulation unit 121, a 3D model spatial analysis unit 122, a sensor data acquisition unit 123, a sensor data spatial analysis unit 124, a posture candidate calculation unit 125, a posture estimation unit 126, and a notification unit 127. The sensing simulation unit 121 and the 3D model spatial analysis unit 122 are examples of the first identification unit 11 described above. The sensor data acquisition unit 123 and the sensor data spatial analysis unit 124 are examples of the second identification unit 12 described above. The posture candidate calculation unit 125 and the posture estimation unit 126 are examples of the estimation unit 13 described above.

[0028] The storage unit 110 includes, for example, a non-volatile storage device such as flash memory and a memory such as RAM (Random Access Memory), i.e., a volatile storage device. The storage unit 110 stores at least the 3D model 111 and the 3D sensor data 112.

[0029] The 3D model 111 is data that defines the three-dimensional shape of an object, including the ground area surrounding the object. In this embodiment, the object is a vessel 312. Therefore, the 3D model 111 includes three-dimensional shape data of the dump truck 31, including the vessel 312 and the driver's seat 311, and three-dimensional shape data of the ground area (height information, etc.) within a predetermined range around the dump truck 31. However, the angle and orientation of the vessel 312 may change depending on how the dump truck 31 is stopped and tilted. Therefore, the three-dimensional shape data of the vessel 312 in the 3D model 111 may be compared with the 3D sensor data 112 independently of the other three-dimensional shape data.

[0030] The 3D sensor data 112 includes measurement data from the 3D sensor 20 and measurement data from a virtual 3D sensor obtained through sensing simulation in a virtual space (virtual 3D sensor data), which will be described later. The 3D sensor data 112 is point cloud data such as the distance, direction, and three-dimensional coordinates from the measurement source to the object (after the laser light has been reflected). Furthermore, the 3D sensor data 112 is a collection of data measured from the 3D sensor 20 and from the installation positions of multiple virtual 3D sensors.

[0031] The sensing simulation unit 121 simulates the measurement (sensing) of an object by a 3D sensor in a virtual space, which is a three-dimensional space corresponding to the real space in which the object exists. Specifically, the sensing simulation unit 121 reads a 3D model 111 from the storage unit 110 and places the 3D model 111, including the object, at a predetermined location in the virtual space. Then, the sensing simulation unit 121 places multiple virtual 3D sensors at multiple different locations in the virtual space and acquires virtual 3D sensor data of the object measured by each virtual 3D sensor. Here, a "virtual 3D sensor" is a software module that realizes sensing capabilities equivalent to the 3D sensor 20 described above in the virtual space. Note that the sensing simulation unit 121 only needs to acquire virtual 3D sensor data of the object measured by at least one virtual 3D sensor.

[0032] The 3D model spatial analysis unit 122 analyzes a first region, which is a space in the virtual space where no arbitrary object may exist, and a third region, which is a space where an arbitrary object exists. Specifically, based on the 3D model 111, the 3D model spatial analysis unit 122 identifies a space in the three-dimensional space where no arbitrary object exists as the first region. Furthermore, based on the 3D model 111, the 3D model spatial analysis unit 122 identifies a space in the three-dimensional space where an arbitrary object exists as the third region. Specifically, based on the virtual three-dimensional sensor data and the three-dimensional model 111, the 3D model spatial analysis unit 122 identifies the space between the virtual three-dimensional sensor and an arbitrary object in the virtual space as at least the first region. Furthermore, based on the virtual three-dimensional sensor data and the three-dimensional model 111, the 3D model spatial analysis unit 122 identifies a space in the virtual space where an arbitrary object exists as the third region. In particular, the 3D model spatial analysis unit 122 identifies the first and third regions based on multiple virtual 3D sensor data and the 3D model 111. By combining multiple virtual 3D sensor data and determining the presence or absence of an object from various directions, even if the presence or absence of an object cannot be determined from virtual 3D sensor data measured from one direction, it may be possible to determine the presence or absence of an object using virtual 3D sensor data measured from another direction. Therefore, the accuracy of identifying the first region is improved. Furthermore, the 3D model spatial analysis unit 122 identifies the space between the virtual 3D sensors in the virtual space and the ground region included in the 3D model 111 as the first region, based on the 3D model 111. This allows the first region to be identified not only as the space between the measurement source and the object, but also as the space around the object between the measurement source and the ground. Therefore, the accuracy of identifying the first region is further improved.

[0033] The sensor data acquisition unit 123 acquires 3D sensor data 112 obtained by a 3D sensor 20 installed in the real space which is the measurement target space, which measures the object.

[0034] The sensor data spatial analysis unit 124 analyzes a second region, which is a space in real space where no arbitrary object may exist, and a fourth region, which is a space where any arbitrary object exists. Specifically, based on the 3D sensor data 112, the sensor data spatial analysis unit 124 identifies the space in the measurement target space that does not include the measurement point as the second region. The sensor data acquisition unit 123 also identifies the space in the measurement target space that includes the measurement point as the fourth region, based on the 3D sensor data 112. Specifically, based on the 3D sensor data 112, the sensor data acquisition unit 123 identifies the space between the 3D sensor 20 and the measurement point as at least the second region. Furthermore, based on the 3D sensor data 112, the sensor data spatial analysis unit 124 identifies the space between the 3D sensor 20 in real space and the ground surrounding the object as part of the second region. This further improves the accuracy of identifying the second region, as described above.

[0035] The posture candidate calculation unit 125 uses the 3D model 111 to calculate one or more posture candidates for the object in real space.

[0036] The posture estimation unit 126 estimates the posture of the object based on the first comparison result between the first and second regions. This improves the accuracy of the object's posture estimation because it can take into account measurement errors in the virtual space where any object may not exist and in the real space. Specifically, the posture estimation unit 126 should estimate the object's posture in such a way that the difference between the first and second regions is small. This further improves the estimation accuracy.

[0037] Furthermore, the posture estimation unit 126 may estimate the posture of the object based on the first comparison result and the second comparison result between the third and fourth regions. This allows for consideration of measurement errors between the virtual space and the real space in which any object exists, thereby further improving the accuracy of the object's posture estimation. Specifically, the posture estimation unit 126 may estimate the object's posture in such a way that the difference between the third and fourth regions is minimized. This further improves the estimation accuracy.

[0038] The notification unit 127 notifies the autonomously controlled object, such as the construction machine 32, of the attitude estimated by the attitude estimation unit 126.

[0039] Figure 5 is a flowchart showing the flow of area identification processing based on sensing simulation. First, the sensing simulation unit 121 places virtual 3D sensors at multiple locations in a virtual space including the 3D model 111 and performs a simulation to measure the target object using each virtual 3D sensor (S11).

[0040] Figure 6 is a diagram illustrating the concept of installing multiple virtual 3D sensors in a sensing simulation. In this example, virtual 3D sensors 201 to 213 are installed around the dump truck 31, which is an object placed in the virtual space, at locations in the virtual space that correspond to the assumed locations (assumed locations) where the 3D sensor 20 is assumed to be installed in the real space. Note that the number of virtual 3D sensors and their assumed locations are merely examples. For example, if the 3D sensor 20 is attached to the backhoe of the 3D sensor 20, the assumed location may be above the vessel 312.

[0041] The sensing simulation unit 121 then acquires multiple virtual 3D sensor data measured by each virtual 3D sensor (S12). Figure 7 shows an example of point cloud data measured by a virtual 3D sensor. In this example, point cloud data PD1 is virtual 3D sensor data measured by a dump truck 31 when a virtual 3D sensor is installed at the measurement source P0 in the virtual space. Figure 7 shows an example of a two-dimensional visualization of the distribution of point cloud data PD1 in a top view with the dump truck 31 in the virtual space at the center.

[0042] Next, for each installation position, the 3D model space analysis unit 122 determines the spatial attribute for each voxel constituting the virtual space from each virtual 3D sensor data (S13). Specifically, the 3D model space analysis unit 122 generates a depth image from an arbitrary viewpoint in the virtual space from the virtual 3D sensor data acquired in step S12. Note that the technique according to the present disclosure does not necessarily generate a depth image. FIG. 8 is a diagram showing an example of a depth image DIMG generated from point cloud data. The depth image DIMG shows an example of an image from a viewpoint above the dump truck 31 in the virtual space at the center. Note that the 3D model space analysis unit 122 may generate a depth image from the viewpoint of the measurement source of each virtual 3D sensor.

[0043] Then, the 3D model space analysis unit 122 determines the spatial attribute for each voxel constituting the virtual space based on the depth image from the viewpoint of the measurement source of each virtual 3D sensor. Here, the "spatial attribute" is information indicating the attribute in each set of a plurality of voxels constituting the three-dimensional space. Also, a voxel can be said to be a narrow sense of space. The spatial attributes include, for example, three types: "occupied", "free", and "unknown", but are not limited thereto. Also, the spatial attribute is determined in any three-dimensional space of the virtual space and the real space.

[0044] FIG. 9 is a diagram for explaining the concept of three types of spatial attributes. The upper part of FIG. 9 shows the concept of the spatial attribute SA1 "occupied". "occupied" corresponds to the spatial attributes of the third region and the fourth region described above. That is, "occupied" indicates that an arbitrary object exists in the voxel, in other words, data of the measurement point of the 3D sensor data exists in the voxel. Here, assume that the virtual 3D sensor measures the distance from the measurement source P0 to the measurement point P1 as the measurement distance D1. And when the voxel B1 includes the measurement point P1, the 3D model space analysis unit 122 determines the spatial attribute SA1 of the voxel B1 as "occupied".

[0045] The middle section of Figure 9 shows the concept of the spatial attribute SA2 "free". "Free" corresponds to the spatial attributes of the first and second regions described above. In other words, "free" indicates that there is a possibility that no arbitrary object exists in the voxel, or to put it another way, that there is no data for the measurement point of the 3D sensor data explicitly (clearly) in the voxel (but the measurement point exists beyond the line segment from the measurement source to the voxel). Here, suppose the virtual 3D sensor measures the distance from the measurement source P0 to the measurement point P2 as the measurement distance D2. Then, when the laser light from the virtual 3D sensor passes through voxel B2 and reaches the measurement point P2, there is no arbitrary object in voxel B2 that reflects the laser light. Therefore, if voxel B2 does not include the measurement point P2 and is located between the measurement source P0 and the measurement point P2, the 3D model spatial analysis unit 122 determines that the spatial attribute SA1 of voxel B2 is "free". In the following, a set of voxels with the spatial attribute "free" may be referred to as a "free space."

[0046] The lower part of Figure 9 illustrates the concept of spatial attribute SA3 "unknown". Here, it is assumed that the virtual 3D sensor measures the distance from the measurement source P0 to the measurement point P3 as the measurement distance D3. In other words, if the laser light from the virtual 3D sensor does not reach voxel B3, voxel B3 does not become a measurement point and becomes a point that could not be measured. In this case, the 3D model spatial analysis unit 122 determines the spatial attribute SA3 of voxel B3 to be "unknown". The 3D model spatial analysis unit 122 also determines the spatial attribute to be "unknown" for voxels outside the measurement space.

[0047] FIG. 10 is a diagram for explaining the relationship between the measurement points of sensor data and the three types of spatial attributes of each voxel. Here, k below is an integer of 3 or more. Also, j below is an integer of 1 or more. As described above, assume that the virtual 3D sensor measures the distance from the measurement source P0 to the measurement point P1 as the measurement distance D1. In this case, since the voxel B1 includes the measurement point P1, it is determined as the spatial attribute SA1 "occupied". The voxels B21, ··· B2k−2, B2k−1, B2k are determined as the spatial attribute SA2 "free" because they are the voxels between the measurement source P0 and the measurement point P1. In other words, the voxels B21 to B2k are the voxels on the line segment between the measurement source P0 and the measurement point P1, and are the voxels other than the voxels at both ends of the line segment. The voxels B31 to B3j are the voxels on the extension line of the line segment between the measurement source P0 and the measurement point P1. That is, the voxels B31 to B3j are the voxels in the direction of the measurement point P1 with respect to the measurement source P0 and are the voxels beyond the measurement point P1. Therefore, the voxels B31 to B3j are determined as the spatial attribute SA3 "unknown". As described above, the spatial attribute SA3 "unknown" corresponds not only to the voxels on the extension line of the line segment between the measurement source P0 and the measurement point P1, but also to the voxels outside the measurement space that are in the direction where measurement was not possible.

[0048] After step S13, the 3D model spatial analysis unit 122 sets the spatial attribute to "free" for voxels that are determined to be "free" in any of the determination results from each virtual 3D sensor data (S14). A voxel that is determined to be "free" in any of the determination results from each virtual 3D sensor data is typically a voxel whose spatial attribute is "unknown" in sensing from one direction and whose spatial attribute is "free" in sensing from another direction. For example, when measured from the virtual 3D sensor 201 in Figure 6, it is not possible to determine whether the space hidden by the driver's seat 311 is a vessel 312 or free space. Therefore, in the determination result from the virtual 3D sensor data measured by the virtual 3D sensor 201, the spatial attribute of the group of voxels hidden by the driver's seat 311 may be determined to be "unknown". On the other hand, the virtual 3D sensor 209 in Figure 6 is installed on approximately the opposite side of the dump truck 31 from the virtual 3D sensor 201. Therefore, when the dump truck 31 is measured from the virtual 3D sensor 209, the 3D model spatial analysis unit 122 can distinguish and determine the spatial attributes of the voxel group determined to be "unknown" from the measurement results of the virtual 3D sensor 201 into "occupied" and "free". In other words, the spatial attribute of the voxel corresponding to the vessel 312 is determined to be "occupied". And the spatial attribute of the voxel group between the virtual 3D sensor 209 and the vessel 312 is determined to be "free". Therefore, if different spatial attributes, "free" and "unknown," are determined for the same voxel from virtual 3D sensor data obtained from different virtual 3D sensors, the 3D model spatial analysis unit 122 adopts "free" and confirms (determines) the spatial attribute of the voxel as "free." The 3D model spatial analysis unit 122 then associates the determined (confirmed) spatial attribute with each voxel and stores it in the storage unit 110 or the like.

[0049] After step S14 in Figure 5, the 3D model spatial analysis unit 122 extracts a set of voxels with the spatial attribute "occupied" (S15). In other words, the 3D model spatial analysis unit 122 identifies the extracted voxel set as a third region based on the virtual 3D sensor data. The 3D model spatial analysis unit 122 may also calculate the distance conversion field for "occupied" in the extracted voxel set for each installation location.

[0050] Here, the distance transformation field is information that sets the distance of each voxel in a predetermined range of voxels to a voxel of a specific spatial attribute. By using the distance transformation field, the error calculation process described later can be sped up.

[0051] Figure 11 shows an example of a distance conversion field for a spatial attribute "occupied". The voxel set BS1 shows the extracted voxels with the spatial attribute "occupied" in black. The distance conversion field DTF1 contains information where, for each voxel, the distance between each voxel and the voxel with the spatial attribute "occupied" is set as a numerical value. Specifically, in the distance conversion field DTF1, the value of the voxel with the spatial attribute "occupied" is set to "0", and the values ​​of the other voxels are set to the distance (number of voxels) to the voxel with the spatial attribute "occupied". Note that the distance value does not have to be a discrete integer value; it can also be a continuous value.

[0052] Figure 12 shows examples of images of a voxel set with the spatial attribute "occupied" and a distance transformation field. The image of the voxel set BS1 is an example of an image of voxels with the spatial attribute "occupied". The image of the distance transformation field DTF1 is an example of an image where the distance (numerical value) of each voxel is represented as a pixel value. Note that the multidirectional lines in both images are noise information from the imaging tool and do not represent the actual spatial attribute or distance values. The same applies to subsequent examples.

[0053] Furthermore, after step S14 in Figure 5, the 3D model spatial analysis unit 122 extracts a set of voxels with the spatial attribute "free" (S16). In other words, the 3D model spatial analysis unit 122 identifies the extracted set of voxels as the first region based on the virtual 3D sensor data.

[0054] Figure 13 shows an example of point cloud data where the spatial attribute is determined to be "free". In this example, point cloud data PD2 represents a set of voxels with the spatial attribute "free" determined from virtual 3D sensor data measured by a virtual 3D sensor installed at a measurement source P0 in the virtual space. Figure 13 shows an example of a two-dimensional visualization of the distribution of point cloud data PD2 with the spatial attribute "free" in a top view with the dump truck 31 in the virtual space at the center. Figure 14 shows an example of overlaying point cloud data PD3 where the spatial attribute is determined to be "occupied" or "free". Point cloud data PD3 is an example of overlaying point cloud data PD1 from Figure 7 and point cloud data PD2 from Figure 13.

[0055] Furthermore, after step S17 in Figure 5, the 3D model spatial analysis unit 122 may calculate a distance conversion field (free distance conversion field) for "free" in the extracted voxel set for each installation position. Figure 15 shows an example of a distance conversion field for a spatial attribute of "free". The voxel set BS2 shows the extracted voxels with the spatial attribute "free" in shaded areas. The distance conversion field DTF2 is information in which the distance between each voxel and the voxel with the spatial attribute "free" is set as a numerical value. Specifically, in the distance conversion field DTF2, the value of the voxel with the spatial attribute "free" is set to "0", and the values ​​of the other voxels are set to the distance (number of voxels) to the voxel with the spatial attribute "free".

[0056] Figure 16 shows an example of an image of a voxel set with the spatial attribute "free" and a distance transformation field. The image of the voxel set BS2 is an example of an image of voxels with the spatial attribute "free". Object region A21 indicates the region corresponding to the object. The image of the distance transformation field DTF2 is an example of an image where the distance (numerical value) of each voxel is used as the pixel value. Object region A22 indicates the region corresponding to the object.

[0057] Figure 17 is a flowchart showing the flow of area identification processing based on real-space sensing. First, the sensor data acquisition unit 123 acquires 3D sensor data 112 measured by the dump truck 31 using the 3D sensor 20 (S21). Figure 18 is a diagram showing an example of point cloud data measured by the 3D sensor. In this example, the point cloud data PD4 is the 3D sensor data measured by the dump truck 31 when the 3D sensor 20 is installed at the measurement source P4 in real space. Figure 18 then shows an example of a two-dimensional visualization of the distribution of the point cloud data PD4 in a top view with the dump truck 31 in real space at the center.

[0058] Next, the sensor data spatial analysis unit 124 determines the spatial attributes of each voxel that constitutes the measurement target space from the 3D sensor data (S22). Specifically, the sensor data spatial analysis unit 124 determines the spatial attributes of each voxel in the same manner as in step S13 described above.

[0059] After step S22, the sensor data spatial analysis unit 124 extracts a set of voxels with the spatial attribute "occupied" (S23), similar to step S15 described above. In other words, the sensor data spatial analysis unit 124 identifies the extracted set of voxels as the fourth region based on the 3D sensor data.

[0060] Furthermore, after step S22 in Figure 17, the sensor data spatial analysis unit 124 extracts a set of voxels with the spatial attribute "free" (S24), similar to step S17 described above. In other words, the sensor data spatial analysis unit 124 identifies the extracted set of voxels as a second region based on the 3D sensor data.

[0061] Figure 19 shows an example of point cloud data where the spatial attribute is determined to be "free". In this example, point cloud data PD5 represents a set of voxels with the spatial attribute "free" determined from 3D sensor data measured by a 3D sensor 20 installed at a measurement source P4 in real space. Figure 19 also shows an example of a two-dimensional visualization of the distribution of point cloud data PD5 with the spatial attribute "free" in a top view centered on a dump truck 31 in real space. Figure 20 shows an example of overlaying point cloud data PD6 where the spatial attribute is determined to be "occupied" or "free". Point cloud data PD6 is an example of overlaying point cloud data PD4 from Figure 18 and point cloud data PD5 from Figure 19.

[0062] Figure 21 is a flowchart showing the flow of the estimation process. First, the attitude estimation unit 126 obtains a set of voxels for each spatial attribute extracted by the region identification process based on the sensing simulation (S31). Specifically, the attitude estimation unit 126 may obtain the set of voxels for the spatial attributes "occupied" and "free" extracted based on the 3D model in steps S15 and S16 of Figure 5. The attitude estimation unit 126 may also obtain multiple sets of two types (spatial attributes "occupied" and "free") of distance conversion fields calculated for each installation position in the region identification process based on the sensing simulation in Figure 5. Furthermore, the attitude estimation unit 126 obtains a set of voxels for each spatial attribute extracted by the region identification process based on real-space sensing (S32). Specifically, in steps S23 and S24 of Figure 17, the posture estimation unit 126 may obtain the respective voxel sets for the spatial attributes "occupied" and "free" extracted based on the 3D sensor data.

[0063] After steps S31 and S32, the attitude estimation unit 126 calculates the distance (error Eo) between the sensing simulation result and the real-space sensing result for a set of voxels with the spatial attribute "occupied" (S33). For example, the attitude estimation unit 126 compares the position of each voxel in a set of voxels with the spatial attribute "occupied" obtained in step S31 with the position of each voxel with the spatial attribute "occupied" obtained in step S32 and calculates the error Eo. Alternatively, for example, the attitude estimation unit 126 takes sensor data (set of voxels) based on real-space sensing as input and calculates the distance to the data (3D model) from the sensing simulation using each distance conversion field in the spatial attribute "occupied". Specifically, the attitude estimation unit 126 may calculate the error Eo as the average value per voxel from the sum of the values ​​of voxels corresponding to the sensor data in the distance conversion field. Furthermore, the attitude estimation unit 126 uses a distance conversion field as an example to obtain the shortest distance between the 3D model and the sensor data. The attitude estimation unit 126 uses the comparison result with the smallest error among multiple comparison results as the second comparison result between the third and fourth regions.

[0064] Figure 22 shows an example of comparing a 3D model with 3D sensor data using a distance transformation field with the spatial attribute "occupied". For simplification, Figure 22 shows only one face of the voxel set and omits the depth direction. This example shows that the average distance, which is the error of the spatial attribute "occupied" between the sensing simulation and real-world spatial sensing, is 0.833.

[0065] Furthermore, after steps S31 and S32, the attitude estimation unit 126 calculates the distance (error Ef) between the sensing simulation result and the real-space sensing result for the voxel set with the spatial attribute "free" (S34). For example, the attitude estimation unit 126 compares the position of each voxel in the voxel set with the spatial attribute "free" obtained in step S31 with the position of each voxel in the voxel set with the spatial attribute "free" obtained in step S32 and calculates the error Ef. Alternatively, for example, the attitude estimation unit 126 takes a voxel set in free space based on real-space sensing as input and uses each distance conversion field in the spatial attribute "free" to calculate the distance to the free space data (calculated from the 3D model) obtained by the sensing simulation. The attitude estimation unit 126 uses the distance conversion field as an example to obtain the shortest distance between the free spaces calculated from the 3D model and the sensor data, respectively. Furthermore, the attitude estimation unit 126 selects the comparison result with the smallest error among multiple comparison results as the first comparison result between the first region and the second region.

[0066] Figure 23 shows an example of comparing a 3D model with 3D sensor data using a distance transformation field with the spatial attribute "free". Note that, as with Figure 22 above, the depth direction is omitted in Figure 23. This example shows that the average distance, which is the error between the sensing simulation and the real-world spatial sensing with the spatial attribute "free", is 0.6.

[0067] Furthermore, the posture candidate calculation unit 125 calculates one or more posture candidates for the object in real space (S35). For example, the posture candidate calculation unit 125 calculates the posture of the object from one or more installation positions when the 3D model 111 is placed in real space, for example, candidate sets of three-dimensional coordinates.

[0068] After steps S33, S34, and S35, the attitude estimation unit 126 minimizes the score obtained based on the two errors Eo and Ef (S36). Here, the function for calculating the score is denoted as f(Eo, Ef). For example, the score calculation function f(Eo, Ef) may be defined as Eo + α(Ef * Ef), where α is a predetermined coefficient. However, the score calculation function f is not limited to this.

[0069] Then, the attitude estimation unit 126 substitutes the errors Eo and Ef calculated in steps S33 and S34 into the score calculation function f(Eo, Ef) to calculate a score as the solution. At this time, the attitude estimation unit 126 finds the attitude of the object (rotation R and translation T) when the score is minimized. Here, various methods can be applied to the minimization method. For example, grid search in 6-axis space (rotation + translation) or ICP (Iterative Closest Points) may be used as the minimization method.

[0070] Figure 24 shows an example of grid search. Point cloud data PD7 is 3D sensor data obtained when a 3D sensor is installed at the measurement source P5 in three-dimensional space, and the dump truck 31 is measured. Figure 24 shows an example of a two-dimensional visualization of the distribution of point cloud data PD7 in a top view with the dump truck 31 in three-dimensional space at the center. Bounding box BD is a representation of multiple bounding boxes arranged in the grid search from a top view.

[0071] The attitude estimation unit 126 uses grid search to place multiple bounding boxes of the same size as the distance conversion field and evaluates each one. The attitude estimation unit 126 then evaluates the translation T at the position of the bounding box. The attitude estimation unit 126 also evaluates the rotation R by comparing it with multiple distance conversion fields at different angles.

[0072] The attitude estimation unit 126 then estimates the attitude candidate closest to the minimized score as the attitude of the object (S37). The notification unit 127 then notifies the autonomously controlled object of the estimated attitude (S38). For example, the notification unit 127 may transmit the estimated attitude information, including the estimated attitude, to the construction machine 32 via the network N. As a result, the construction machine 32 can accurately grasp the attitude of the vessel 312 based on the estimated attitude information and perform autonomous control.

[0073] Next, the effects of the technology described herein will be detailed. As a premise, Figure 25 is a diagram showing an example of a top view of the 3D model 111. The 3D model 111 includes the driver's cab area 1111 and the vessel area 1112 of the dump truck 31, as well as the ground area 1110 surrounding the dump truck 31. By including the ground area 1110 in the 3D model 111, the accuracy of free space determination can be improved.

[0074] Figure 26 shows a comparison of data after matching point cloud data PD1 measured by a 3D sensor 20 in the related technology with a 3D model 111. Point cloud data PD1 is the same as in Figure 8 above, and for illustrative purposes, the actual vessel region A30 is enclosed by a dashed line. Point cloud data PD8 shows the result of estimating the attitude (position from the top surface) of the construction machine 32 by matching the 3D model 111 with point cloud data PD1 using the related technology. For illustrative purposes, the estimated vessel region A31 in point cloud data PD8 is enclosed by a dashed line. The comparison result between point cloud data PD1 and PD8 shows that there is an error ER1 between the vessel region A30 and the estimated vessel region A31.

[0075] In contrast, Figure 27 is a diagram showing a comparative example of data after matching point cloud data PD1 measured by the 3D sensor 20 in the technology of this disclosure with the 3D model 111. Point cloud data PD9 shows the result of estimating the attitude (position from the top surface) of the construction machine 32 by matching the 3D model 111 with point cloud data PD1 using the information processing device 100 in the disclosure described above. For illustrative purposes, the estimated vessel region A32 in point cloud data PD9 is enclosed by a dashed line. The comparison result between point cloud data PD1 and PD9 shows that there is an error ER2 between vessel region A30 and estimated vessel region A32. Therefore, by comparing Figure 26 and Figure 27, it is shown that the error ER1 has been reduced to an error ER2 by the technology of this disclosure.

[0076] Next, the concept of the technology relating to this disclosure will be illustrated and explained. Figure 28 is a diagram illustrating the concept of the error in matching 3D sensor data 112 and 3D model 111 in the related technology. In the related technology, it is assumed that the 3D sensor 20 measures measurement points P11, etc., from the measurement source P0, while non-measurement points P12, etc., are not measured. In this case, the related technology recognizes the measurement area A40 as the object from the 3D sensor data 112 from the 3D sensor 20. However, the measurement area A40 indicates that only a part of the actual 3D model 111 could be measured. Therefore, when the 3D model 111 is matched based on the measurement area A40, estimated point P21 included in estimated area A41 and estimated point P22 included in estimated area A42 can be identified as points corresponding to measurement point P11. Therefore, when the 3D model 111 is matched based on the measurement area A40, it will match to either part of estimated area A41 or A42, making it difficult to identify the estimated area. In other words, the accuracy of estimating the position and orientation of the object is insufficient due to a lack of 3D sensor data 112.

[0077] In contrast, Figure 29 is a diagram illustrating the concept of matching 3D sensor data 112 and 3D model 111 in the technology of this disclosure. In the technology of this disclosure, the 3D sensor 20 and virtual 3D sensor 201, etc., measure measurement points P11, etc. and measurement point P13 from the measurement source P0, while non-measurement points P12, etc., are not measured. Here, measurement point P13 is the ground around the object, etc., and corresponds to an object other than the object. The information processing device 100 of this disclosure recognizes the measurement area A40 as the object from the 3D sensor data 112. At this time, the information processing device 100 determines the free space P15 between the measurement source P0 and measurement point P11, and the free spaces P16 and P14 between the measurement source P0 and measurement point P13. The information processing device 100 then recognizes the free space measurement area A43 including the free spaces P15 and P16. The information processing device 100 also recognizes the free space P14 within the measurement area A40. Therefore, the information processing device 100 matches the 3D model 111 and the free space estimation area A45 based on the measurement area A40 and the free space measurement area A43. In this case, the measurement point P11 is identified as the estimation point P22, the free space P14 as the free space P17, the free space P15 as the free space P18, and the free space P16 as the free space P19. In other words, the 3D model 111 and the free space estimation area A45 can be identified. Therefore, the technology according to this disclosure can improve the accuracy of estimating the position and orientation of an object because the number of matching target data increases compared to related technologies.

[0078] Furthermore, the technology disclosed herein uses a non-learning-type pose estimation method, eliminating the need for learning from field data, thus facilitating its adoption. Moreover, by employing a point cloud matching method as the non-learning-type pose estimation technique, the technology disclosed herein can improve estimation accuracy compared to feature matching methods, even when the structure of the 3D sensor data and the 3D model do not match. In this embodiment, using LiDAR as the 3D sensor for the point cloud matching method results in smaller measurement errors and a longer sensing distance compared to stereo cameras, making it suitable for outdoor use. However, LiDAR has a lower resolution (sparser measurement data) compared to stereo cameras, meaning that data measured from a single LiDAR (mounted on a backhoe) can only measure a portion (one side) of the dump. In other words, a single 3D sensor can only measure a portion of the target object. Therefore, if the amount of 3D sensor data is small, there is a possibility of matching parts of the 3D model that differ from the target object, potentially leading to larger pose estimation errors. Furthermore, while point cloud matching has a smaller estimation error compared to feature matching, it suffers from the problem of poor configuration due to a lack of essential data. Therefore, the pose estimation accuracy may be insufficient in some cases.

[0079] In contrast, the technology disclosed herein uses the comparison results of free-space point cloud data when comparing 3D model data including the object with 3D sensor data. This reduces the error in estimating the object's posture and improves posture estimation accuracy, even when there is limited 3D sensor data for measurement points. Furthermore, since the technology disclosed herein can be implemented by adding a free-space matching method to a point cloud matching method using a 3D sensor, as described in related technologies, it is easy to introduce and offers improved accuracy.

[0080] Furthermore, the method for calculating the error in the technology disclosed herein is not limited to the method using the distance conversion field described above. For example, the attitude estimation unit 126 calculates the shortest distance (e.g., three-dimensional coordinates) between each point (a) of the 3D sensor data A and each point (b) of the 3D model B, and then calculates the average distance between them, and the error E Ａ,ＢYou may calculate it. In that case, for example, you may use either of the following two formulas. Note that N below is the total number of points a.

[0081] (Other Embodiments) Figure 30 is a block diagram showing the hardware configuration of the information processing device 100. The information processing device 100 corresponds to the information processing device 1 and information processing device 100 described above. The information processing device 100 includes a memory 101, a processor 102, and a network interface 103.

[0082] Memory 101 is composed of a combination of volatile memory and non-volatile memory. Volatile memory is, for example, a volatile storage device such as RAM, and is a storage area for temporarily holding information during the operation of the processor 102. Non-volatile memory is, for example, a non-volatile storage device such as a hard disk or flash memory. Memory 101 stores at least a computer program on which the processing of the information processing method in the information processing device 100 according to this disclosure is implemented. Memory 101 may also include storage located away from the processor 102. In this case, the processor 102 may access memory 101 via an I / O (Input / Output) interface, which is not shown.

[0083] The processor 102 is a control device that controls each component of the information processing device 100. The processor 102 reads and executes software (computer programs) from the memory 101. In this way, the processor 102 implements the functions of the sensing simulation unit 121, the 3D model spatial analysis unit 122, the sensor data acquisition unit 123, the sensor data spatial analysis unit 124, the attitude candidate calculation unit 125, the attitude estimation unit 126, and the notification unit 127. That is, the processor 102 performs the processing of the information processing method in the information processing device 100 according to this disclosure. The processor 102 may be, for example, a microprocessor, an MPU (Multi Processing Unit), or a CPU (Central Processing Unit). Furthermore, the processor 102 may include multiple processors.

[0084] The network interface 103 may be used to communicate with network nodes. The network interface 103 may include, for example, a network interface card (NIC) compliant with IEEE 802.3 series. IEEE stands for Institute of Electrical and Electronics Engineers. The network interface 103 may also include wireless LAN (Local Area Network), wired LAN, Wi-Fi®, Bluetooth®, etc.

[0085] Furthermore, the technology disclosed herein is applicable to attitude estimation of objects other than the vessel 312 of a dump truck 31 used for autonomous control of construction machinery. Moreover, the technology disclosed herein is applicable not only to autonomous control but also to various technological fields that estimate the attitude of an object measured by a three-dimensional sensor.

[0086] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0087] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments, rather than being associated with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps described in any of the drawings may be changed as appropriate.

[0088] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note A1) An information processing device comprising: a first identification means for identifying a space in a three-dimensional space in which no arbitrary object exists as a first region, based on a three-dimensional model including an object; a second identification means for identifying a space in a measurement target space in which no measurement points are not included as a second region, based on three-dimensional sensor data obtained by measuring the object with a three-dimensional sensor; and an estimation means for estimating the posture of the object based on a first comparison result between the first region and the second region. (Note A2) The information processing device according to Note A1, wherein the first identification means identifies a space in the three-dimensional space in which no arbitrary object exists as a third region, based on the three-dimensional model; the second identification means identifies a space in the measurement target space in which measurement points are included as a fourth region, based on the three-dimensional sensor data; and the estimation means estimates the posture of the object based on the first comparison result and a second comparison result between the third region and the fourth region. (Note A3) The information processing apparatus according to Note A1 or A2, wherein the first identifying means identifies the space between the virtual 3D sensor and the object as at least the first region based on virtual 3D sensor data obtained by measuring the object with a virtual 3D sensor installed in the virtual space which is the 3D space and the 3D model, and the second identifying means identifies the space between the 3D sensor and the measurement point as at least the second region based on 3D sensor data obtained by measuring the object with a 3D sensor installed in the real space which is the measurement target space. (Note A4) The information processing apparatus according to Note A3, wherein the first identifying means identifies the first region based on a plurality of virtual 3D sensor data obtained by measuring the object with a plurality of virtual 3D sensors installed at different positions in the virtual space and the 3D model.(Note A5) The information processing apparatus according to Note A3 or A4, wherein the three-dimensional model further includes the ground region surrounding the object, the first identifying means identifies the first region based on the three-dimensional model, including the space between the virtual three-dimensional sensor in the virtual space and the ground region, and the second identifying means identifies the second region based on the three-dimensional sensor data, including the space between the three-dimensional sensor in the real space and the ground surrounding the object. (Note A6) The information processing apparatus according to Note A1, wherein the estimation means estimates the posture of the object so as to reduce the difference between the first region and the second region. (Note A7) The information processing apparatus according to Note A2, wherein the estimation means estimates the posture of the object so as to reduce the difference between the third region and the fourth region. (Note B1) An information processing system comprising a three-dimensional sensor and an information processing device, wherein the information processing device comprises: a first identification means for identifying a space in three-dimensional space where no object exists as a first region based on a three-dimensional model including an object; a second identification means for identifying a space in a measurement target space where no object exists as a second region based on three-dimensional sensor data measured by the three-dimensional sensor of the object; and an estimation means for estimating the posture of the object based on a first comparison result of the first region and the second region. (Note C1) An information processing method comprising a computer identifying a space in three-dimensional space where no object exists as a first region based on a three-dimensional model including an object; identifying a space in a measurement target space where no object exists as a second region based on three-dimensional sensor data measured by the three-dimensional sensor of the object; and estimating the posture of the object based on a first comparison result of the first region and the second region.(Note D1) An information processing program that causes a computer to perform the following: a first identification process that identifies a space in a three-dimensional space where no arbitrary object exists as a first region, based on a three-dimensional model including the object; a second identification process that identifies a space in a measurement target space where no arbitrary object exists as a second region, based on three-dimensional sensor data obtained by measuring the object with a three-dimensional sensor; and an estimation process that estimates the pose of the object based on a first comparison result of the first region and the second region.

[0089] Some or all of the elements (e.g., configuration and function) described in Appendices A2 to A7 that are dependent on Appendice A1 {e.g., device} may also be dependent on Appendices B1 {e.g., system}, C1 {e.g., method}, and D1 {e.g., program} in the same way as those described in Appendices A2 to A7. Some or all of the elements described in any appendice may be applied to various hardware, software, recording means for recording software, systems, and methods.

[0090] This application claims priority based on Japanese Patent Application No. 2024-168142, filed on 27 September 2024, and incorporates all of its disclosures herein.

[0091] 1 Information processing device, 11 First identification unit, 12 Second identification unit, 13 Estimation unit, 1000 Information processing system, 20 3D sensor, 31 Dump truck, 311 Driver's seat, 312 Vessel, 32 Construction machinery, N Network, 201-213 Virtual 3D sensor, 100 Information processing device, 110 Storage unit, 111 3D model, 112 3D sensor data, 121 Sensing simulation unit, 122 3D model spatial analysis unit, 123 Sensor data acquisition unit, 124 Sensor data spatial analysis unit, 125 Pose candidate calculation unit, 126 Pose estimation unit, 127 Notification unit, DIMG Depth image, PD1-PD9 Point cloud data, P0, P4, P5 Measurement source, P1, P2, P3 Measurement point, D1 Measurement distance, D2, D3 Measured distance, B1, B2, B21, B2k-2, B2k-1, B2k, B3, B31, B3j Voxel, SA1, SA2, SA3 Spatial attribute, BS1, BS2 Voxel set, DTF1, DTF2 Distance transformation field, A21, A22 Object region, BD Bounding box, 1110 Ground region, 1111 Driver's seat region, 1112 Vessel region, A30 Vessel region, A31, A32 Estimated Vessel region, ER1, ER2 Error, P11 Measured point, P12 Non-measured point, P13 Measured point, P14, P15, P16, P17, P18, P19 Free space, P21 Estimated point, P22 Estimated point, A40 Measured region, A41 Estimated region, A42 Estimation area, A43 Free space measurement area, A44 Estimation area, A45 Free space estimation area, 101 Memory, 102 Processor, 103 Network interface

Claims

1. An information processing device comprising: a first identification means for identifying a space in three-dimensional space where no object exists, based on a three-dimensional model including an object, as a first region; a second identification means for identifying a space in the measurement target space that does not include a measurement point, based on three-dimensional sensor data obtained by measuring the object with a three-dimensional sensor, as a second region; and an estimation means for estimating the orientation of the object based on a first comparison result of the first region and the second region.

2. The information processing apparatus according to claim 1, wherein the first identification means identifies a space in the three-dimensional space in which an arbitrary object exists as a third region based on the three-dimensional model, the second identification means identifies a space in the measurement target space in which a measurement point is included as a fourth region based on the three-dimensional sensor data, and the estimation means estimates the posture of the object based on the first comparison result and a second comparison result between the third region and the fourth region.

3. The information processing apparatus according to claim 1 or 2, wherein the first identifying means identifies the space between the virtual 3D sensor and the object as at least the first region based on virtual 3D sensor data obtained by measuring the object with a virtual 3D sensor installed in the virtual space which is the 3D space and the 3D model, and the second identifying means identifies the space between the 3D sensor and the measurement point as at least the second region based on 3D sensor data obtained by measuring the object with a 3D sensor installed in the real space which is the measurement target space.

4. The information processing apparatus according to claim 3, wherein the first identifying means identifies a first region based on a plurality of virtual three-dimensional sensor data obtained by measuring the object with a plurality of virtual three-dimensional sensors installed at different locations in the virtual space and the three-dimensional model.

5. The information processing apparatus according to claim 3, wherein the three-dimensional model further includes a ground area surrounding the object, the first identifying means identifies the first area based on the three-dimensional model, including the space between the virtual three-dimensional sensor in the virtual space and the ground area, and the second identifying means identifies the second area based on the three-dimensional sensor data, including the space between the three-dimensional sensor in the real space and the ground surrounding the object.

6. The information processing apparatus according to claim 1, wherein the estimation means estimates the posture of the object in such a way that the difference between the first region and the second region is minimized.

7. The information processing apparatus according to claim 2, wherein the estimation means estimates the posture of the object in such a way that the difference between the third region and the fourth region is minimized.

8. An information processing system comprising a three-dimensional sensor and an information processing device, wherein the information processing device includes: a first identification means for identifying a space in three-dimensional space where no object exists, based on a three-dimensional model including an object, as a first region; a second identification means for identifying a space in a measurement target space where no object exists, based on three-dimensional sensor data measured by the three-dimensional sensor of the object, as a second region; and an estimation means for estimating the orientation of the object based on a first comparison result of the first region and the second region.

9. The information processing system according to claim 8, wherein the first identification means identifies a space in the three-dimensional space in which an arbitrary object exists as a third region based on the three-dimensional model, the second identification means identifies a space in the measurement target space in which an arbitrary object is included as a fourth region based on the three-dimensional sensor data, and the estimation means estimates the posture of the object based on the first comparison result and the second comparison result between the third region and the fourth region.

10. The information processing system according to claim 8 or 9, wherein the first identifying means identifies the space between the virtual 3D sensor and the object as at least the first region, based on virtual 3D sensor data obtained by measuring the object with a virtual 3D sensor installed in the virtual space which is the 3D space and the 3D model, and the second identifying means identifies the space between the 3D sensor and the arbitrary object as at least the second region, based on 3D sensor data obtained by measuring the object with a 3D sensor installed in the real space which is the measurement target space.

11. The information processing system according to claim 10, wherein the first identifying means identifies a first region based on a plurality of virtual three-dimensional sensor data obtained by measuring the object with a plurality of virtual three-dimensional sensors installed at different locations in the virtual space and the three-dimensional model.

12. The information processing system according to claim 10, wherein the three-dimensional model further includes a ground area surrounding the object, the first identifying means identifies the first area based on the three-dimensional model, including the space between the virtual three-dimensional sensor in the virtual space and the ground area, and the second identifying means identifies the second area based on the three-dimensional sensor data, including the space between the three-dimensional sensor in the real space and the ground surrounding the object.

13. The information processing system according to claim 8, wherein the estimation means estimates the posture of the object in such a way that the difference between the first region and the second region is minimized.

14. The information processing system according to claim 9, wherein the estimation means estimates the posture of the object in such a way that the difference between the third region and the fourth region is minimized.

15. An information processing method comprising: a computer identifying a space in a three-dimensional space where no object exists, based on a three-dimensional model including the object, as a first region; identifying a space in a measurement target space where no object exists, based on three-dimensional sensor data measured by the object using a three-dimensional sensor, as a second region; and estimating the orientation of the object based on a first comparison result of the first region and the second region.

16. The information processing method according to claim 15, wherein, based on the three-dimensional model, a space in which an arbitrary object exists within the three-dimensional space is identified as a third region, based on the three-dimensional sensor data, a space in which an arbitrary object is included within the measurement target space is identified as a fourth region, and the orientation of the object is estimated based on the first comparison result and the second comparison result between the third region and the fourth region.

17. The information processing method according to claim 15 or 16, wherein, based on virtual 3D sensor data obtained by measuring the object with a virtual 3D sensor installed in the virtual space which is the 3D space, and the 3D model, the space between the virtual 3D sensor and the object is identified as at least the first region, and based on the 3D sensor data obtained by measuring the object with a 3D sensor installed in the real space which is the measurement target space, the space between the 3D sensor and the arbitrary object is identified as at least the second region.

18. The information processing method according to claim 17, which identifies the first region based on a plurality of virtual three-dimensional sensor data obtained by a plurality of virtual three-dimensional sensors installed at different locations in the virtual space, and the three-dimensional model.

19. The information processing method according to claim 17, wherein the three-dimensional model further includes a ground area surrounding the object, and based on the three-dimensional model, the space between the virtual three-dimensional sensor in the virtual space and the ground area is identified as the first region, and based on the three-dimensional sensor data, the space between the three-dimensional sensor in the real space and the ground surrounding the object is identified as the second region.

20. An information processing program that causes a computer to perform the following: a first identification process that identifies a space in a three-dimensional space where no arbitrary object exists as a first region, based on a three-dimensional model including the object; a second identification process that identifies a space in a measurement target space where no arbitrary object exists as a second region, based on three-dimensional sensor data obtained by measuring the object with a three-dimensional sensor; and an estimation process that estimates the pose of the object based on a first comparison result of the first region and the second region.

Images