Spatial model construction system

Abstract

The purpose of the present invention is to provide a spatial model construction system that makes it possible to construct a spatial model for accurately recognizing the shape, the size, or the position of a target object in real space. A sensor device 1 acquires image sensor data consisting of a point cloud in real space, and transmits the image sensor data to a management device 2. After converting the image sensor data consisting of the point cloud in the real space into a plurality of items of voxel grid data consisting of voxels of different scales and then setting a prescribed weight for a voxel in which the point cloud of the image sensor data is present, the management device 2 aggregates weights of each voxel of each item of voxel grid data. The management device 2 calculates the likelihood of at least one of the size, the shape, or the position of a target object in the real space on the basis of the result of aggregation of the weights of the voxel.

Description

[Technical Field]

[0001] This invention relates to a spatial model construction system that constructs a spatial model based on image sensor data from real space. [Background technology]

[0002] In recent years, smart cities have attracted attention as a way to create cities that address social issues by utilizing ICT (Information and Communication Technology) to improve the quality of life, promote economic circulation through the creation of new value, and solve social problems. A smart city is an urban area that efficiently monitors and manages infrastructure and resources using information collected by various sensors and devices.

[0003] Furthermore, robotic services in smart cities can meet a variety of needs by providing services in industry, healthcare, greening, and other sectors. A vision system for building real-world models is crucial in realizing these smart city robotic services.

[0004] For example, in agricultural harvesting, if a vision system cannot accurately detect and locate crops, the robot will be unable to harvest them. Furthermore, obstacles such as branches can obstruct the robot's path during harvesting, potentially leading to crop failure or damage to the robot's manipulator. This is true not only in agriculture but also in various other fields such as industry and transportation. Therefore, efficient vision systems are crucial for robots performing tasks in smart cities.

[0005] Incidentally, recently, along with cameras and millimeter-wave radar, sensors called LIDAR (light detection and ranging) have been attracting attention. LIDAR is a type of sensor that uses laser light, and compared to radio waves, it has a higher density of radiant flux. By scanning and irradiating an object with short-wavelength laser light, it can accurately detect not only the distance to the object but also its position and shape. For this reason, when LIDAR is used in real space, three-dimensional information of the real space can be acquired as image sensor data consisting of a point cloud of laser light (see, for example, Non-Patent Document 1).

[0006] Furthermore, as mentioned above, LiDAR uses lasers to detect objects, and therefore cannot cover blind spots behind obstacles. However, by installing multiple LiDARs in different locations and building a 3D sensor network, these blind spots can be covered, making it useful for vision systems. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Shackleton, J., VanVoorst, B., & Hesch, J. (2010, August). Tracking people with a 360-degree LIDAR. In 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance (pp. 420-426). IEEE. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] However, depending on the orientation of the LiDAR and the distance to the object, the image sensor data consisting of acquired point clouds can sometimes be sparse, which can make it difficult to accurately recognize the shape, size, or position of the object.

[0009] This invention has been made in view of the above-mentioned problems, and aims to provide a spatial model construction system that can construct a spatial model for accurately recognizing the shape, size, or position of an object in real space. [Means for solving the problem]

[0010] To achieve the above objective, the present invention provides a spatial model construction system in which one or more sensor devices and a management device are connected in a communication manner, and a spatial model is constructed based on image sensor data of real space acquired by each sensor device, wherein the sensor device comprises a sensor unit that acquires image sensor data consisting of point clouds in real space, and a transmission unit that transmits the image sensor data consisting of point clouds in real space acquired by the sensor unit to the management device, and the management device comprises a receiving unit that receives the image sensor data consisting of point clouds in real space transmitted from each sensor device, and the image sensor data consisting of point clouds in real space received by the receiving unit The system is characterized by comprising: a storage unit for storing data; a data processing unit for converting image sensor data consisting of a point cloud in real space into a plurality of voxel grid data consisting of voxels of different scales; a weight setting unit for setting predetermined weights for voxels in which the point cloud of image sensor data exists in each voxel grid data converted by the data processing unit; a weight aggregation unit for aggregating the weights of each voxel in each voxel grid data set by the weight setting unit at a predetermined position in real space; and a likelihood calculation unit for calculating the likelihood of at least one of the size, shape, or position of an object in real space based on the aggregation result of the voxel weights at the predetermined position in real space aggregated by the weight aggregation unit.

[0011] Further, the management device may include an aggregation unit that aggregates image sensor data composed of a point cloud in the real space.

[0012] Further, the sensor device or the management device may include a filter unit that extracts image sensor data in a predetermined target region of the real space from the image sensor data composed of a point cloud in the real space acquired by the sensor unit.

[0013] Further, the data processing unit may convert the image sensor data composed of a point cloud in the real space into voxel grid data composed of two-dimensional or three-dimensional voxels in a grid pattern of a predetermined size.

[0014] Further, the data processing unit may convert the image sensor data composed of a point cloud in the real space into voxel grid data composed of two-dimensional or three-dimensional voxels of a predetermined size centered on a predetermined coordinate point.

[0015] Further, the weight setting unit may set a weight for the voxel based on the density of the point cloud of the image sensor data existing in the voxel.

[0016] Further, the weight setting unit may set a weight for the voxel by adding a predetermined weight to each point cloud of the image sensor data existing in the voxel. [[ID=二十二]]

[0017] Further, the weight aggregation unit may aggregate the weights of each voxel by adding the weights of each voxel of each voxel grid data set by the weight setting unit at a predetermined position in the real space.

[0018] Further, the weight aggregation unit may aggregate the weights of each voxel by calculating the average value of the weights of each voxel of each voxel grid data set by the weight setting unit at a predetermined position in the real space.

[0019] Further, the likelihood calculation unit may calculate at least one likelihood of the size, shape, or position of the object in the real space in the range of 0 to 1 by normalizing the aggregation result of the weights of each voxel aggregated by the weight aggregation unit.

Advantages of the Invention

[0020] According to the present invention, by introducing the concept of likelihood as the possibility that an object exists at a predetermined position in the real space, a spatial model for accurately recognizing the shape, size, or position of the object in the real space can be constructed. Therefore, by using this spatial construction model, it becomes possible to provide an efficient and safe vision system in various fields such as agriculture, industry, and transportation.

Brief Description of the Drawings

[0021] [Figure 1] It is a diagram showing the overall configuration of the spatial model construction system according to the first embodiment of the present invention. [Figure 2] It is a block diagram showing the configuration of the sensor device in FIG. 1. [Figure 3] It is a block diagram showing the configuration of the management device in FIG. 1. [Figure 4] It is a diagram showing (a) image sensor data after two-dimensional conversion, (b) voxel grid data for each scale, and (c) voxel grid data after likelihood calculation. [Figure 5] It is a block diagram showing the configuration of the management device of the spatial model construction system according to the second embodiment of the present invention. [Figure 6] It is a conceptual diagram showing the aggregation of frames in the management device in FIG. 5. [Figure 7] It is a diagram showing the parameters of the evaluation experiment. [Figure 8] It is a diagram showing the relationship between the sensor device and the object in the evaluation experiment. [Figure 9] It is a diagram showing the experimental field in the evaluation experiment. [Figure 10] It is a diagram showing image sensor data of the object by LIDAR-A, B, and C. [Figure 11] This figure shows the voxel grid data of each scale of the object obtained using LIDAR-A. [Figure 12] This figure shows the spatial model of the object after the likelihood calculation using LIDAR-A, B, and C. [Figure 13] This figure shows the relationship between the true value difference and the likelihood obtained by LIDAR-A, B, and C. [Figure 14] This figure shows voxel grid data according to a third embodiment of the present invention. [Figure 15] This is an external view showing the object used in the simulation. [Figure 16] This figure shows (a) a point cloud data model and (b) a likelihood model of the object in the simulation. [Figure 17] This is a floor plan showing the location (laboratory) where the simulation was performed. [Figure 18] (a) This figure shows the simulation results of the point cloud data model and (b) the likelihood model. [Modes for carrying out the invention]

[0022] <First Embodiment> Next, a first embodiment of the spatial model construction system according to the present invention (hereinafter referred to as "this system") will be described with reference to Figures 1 to 4.

[0023] [Overall structure] As shown in Figure 1, this system comprises a plurality of sensor devices 1, a management device 2 connected to each sensor device 1 via a predetermined communication path, and a robot vision system 3 connected to the management device 2 via a predetermined communication path. After each sensor device 1 transmits image sensor data in real space to the management device 2, the management device 2 constructs a spatial model based on the image sensor data in real space and transmits the spatial model to the robot vision system 3.

[0024] In this embodiment, "real space" refers to a three-dimensional space in which a predetermined object controlled by the robot vision system exists, such as in a farm or factory.

[0025] [Configuration of Sensor Device 1] The sensor device 1 is installed in multiple locations near the robot vision system in a farm or factory, and as shown in Figure 2, it comprises a sensor unit 11 that acquires image sensor data consisting of a point cloud in real space, and a terminal device 12 that performs predetermined processing on the image sensor data consisting of a point cloud in real space acquired by the sensor unit 11 and transmits it to the management device 2.

[0026] The aforementioned sensor unit 11 is a sensor known as LIDAR (light detection and ranging). This LIDAR is a type of sensor that uses laser light, and compared to radio waves, it has a higher density of radiant flux. By scanning an object with short-wavelength laser light, it acquires three-dimensional image sensor data consisting of point clouds in real space.

[0027] Examples of LiDAR systems include those that acquire 360-degree omnidirectional image sensor data by rotating multiple laser emitters, and those that acquire image sensor data by directly irradiating with laser light within a predetermined light irradiation angle range. Furthermore, while the accuracy of the image sensor data increases with the number of laser emitters, this also increases the cost, so inexpensive LiDAR systems with fewer emitters may be used.

[0028] As shown in Figure 2, the terminal device 12 consists of a capture unit 121, a storage unit 122, a filter unit 123, and a transmission unit 124. Of these, the capture unit 121 captures image sensor data consisting of point clouds received from the sensor unit 11 and temporarily stores it in the storage unit 122 at predetermined scan intervals. The filter unit 123 receives image sensor data consisting of point clouds in real space from the storage unit 122 and extracts image sensor data consisting of point clouds in the target area where an object in real space exists. The transmission unit 124 fragments the image sensor data consisting of point clouds in the target area in real space extracted by the filter unit 123 into packets and transmits them to the management device 2.

[0029] In this embodiment, a LIDAR sensor is used as the sensor unit 11, but other sensors may be used as long as they are capable of acquiring image sensor data consisting of point clouds.

[0030] [Configuration of Control Device 2] As shown in Figure 3, the management device 2 includes a receiving unit 21 that receives image sensor data consisting of point clouds in a target area in real space transmitted from each terminal device 12, a storage unit 22 that stores the image sensor data consisting of point clouds in a target area in real space received by the receiving unit 21, and a construction unit 23 that constructs a spatial model based on the image sensor data consisting of point clouds in a target area in real space stored in the storage unit 22.

[0031] The construction unit 23 includes a data processing unit 231 that converts image sensor data consisting of a point cloud in a target area in real space into a plurality of voxel grid data, a weight setting unit 232 that sets predetermined weights for each voxel in each voxel grid data, a weight aggregation unit 233 that aggregates the weights of the voxels in each voxel grid data, and a likelihood calculation unit 234 that calculates the likelihood (likelihood of the object existing) of at least one of the size, shape, or position of an object in the target area in real space based on the aggregation result of the voxel weights.

[0032] As shown in Figure 4(a), the data processing unit 231 converts the three-dimensional image sensor data, which consists of point clouds in the target area in real space acquired by each sensor device 1, into two-dimensional image sensor data by projecting it in the depth direction onto a two-dimensional space consisting of the X and Y axes.

[0033] Specifically, the data processing unit 231 converts two-dimensional image sensor data consisting of a point cloud in a target region in real space into multiple voxel grid data consisting of voxels of different scales. In this embodiment, as shown in Figure 4(b), the data processing unit 231 converts image sensor data consisting of a point cloud in a target region in real space into voxel grid data consisting of a grid of two-dimensional voxels of three different scales: a first scale (large scale: left figure), a second scale (medium scale: middle figure), and a third scale (small scale: right figure). A voxel grid is a two-dimensional space consisting of the X and Y axes that has been voxed at a predetermined scale, and voxel grid data is obtained by plotting image sensor data on this voxel grid. In the voxel grid data, voxels in which image sensor data exists are colored. Also, the voxels of each scale overlap each other.

[0034] The weight setting unit 232 sets a predetermined weight for each voxel grid data converted by the data processing unit 231 for the voxels in which the point cloud of image sensor data exists. This weight setting unit 232 sets a predetermined weight for voxel v of the voxel grid of scale k. ki Weight w for each voxel (where i represents each voxel) ki This is determined from the spatial characteristics of each voxel, such as the density of the point cloud of the image sensor data. For example, voxel v ki If the image sensor data contains multiple point clouds, the number of points in the image sensor data is counted to determine the weight w ki One option is to set the voxel v ki The number of points in the image sensor data included in the point cloud is divided by the volume or area of ​​the voxels to obtain the point cloud density, thereby creating weights w kiSet it.

[0035] The weight aggregation unit 233 aggregates the weights of each voxel of each voxel grid data set by the weight setting unit 232 at a predetermined position in the real space. In the present embodiment, the weight aggregation unit 233 aggregates the weights set for each voxel of the voxel grid data of the first scale, the second scale, and the third scale at a predetermined position in the target region of the real space. For example, at a predetermined position c in the target region of the real space, the weight of the voxel of the voxel grid of the first scale is w 1c , the weight of the voxel of the voxel grid of the second scale is w 2c , and the weight of the voxel of the voxel grid of the third scale is w 3c . In this case, the value obtained by adding the weights of each voxel at the predetermined position c (w 1c + w 2c + w 3c ) is used as the aggregation result of the weights. Alternatively, the average value obtained by dividing the value obtained by adding the weights of each voxel at the predetermined position c by the number (n) of voxels existing at the predetermined position c (w 1c + w 2c + w 3c / n) is used as the aggregation result of the weights.

[0036] The likelihood calculation unit 234 calculates at least one likelihood of the size, shape, or position of the object in the predetermined region of the real space based on the aggregation result of the weights of the voxels aggregated by the weight aggregation unit 233. In the present embodiment, the likelihood calculation unit 234 estimates that the higher the aggregation value of the weights of the voxels at a predetermined position in the real space, the higher the likelihood that the object exists in the target region of the real space, and thus calculates the likelihoods of the size, shape, and position of the object in the real space. For example, as shown in FIG. 4(c), the likelihood calculation unit 234 normalizes the aggregation result of the weights of the voxels at a predetermined position to calculate at least one likelihood of the size, shape, or position of the object in the range of 0 to 1 (0 or more and 1 or less), and the higher the likelihood, the darker the color is. Note that the likelihood calculation unit 234 may calculate the value itself of the aggregation result of the weights by the weight aggregation unit 233 as the likelihood.

[0037] In this way, by introducing the concept of likelihood as the probability of an object existing at a predetermined location in real space, it is possible to construct a spatial model for accurately recognizing the shape, size, or position of an object in real space. Therefore, by using this spatial model at a predetermined location in real space, it becomes possible to provide an efficient robot vision system 3 in various fields such as agriculture, industry, and transportation.

[0038] <Second Embodiment> Next, a second embodiment of the present invention will be described, mainly with reference to Figures 5 and 6. Components identical to those in the first embodiment will be given the same reference numerals, and their descriptions will be omitted.

[0039] In this embodiment, the construction unit 23 includes an aggregation unit 235 that aggregates image sensor data consisting of point clouds in a target area in real space, which is stored in the storage unit 22, as shown in Figure 5. This aggregation unit 235 aggregates the image sensor data of the target area in real space transmitted from each sensor device 1 into multiple frames. One frame refers to a collection of image sensor data acquired by the sensor device 1 in a single scan.

[0040] To explain in more detail, as shown in Figure 6, the number of point clouds of image sensor data that can be acquired in just one frame from sensor device 1 is small. When the voxel scale is large, a rough estimate can be made, but when the voxel scale is small, gaps appear, and even when likelihood is used, the shape may appear incomplete.

[0041] In contrast, by aggregating 2 or 3 frames, the number of point clouds from the image sensor data increases, and the missing parts in the shape calculated using likelihood are gradually filled in. Furthermore, aggregating 4 frames increases the number of point clouds from the image sensor data even more, allowing us to estimate that the overall shape is rounded while also estimating the possibility that some corners exist, thereby improving the accuracy of the shape calculated using likelihood.

[0042] For example, in a situation where a robotic arm grasps an object, if the number of frames is small and the shape is incomplete using likelihood estimation, there is a possibility of grasping failure. However, with a large number of frames and improved likelihood accuracy, by controlling the robotic arm while estimating the shape without corners and taking into account the possibility of corners, it becomes possible to quickly approach the object and, while confirming the shape using the robotic arm's tactile sensors to determine whether or not there are corners, it becomes possible to quickly and reliably grasp the object.

[0043] Alternatively, when an autonomous vehicle passes a pillar, the distance the vehicle can get close to the pillar will vary depending on whether the pillar is cylindrical or rectangular. However, if the number of image sensor data points is small and the frames are not aggregated, the shape may be estimated to have no corners, and there is a possibility that the vehicle will make contact when passing. In this respect, if the number of frames is large and the likelihood of the shape is improved, the possibility of corners can also be estimated, and by slowing down when the vehicle approaches and acquiring more image sensor data over time, it becomes possible to pass by the object quickly and reliably.

[0044] <Third Embodiment> Next, a third embodiment of the present invention will be described, mainly with reference to Figure 14. Components identical to those in the first embodiment will be given the same reference numerals, and their descriptions will be omitted.

[0045] In this embodiment, the configuration is the same as in the first embodiment shown in Figure 3, and image sensor data consisting of a point cloud in real space is converted into voxel grid data consisting of two-dimensional or three-dimensional voxels of a predetermined size centered on predetermined coordinate points of the image sensor data.

[0046] Specifically, as shown in Figure 14, the data processing unit 231 in the management device converts image sensor data consisting of a point cloud in real space into voxel grid data consisting of three-dimensional voxels of a predetermined size centered on predetermined coordinate points (grid points) in real space. For example, at the upper right coordinate point (grid point) g1, a voxel (dotted line) of a predetermined size centered on the predetermined coordinate point (grid point) g1 is set. Similarly, for the other coordinate points (grid points) g2, g3, and g4, voxels (dotted lines) of a predetermined size centered on the predetermined coordinate points (grid points) g2, g3, and g4 are also set. Note that the voxels centered on each coordinate point (grid point) overlap with adjacent ones.

[0047] Furthermore, the weight setting unit 232 sets a predetermined weight for each voxel containing point clouds of image sensor data in the voxel grid data of the sphere converted by the data processing unit 231. For example, a voxel centered on the upper right coordinate point (grid point) g1 contains 6 point clouds of image sensor data, so the weight of the voxel (6 / V1) is set by dividing the number of point clouds of image sensor data in that voxel (6) by the volume of the voxel (V1). Similarly, a voxel centered on the lower left coordinate point (grid point) g2 contains 3 point clouds of image sensor data, so the weight of the voxel (3 / V2) is set by dividing the number of point clouds of image sensor data in that voxel (3) by the volume of the voxel (V2).

[0048] Furthermore, the weight aggregation unit 233 aggregates the weights of each voxel in each voxel grid data set by the weight setting unit 232 at a predetermined position in real space. For example, at the lower left coordinate point (grid point) g2, the voxels of the sphere at the upper right coordinate point (grid point) g1 and the voxels of the sphere at the lower left coordinate point (grid point) g2 overlap. Therefore, the weight of the lower left coordinate point (grid point) g2 is aggregated by adding the weight of the voxel of the sphere at the upper right coordinate point (grid point) g1 and the weight of the voxel of the sphere at the lower left coordinate point (grid point) g2, or by calculating the average value of the weights of the voxels of the sphere at the upper right coordinate point (grid point) g1 and the weight of the voxel of the sphere at the lower left coordinate point (grid point) g2.

[0049] Furthermore, the likelihood calculation unit 234 calculates the likelihood of at least one of the size, shape, or position of an object in a predetermined region of real space, based on the aggregated weights of each voxel of the sphere aggregated by the weight aggregation unit 233 at a predetermined position in real space.

[0050] Although embodiments of the present invention have been described above with reference to the drawings, the present invention is not limited to the illustrated embodiments. Various modifications and variations can be made to the illustrated embodiments within the same scope as the present invention, or within the equivalent scope. [Examples]

[0051] <Example 1> Next, the methods and results of the evaluation experiment of this system will be described below.

[0052] (1) Experiment setup This scenario assumes that the robot vision system 3 uses three LIDAR sensors to capture a ball as an object. The parameters for this experiment are shown in Figure 7. The system for this experiment was implemented based on a system consisting of three sets of LIDAR sensors A, B, and C (sensor unit 11), a terminal device 12, and a management device 2. As shown in Figure 8, the distance from the LIDAR sensor to the object was set to 1 m, and the angle between the LIDAR sensors was set to an equal 120 degrees so that the entire object was covered by at least one LIDAR sensor. As shown in the experimental field in Figure 9, the floor of the experimental field was covered with carpet, and the height of each LIDAR sensor from the floor was set to 150 cm. The diameter of the sphere used as the object was 202.7 mm. Under the above experimental setup, frames of point cloud image sensor data from each LIDAR sensor were recorded.

[0053] (2) Pretreatment Frames of image sensor data consisting of point clouds acquired by each LIDAR sensor were used. The binary format of the point cloud image sensor data was converted to the PCD format used by the open3D library. Image sensor data that was clearly irrelevant to spatial evaluation, such as point cloud image sensor data of the floor, was removed using the RANSAC algorithm. Finally, for simplification, the 3D PCD data was converted to 2D PCD data by projection in the depth direction from each LIDAR sensor.

[0054] (3) Metrics and benchmarks For evaluation purposes, the actual diameter of the object was used as the true value. The difference between the true and estimated diameters was used as the evaluation metric. The estimated diameter was calculated as follows: First, the pair of voxel centers furthest from each other in the voxel space was extracted. Next, the distance between the two central voxels was measured as the maximum distance in the voxel space. This maximum distance was defined as the estimated diameter. The proposed method of the present invention was compared with a benchmark using a uniform voxel size. The voxel scales were set to 1 mm, 5 mm, 10 mm, 15 mm, and 20 mm.

[0055] (4) Spatial model of the proposed method of the present invention in evaluation This paper describes the evaluation of a spatial model using the proposed method of the present invention. One frame of image sensor data consisting of point clouds acquired using a LIDAR sensor was used. Figure 10 shows images of the image sensor data consisting of point clouds obtained using LIDAR sensors A, B, and C.

[0056] The methodology of the proposed method of the present invention, as described in (2) above, is applied. The proposed method uses voxels of five scales: 1 mm, 5 mm, 10 mm, 15 mm, and 20 mm. Figure 11 shows the voxelized images of the data obtained from LIDAR sensor A at each scale. In this figure, black voxels contain points, and white voxels do not contain points.

[0057] The likelihood is calculated using a voxel grid for all scales. i Voxel V of each voxel grid data i The weights are as follows: In this evaluation, V i If a dot is included in w i = 1, if no points are included, w i = 0. Then, the likelihood of each position in real space is calculated using the method of the first embodiment.

[0058] Figure 12 visualizes the spatial likelihood distribution for each LIDAR sensor A, B, and C. The maximum and minimum likelihood values ​​are 1.0 and 0.0, respectively. In Figure 12, darker colors indicate a high likelihood (probability) of the object (ball) being present. As can be seen from Figure 12, different likelihoods are mixed together. In the region where the original point exists, as seen in Figure 12, the likelihood is high. Furthermore, even in the region where no point is captured in Figure 12, the actual shape of the object is interpolated.

[0059] (5) Results Figure 13 plots the estimated diameter results obtained by the proposed method of the present invention. The X-axis in Figure 13 represents the likelihood of the proposed method, and the Y-axis represents the difference between the estimated diameter and the true value. The results of a benchmark method using a uniform voxel scale are also shown as a dashed line plot.

[0060] As seen in Figure 13(a), in the benchmark method, setting a small voxel scale reduces the difference from the true value. However, as shown in Figure 11, setting a small voxel scale results in a shape that differs significantly from the original ball. In the proposed method of the present invention, when the likelihood of the region is 1.0, the difference from the true value approaches zero. Furthermore, as shown in Figure 12, the proposed method of the present invention successfully generates the original shape (spatial model) of the object. In Figure 13(b), essentially the same observations as in Figure 13(a) are observed. However, in Figure 13(c), which shows the results for LIDAR sensor C, the benchmark method with a small voxel scale does not necessarily result in a small difference from the true value. For example, the 20mm benchmark method performs better than the 15mm benchmark method. This suggests that even though the likelihood of the proposed method of the present invention performs best among the results of the three LIDAR sensors, the minimum voxel scale is not necessarily the optimal scale for minimizing estimation errors.

[0061] <Example 2> Next, the evaluation experiment of this system's simulation is described below.

[0062] [Simulation Overview] • Create a point cloud data model and likelihood model (invention) of the image sensor data of the object. • Perform pathfinding using each model on a map recreated within the inventor's laboratory. • Move each model, which is a reproduction of the target object, along the path. • Measure the number of collisions between each model and obstacles in the simulation.

[0063] [Simulation experiment location] • Shibaura Institute of Technology Toyosu Campus, Research Building, 14th Floor, Room 14Q32 (See Figure 17)

[0064] [Object] NOAA MOBILE-X (see Figure 15) • Dimensions: Length 850mm, Width 530mm, Height 850mm ·Forward speed: ~6km (legal speed), backward speed: ~2km ·Wheel 165mm

[0065] [How to obtain data] Four LiDAR (Livox Avia) units were placed around the target object. • Point cloud data of the target object is acquired using LiDAR. • Integrate point cloud data from four LiDAR sensors to create each model.

[0066] [Two models to compare in simulation] As shown in Figure 16, a point cloud data model and likelihood model of the object obtained using the LiDAR were created. As shown in Figure 16(a), the point cloud data model of the object is sparse, and in particular, there are areas where the shape of the edges is missing. As shown in Figure 16(b), the likelihood model for the object indicates that areas with darker black areas have a low likelihood (high probability of the object existing) and areas with lighter black areas have a high likelihood (high probability of the object existing). Furthermore, there are no missing parts in the shape, starting with the edges of the object, and there are parts where the shape is completed.

[0067] [Results of the travel path in the simulation] When the movement path was explored in the simulation, the point cloud data model took the movement path (polyline) shown in Figure 18(a), and the likelihood model took the movement path (polyline) shown in Figure 18(b). The shaded area within the frame represents the movable region calculated by each model, the polyline within the frame represents the movement path of each model (the upper right is the starting point), and the multiple square frames (gray) represent obstacles.

[0068] In the point cloud data model, as shown in Figure 18(a), the object passed through a position close enough to come into contact with the obstacle, with a travel distance of 8.14 m and 10 collisions with the obstacle.

[0069] On the other hand, the likelihood model, as shown in Figure 18(b), assumes that the vehicle passes through the obstacle at a sufficient distance, with a travel distance of 8.81 mm and zero collisions with the obstacle.

[0070] Therefore, while the point cloud data model has a wide area of ​​movement, it also has many collisions and safety issues. On the other hand, the likelihood model has a limited area of ​​movement, and the movement distance is slightly longer, but it can be confirmed that there are no collisions and safety is improved.

[0071] The verification results suggest that the point cloud data model repeatedly collided with obstacles because the computer misinterpreted the presence of actual object edges as nonexistent due to the sparse point cloud data. On the other hand, the likelihood model allowed the computer to adequately recognize the object's edges, including interpolating their shape, enabling safe passage without collisions. As a result, using the likelihood model according to the present invention makes it possible to provide efficient and safe vision systems in various fields such as agriculture, industry, and transportation. [Explanation of Symbols]

[0072] 1...Sensor device 11...Sensor unit 12…Terminal device 121... Supplementary section 122...Storage Unit 123...Filter section 124...Transmitter 2…Management device 21... Receiver 22...Storage Unit 23...Construction Department 231...Data Processing Unit 232...Weight setting section 233...Weight Calculation Unit 234... Likelihood calculation unit 235... Consolidation Department 3…Robot vision system

Claims

1. A spatial model construction system is provided in which one or more sensor devices and a management device are connected in a communication-enabled manner, and a spatial model is constructed based on real-space image sensor data acquired by each sensor device. The aforementioned sensor device is A sensor unit that acquires image sensor data consisting of point clouds in real space, The system comprises a transmission unit that transmits image sensor data consisting of a point cloud in real space acquired by the sensor unit to the management device, The aforementioned control device is A receiving unit that receives image sensor data consisting of point clouds in real space transmitted from each of the aforementioned sensor devices, A storage unit for storing image sensor data consisting of a point cloud in real space received by the receiving unit, A data processing unit that converts image sensor data consisting of point clouds in real space into multiple voxel grid data consisting of voxels of different scales, A weight setting unit sets a predetermined weight for each voxel grid data converted by the data processing unit, where a point cloud of image sensor data exists in the voxel grid data. A weight aggregation unit aggregates the weights of each voxel in each voxel grid data set by the weight setting unit at a predetermined position in real space, A spatial model construction system characterized by comprising: a likelihood calculation unit that calculates the likelihood of at least one of the size, shape, or position of an object in real space based on the aggregated weights of voxels at predetermined positions in real space aggregated by the weight aggregation unit.

2. The spatial model construction system according to claim 1, comprising an aggregation unit for aggregating image sensor data consisting of point clouds in real space.

3. The spatial model construction system according to claim 1, wherein the sensor device or management device comprises a filter unit that extracts image sensor data in a predetermined target area in real space from image sensor data consisting of a point cloud in real space acquired by the sensor unit.

4. The spatial model construction system according to claim 1, wherein the weight setting unit sets a weight for a voxel based on the density of the point cloud of image sensor data present within the voxel.

5. The spatial model construction system according to claim 1, wherein the weight setting unit sets the weight for each point cloud of image sensor data present within the voxel by adding a predetermined weight to the voxel.

6. The spatial model construction system according to claim 1, wherein the weight aggregation unit aggregates the weights of each voxel by adding the weights of each voxel in each voxel grid data set by the weight setting unit at a predetermined position in real space.

7. The spatial model construction system according to claim 1, wherein the likelihood calculation unit calculates the likelihood of at least one of the size, shape, or position of an object in real space in the range of 0 to 1 by normalizing the aggregated weights of each voxel aggregated by the weight aggregation unit.

Images