Information processing device, information processing method, and program

Abstract

An information processing device and the like that generates highly visible three-dimensional information is provided. [Solution] The information processing device 10 includes a data acquisition unit 12a that acquires multiple point cloud data corresponding to multiple sensing operations from LiDARs 21 to 24 that repeatedly sense the surroundings, a moving object detection unit 12b that detects multiple points corresponding to moving objects based on the comparison results of the multiple point cloud data, a moving object removal unit 12c that removes the multiple points corresponding to moving objects from at least one of the multiple point cloud data, and a three-dimensional information generation unit 12d that generates three-dimensional three-dimensional information based on the point cloud data after the multiple points corresponding to moving objects have been removed.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] As a technique for removing noise from point cloud data, for example, the technique described in Patent Document 1 is known. That is, Patent Document 1 describes "an information processing device including a classification unit that classifies three-dimensional point cloud data including distance values ​​to objects in a predetermined region into one or more clusters based on the distance values, and a determination unit that determines the cluster to be adopted based on a distribution of the distance values ​​of the three-dimensional point cloud data included in the classified cluster." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-162856 Summary of the Invention [Problem to be solved by the invention]

[0004] As mentioned above, in Patent Document 1, three-dimensional point cloud data is classified into clusters based on distance values, but it is desirable to generate highly visible three-dimensional information with moving objects such as people, animals, and vehicles removed.

[0005] Therefore, an object of the present disclosure is to provide an information processing device or the like that generates highly visible three-dimensional information. [Means for solving the problem]

[0006] In order to solve the above-described problems, an information processing device according to the present disclosure includes a data acquisition unit that acquires a plurality of point cloud data corresponding to a plurality of sensing operations from a LiDAR that repeatedly senses the surroundings; a moving object detection unit that detects a plurality of points corresponding to a moving object based on a comparison result of the plurality of point cloud data; a moving object removal unit that removes the plurality of points corresponding to the moving object from at least one of the plurality of point cloud data; and a three-dimensional information generation unit that generates three-dimensional three-dimensional information based on the point cloud data after the plurality of points corresponding to the moving object have been removed. The LiDAR is installed at a position higher than the height of an object onto which laser light is irradiated from the LiDAR, and when there are multiple points that do not correspond to each other in a pair of point cloud data that are adjacent in order when the sensing is repeated, the moving object detection unit extracts an area including the multiple points in each of the point cloud data, and when the multiple points included in the area of ​​one of the pair of point cloud data have a smaller depth based on the position of the LiDAR than the multiple points included in the area of ​​the other of the pair of point cloud data, detects the multiple points included in the area of ​​the one of the point cloud data as the moving object. It was decided that. Other details will be explained in the embodiments. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to provide an information processing device and the like that generates highly visible three-dimensional information. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is an explanatory diagram showing an example of a construction site in which a LiDAR is installed in the information processing device according to the first embodiment. [Figure 2] 1 is a functional block diagram of an information processing device according to a first embodiment. [Figure 3] 4 is a flowchart showing the flow of processing in the information processing device according to the first embodiment. [Figure 4A] 3 is an example of point cloud data including a moving object in the information processing device according to the first embodiment. [Figure 4B] 10 is an example of point cloud data that does not include a moving object, in the information processing device according to the first embodiment. [Figure 5] 1 is a diagram illustrating a hardware configuration of an information processing device according to a first embodiment. [Figure 6] FIG. 10 is a functional block diagram of an information processing device according to a second embodiment. [Figure 7] 10 is a flowchart showing the flow of processing by an information processing device according to a second embodiment. [Figure 8]FIG. 10 is an explanatory diagram showing an example of a construction site in which a LiDAR and a visible light camera are installed in an information processing device according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0009] In the following example, point cloud data of a construction site is generated by LiDAR 21-24 (see FIG. 1 ) and used for progress management of the construction site, but the present invention is not limited to this. For example, point cloud data of the interior of a warehouse may be generated by LiDAR and used for material inventory management. Also, point cloud data of infrastructure facilities may be generated by LiDAR and used for maintenance management. Examples of the infrastructure facilities include air conditioning equipment, refrigeration equipment, medical equipment, power transmission equipment, power distribution equipment, substation equipment, communication equipment, gas equipment, water supply equipment, railway tracks and power lines, and roads. In addition, point cloud data of a plant may be generated by LiDAR and used for plant maintenance management. Examples of the plants include power plants, manufacturing plants, chemical plants, and water treatment plants.

[0010] In the following, first, the LiDARs 21 to 24 (see FIG. 1) and the surrounding environment will be briefly described, and then the information processing device 10 (see FIG. 2) according to the embodiment will be described in detail.

[0011] First Embodiment FIG. 1 is an explanatory diagram showing an example of a construction site in which LiDARs 21 to 24 are installed in the information processing device according to the first embodiment. In the example of FIG. 1, four poles P1 extending in the height direction are installed at a construction site. LiDARs 21 to 24 are installed near the top end of each pole P1. The LiDARs 21 to 24 are devices that measure the distance from themselves to each point on the surface of an object by irradiating the object with laser light and detecting the reflected light. The LiDAR 21 is installed at a position higher than the height of the object (irradiated object) onto which the laser light is irradiated from the LiDAR 21. For example, if the object is a sand dune (sand pile) or heavy machinery, the LiDAR 21 is installed in advance at a position higher than the highest point (peak) of the object. This is the same for other types of object, and also for the other LiDARs 22 to 24. The number and arrangement of the LiDARs can be changed as appropriate.

[0012] The LiDARs 21 to 24 may use, for example, a time-of-flight (TOF) method or a frequency-modulated continuous wave (FMCW) method for measurement. The LiDAR 21 shown in FIG. 1 includes a light-emitting unit including a laser diode, a light-receiving unit including an image sensor, and a predetermined signal processing circuit, all of which are not shown. Laser light emitted from the light-emitting unit is reflected by an object, and the reflected light is photoelectrically converted by the light-receiving unit. The signal processing unit generates point cloud data based on a signal obtained by photoelectric conversion of the light-receiving unit. The point cloud data is a collection of data including the distance from the LiDAR 21 to each point on the surface of the object. Each point in the point cloud data is associated with a three-dimensional coordinate value based on the position of the LiDAR 21.

[0013] Also, a mirror (not shown) that reflects laser light from a light emitting unit (not shown) in a predetermined manner may be provided in the LiDAR 21 with a predetermined inclination relative to the horizontal plane. Furthermore, a drive unit (not shown) that continuously changes the irradiation direction of the laser light by rotating this mirror in a predetermined manner (or rotating it within a predetermined range such as an azimuth angle) may be provided in the LiDAR 21. The same applies to the remaining LiDARs 22 to 24.

[0014] In the example of FIG. 1, an area surrounded by four poles P1 at a construction site is the main measurement target, and laser light is emitted from each of the LiDARs 21 to 24. Explaining this in more detail, for example, in the LiDAR 21, sensing using laser light is repeated multiple times at predetermined intervals. In one sensing session, the azimuth direction when the laser light is emitted is continuously changed, and the mirror (not shown) is rotated (or turned) once or multiple times, and the light receiving unit (not shown) is exposed for a predetermined time (for example, several seconds or several tens of seconds). This generates point cloud data of objects present around the LiDAR 21.

[0015] In the LiDAR 21, such sensing is repeated multiple times, and point cloud data is generated for each of the multiple sensing times. In the repeated sensing by the LiDAR 21, the sensing range (angle range irradiated with laser light) for the first, second, third, ... is approximately the same. The same is true for the other LiDARs 22 to 24. In this way, the LiDARs 21 to 24 repeat sensing of their surroundings in a predetermined manner. The point cloud data generated by each of the LiDARs 21 to 24 is transmitted to the information processing device 10 (see FIG. 2) via wired communication or wireless communication.

[0016] It is assumed that the sensing time periods of the LiDARs 21 to 24 overlap. The plurality of point cloud data generated by the LiDARs 21 to 24 through sensing in a common time period is integrated by the information processing device 10 (see FIG. 2).

[0017] FIG. 2 is a functional block diagram of the information processing device 10. As shown in FIG. The information processing device 10 shown in FIG. 2 is a device that generates three-dimensional stereoscopic information based on point cloud data of LiDARs 21 to 24. As shown in FIG. 2, the information processing device 10 includes a storage unit 11, a processing unit 12, and a communication unit 13. The storage unit 11 stores in advance a predetermined program to be executed by the processing unit 12, as well as information such as the identification information, type, specifications, installation position, and installation attitude of the LiDARs 21 to 24. The storage unit 11 also stores point cloud data acquired via the data acquisition unit 12a in association with the identification information and measurement time (exposure time zone) of the LiDARs 21 to 24. The storage unit 11 also stores the calculation results of the processing unit 12 as appropriate.

[0018] The processing unit 12 executes predetermined processing based on the program stored in the storage unit 11. As shown in Fig. 2, the processing unit 12 includes a data acquisition unit 12a, a moving object detection unit 12b, a moving object removal unit 12c, and a three-dimensional information generation unit 12d.

[0019] The data acquisition unit 12a acquires a plurality of point cloud data corresponding to multiple sensing operations from the LiDAR 21 that repeatedly senses the surroundings. Similarly, the data acquisition unit 12a acquires a plurality of point cloud data from each of the other LiDARs 22 to 24. The moving object detection unit 12b detects a plurality of points corresponding to moving objects based on, for example, a comparison result of a plurality of point cloud data corresponding to a plurality of sensing operations by the LiDAR 21 (similarly for the point cloud data of the other LiDARs 22 to 24). Examples of such moving objects include people, animals, and vehicles.

[0020] The moving object removal unit 12c removes, for example, a plurality of points corresponding to moving objects from the point cloud data generated by the LiDAR 21 (the same applies to the point cloud data of the other LiDARs 22 to 24). The stereoscopic information generating unit 12d generates three-dimensional stereoscopic information based on the point cloud data after the plurality of points corresponding to the moving objects have been removed. The communication unit 13 transmits the three-dimensional information generated by the three-dimensional information generation unit 12d to the user's terminal T1 via the network N1. For example, a computer, a smartphone, or a tablet is used as the terminal T1. The three-dimensional information is then displayed in a predetermined manner on the display of the terminal T1.

[0021] FIG. 3 is a flowchart showing the flow of processing by the information processing device (see also FIG. 2 as appropriate). Regarding steps S101 to S107 in FIG. 3, the processing based on the point cloud data generated by the LiDAR 21 will be mainly described, but similar processing will be performed on the point cloud data generated by the other LiDARs 22 to 24.

[0022] In step S101, the information processing device 10 acquires point cloud data from the LiDARs 21 to 24 by the data acquisition unit 12a (data acquisition process). As described above, for example, sensing is repeated multiple times in the LiDAR 21, and multiple pieces of point cloud data corresponding to the multiple sensing operations are generated. The data acquisition unit 12a acquires multiple pieces of point cloud data corresponding to the multiple sensing operations from the LiDAR 21. For example, if sensing is performed three times by the LiDAR 21, three pieces of point cloud data having approximately the same sensing range are acquired from the LiDAR 21.

[0023] In step S102, the information processing device 10 uses the moving object detection unit 12b to compare the point cloud data. For example, the moving object detection unit 12b compares the point cloud data obtained in the first sensing of the LiDAR 21 with the point cloud data obtained in the second sensing. Assume that during the exposure time of the LiDAR 21, no moving objects such as people or vehicles are present in the sensing range, and the object irradiated with the laser light (such as a sand pile or the ground: see FIG. 1) is stationary. In this case, the sensing ranges for the first and second sensing are substantially the same, and therefore the point cloud data obtained in each sensing is also substantially the same.

[0024] On the other hand, if a predetermined moving object is present within the sensing range of the LiDAR 21 during its exposure time, the position of the moving object changes from moment to moment, and the two point cloud data corresponding to the first and second sensing times will be different in the area of ​​the moving object. In this way, for example, a moving object is detected by comparing the two point cloud data obtained in the first and second sensing times. Similarly, the two point cloud data obtained in the second and third sensing times, and the two point cloud data obtained in the third and fourth sensing times are also compared as appropriate.

[0025] That is, the moving object detection unit 12b determines for each point in a pair of point cloud data adjacent in order of repeated sensing whether a point included in one of the point cloud data is included in the other point cloud data. Here, a predetermined point included in one point cloud data and a predetermined point included in the other point cloud data "correspond" to each other means that the three-dimensional distance between the two points is equal to or less than a predetermined value. The predetermined value is a distance threshold that serves as a criterion for determining whether two points correspond to each other, and is set in advance.

[0026] In step S103, the information processing device 10 determines, by the moving object detection unit 12b, whether or not there are multiple points that do not correspond to each other in a pair of point cloud data that are adjacent in the sensing order. For example, when a moving object is present in the sensing range of the LiDAR 21, multiple points that do not correspond to each other in the two point cloud data obtained by the first and second sensing are detected in an area that includes the trajectory of the moving object during the exposure time (for example, area A1 in FIG. 4A).

[0027] If there are no points that do not correspond to each other in step S103 (S103: No), the processing of the information processing device 10 proceeds to step S107. That is, if substantially all points correspond to each other in a pair of point cloud data whose sensing orders are adjacent (S103: No), the processing of the information processing device 10 proceeds to step S107. In this case, there is no particular moving object in either of the pair of point cloud data. Also, if there are no points that do not correspond to each other in step S103 (S103: Yes), the processing of the information processing device 10 proceeds to step S104.

[0028] In step S104, the information processing device 10 causes the moving object detection unit 12b to extract an area including a plurality of points. That is, if there are a plurality of points that do not correspond to each other in a pair of point cloud data that are adjacent in order when sensing is repeated (S103: Yes), the moving object detection unit 12b extracts an area including these a plurality of points from each of the point cloud data. The processing of step S104 will be described with reference to FIGS. 4A and 4B.

[0029] FIG. 4A is an example of point cloud data including a moving object. In the example of FIG. 4A, it is assumed that point cloud data of a person, which is a moving object, and point cloud data of a sand dune and the ground at a construction site have been acquired. Furthermore, it is assumed that the direction of laser light irradiation changes continuously during one sensing session in the LiDAR 21 (see FIG. 1), and the laser light is irradiated multiple times within a predetermined sensing range (the same applies to the other LiDARs 22 to 24). As shown in FIG. 4A, when a walking person is present within the sensing range of the LiDAR 21, the position of the person (the person's trajectory) is detected at each predetermined time interval in one sensing session. Area A1 shown in FIG. 4A is an area that includes multiple points indicating the person's trajectory. In other words, the trajectory of the person walking during the exposure time of the LiDAR 21 is recorded as multiple points included in area A1.

[0030] FIG. 4B is an example of point cloud data that does not include moving objects. It is assumed that the point cloud data of Fig. 4A is generated in the first sensing, and then the point cloud data of Fig. 4B is generated in the second sensing after a predetermined time has elapsed. In the second sensing shown in Fig. 4B, it is assumed that a person moves outside the sensing range. In this case, the laser light passes through area A1 (see Fig. 4A) where the person was present in the first sensing, and is further reflected by the ground and received by the light receiving unit (not shown) of the LiDAR 21, generating multiple points in area A2 (Fig. 4B).

[0031] In this case, the points included in area A1 (see FIG. 4A) in the point cloud data from the first sensing and the points included in area A2 (see FIG. 4B) in the point cloud data from the second sensing do not correspond to each other (S103 in FIG. 3: Yes). In this case, areas A1 and A2 containing these points are extracted from each point cloud data (S104 in FIG. 3).

[0032] Returning to FIG. 3 again, the explanation will be continued. After extracting the area including the plurality of points that do not correspond to each other in step S104, in step S105, the information processing device 10 detects the area with the smaller depth as a moving object by the moving object detection unit 12b. Here, "depth" means the three-dimensional distance between each point of the point cloud data and the LiDAR 21. For example, the closer the distance between the LiDAR 21 that irradiates the laser light and the point on the surface of the object that reflects the laser light, the smaller the depth of the point.

[0033] As described above, the LiDARs 21 to 24 (see FIG. 1) are installed at a higher position than the object (irradiated object) onto which the laser light is incident. In other words, the LiDARs 21 to 24 are installed at a position overlooking the materials and people at the construction site. Therefore, for example, the distance (depth) between the surface of a person in area A1 (see FIG. 4A) and the LiDAR 21 is smaller than the distance (depth) between the ground (the ground onto which the light that has passed through area A1 is incident) and the LiDAR 21. This is because the person is located closer to the LiDAR 21 than the ground.

[0034] 4A and 4B, the depth of each point in area A1 (see FIG. 4A) in the first sensing is smaller than that in area A2 (see FIG. 4B) in the second sensing. In this case, the moving object detection unit 12b detects area A1 (see FIG. 4A) with the smaller depth as a moving object (S105).

[0035] Note that the magnitude of the average depth of a plurality of points included in area A1 (see FIG. 4A) may be compared with the magnitude of the average depth of a plurality of points included in area A2 (see FIG. 4B). In addition, the information processing device 10 may generate a depth image in which the pixel value is the distance (depth) between each point of the point cloud data and the LiDAR 21, and compare the pixel values ​​(depths) of pixels corresponding to areas A1 and A2 (see Figures 4A and 4B) in this depth image.

[0036] In this way, when a plurality of points included in region A1 (see FIG. 4A) of one of a pair of point cloud data whose sensing order is adjacent to each other have a smaller depth based on the position of the LiDAR 21 than a plurality of points included in region A2 (see FIG. 4B) of the other point cloud data, the moving object detection unit 12b detects the plurality of points included in region A1 of the one point cloud data as a moving object (person). In this way, a plurality of points corresponding to a moving object are detected.

[0037] The "moving object detection process" for detecting a plurality of points corresponding to a moving object based on the comparison result of a plurality of point cloud data includes the processes of steps S102 to S105 in FIG.

[0038] Next, in step S106, the information processing device 10 causes the moving object removal unit 12c to remove multiple points corresponding to moving objects from the point cloud data (moving object removal process). In the example of FIGS. 4A and 4B, the multiple points included in region A1 (see FIG. 4A), where the depth of each point relative to the position of the LiDAR 21 is relatively small, are removed. This improves the visibility when a user checks three-dimensional information based on the point cloud data compared to when the trajectory of the moving object remains as is.

[0039] In the case of FIG. 4B, since no moving object is present, there is no particular need to remove the multiple points included in region A2. Incidentally, it is possible that a moving object exists in both of the two point cloud data based on the first and second sensing. In this case, multiple points corresponding to the moving object are removed from each of the pair of point cloud data. In this way, the moving object removal unit 12c removes multiple points corresponding to the moving object from at least one of the multiple point cloud data.

[0040] In step S107, the information processing device 10 causes the three-dimensional information generation unit 12d to take the union of the point cloud data. That is, the three-dimensional information generation unit 12d generates new point cloud data by taking the union of the multiple point cloud data after the multiple points corresponding to moving objects have been removed. For example, new point cloud data may be generated by taking the union of the point cloud data of FIG. 4A from which the multiple points corresponding to moving objects have been removed and the point cloud data of FIG. 4B. Alternatively, new point cloud data may be generated by taking the union of multiple point cloud data (point cloud data after the moving objects have been removed) based on the results of the first, second, third, etc. sensing. This enables highly accurate three-dimensional information to be generated in the subsequent process (S108).

[0041] Furthermore, the point cloud data of the four LiDARs 21 to 24 (point cloud data after removing moving objects) may be integrated. For example, the point cloud data of the LiDARs 21 to 24 may be integrated using an ICP (Iterative Closest Point) method based on information including the latitude, longitude, and altitude of the LiDARs 21 to 24 and the attitudes of the LiDARs 21 to 24.

[0042] In step S108, the information processing device 10 generates three-dimensional stereoscopic information using the three-dimensional information generating unit 12d (three-dimensional information generation process). That is, the three-dimensional information generating unit 12d generates three-dimensional stereoscopic information based on point cloud data (new point cloud data that is the union of multiple point cloud data) after the multiple points corresponding to the moving object have been removed. To generate such three-dimensional information, for example, the point cloud data may be converted into a predetermined mesh model. A mesh model is a three-dimensional model generated by dividing the three-dimensional surface shape of an object into simple shapes such as triangles and quadrangles. For example, the Alpha Shape method, the Poisson method, or the Marching Cubes method is used as a method for generating such a mesh model.

[0043] In step S108, the information processing device 10 transmits the three-dimensional information to the terminal T1 (see FIG. 2) via the communication unit 13. As a result, a predetermined image based on the three-dimensional information is displayed on the display of the terminal T1. For example, the user may be able to view a mesh model, which is the three-dimensional information, from a desired viewpoint or move the mesh model freely on the screen. This not only improves visibility and convenience for the user, but also improves work efficiency when the user remotely manages the progress of a construction site, etc. After performing the processing of step S109, the information processing device 10 ends the series of processes (END).

[0044] FIG. 5 is a diagram showing the hardware configuration of the information processing device 10. As shown in FIG. The information processing device 10 shown in FIG. 5 is a specific computer, and its hardware configuration includes a CPU 10a (Central Processing Unit), a ROM 10b (Read Only Memory), a RAM 10c (Random Access Memory), an HDD 10d (Hard Disk Drive), an input / output interface 10e, a communication interface 10f, and a media interface 10g, which are connected in a specific manner via an internal bus 10h.

[0045] The CPU 10a is hardware that functions as the processing unit 12 (see FIG. 2). The ROM 10b, RAM 10c, and HDD 10d are hardware that functions as the storage unit 11 (see FIG. 2). The CPU 10a reads out a predetermined program stored in the ROM 10b or HDD 10d and loads it into the RAM 10c, thereby executing a predetermined process.

[0046] The input / output interface 10e is an interface used for exchanging data with the input device 40 and the output device 50. The input device 40 is, for example, a keyboard or a mouse, and is used when a user inputs data. The output device 50 is, for example, a display, and displays the processing results of the information processing device 10 in a predetermined manner. Note that a touch panel type mobile terminal, such as a smartphone or tablet, which combines the input device 40, the output device 50, and a computing function may also be used.

[0047] The communication interface 10f is used for communication with external devices and networks. For example, the communication interface 10f performs predetermined communication with a terminal T1 (see FIG. 2) via a network N1 (see FIG. 2). The media interface 10g is an interface for reading data from a recording medium 60 and storing data in the recording medium 60. Note that the hardware configuration shown in FIG. 5 is an example and is not limited to this.

[0048] <Effects> According to the first embodiment, three-dimensional stereoscopic information of a construction site or the like can be generated based on point cloud data from the LiDARs 21 to 24. This allows the user to grasp the progress of work in real time, enabling efficient progress management. Furthermore, compared to flying a drone to capture images of a construction site or the like from the air, three-dimensional information can be generated more frequently and easily.

[0049] Furthermore, if multiple points corresponding to moving objects are detected in the point cloud data, the moving object removal unit 12c removes these multiple points. As a result, when a user checks the situation of a construction site or the like, the user can view highly visible three-dimensional information with unnecessary moving objects removed.

[0050] Second Embodiment The second embodiment differs from the first embodiment in that three-dimensional information is generated based on point cloud data from LiDARs 21 to 24 (see FIG. 6) and image data from visible light cameras 31 to 34 (see FIG. 6). The rest of the second embodiment is the same as the first embodiment. Therefore, only the parts that are different from the first embodiment will be described, and a description of the overlapping parts will be omitted.

[0051] FIG. 6 is a functional block diagram of an information processing device 10A according to the second embodiment. 6, four LiDARs 21 to 24 and four visible light cameras 31 to 34 are provided. Each of the LiDARs 21 to 24 repeats sensing multiple times, as in the first embodiment. The visible light cameras 31 to 34 are devices that generate image data by photoelectrically converting visible light incident through a lens (not shown) using an image sensor (not shown).

[0052] The field of view of the visible light camera 31 is assumed to overlap with the sensing range of the LiDAR 21 for the most part. The timing when the visible light camera 31 captures images is assumed to be included in the sensing time period (exposure time) of the LiDAR 21. Such a visible light camera 31 may be installed near the LiDAR 21, or may be configured integrally with the LiDAR 21. Other visible light cameras 32, 33, and 34 are similarly installed in association with the LiDARs 22, 23, and 24 in this order. Image data obtained by the visible light cameras 31 to 34 may be still images or videos. The image data of each of the visible light cameras 31 to 34 is recorded in association with the position, orientation, and capture time of the visible light cameras 31 to 34.

[0053] As shown in FIG. 6, the processing unit 12A of the information processing device 10A includes a data acquisition unit 12a, a moving object detection unit 12b, a positioning unit 12e, a moving object removal unit 12c, an interpolation unit 12f, a three-dimensional information generation unit 12d, and a color information addition unit 12g.

[0054] The data acquisition unit 12a acquires a plurality of point cloud data from the LiDAR 21, and acquires at least one frame of image data from the visible light camera 31 (the same applies to the other LiDARs 22 to 24 and the visible light cameras 32 to 34). The moving object detection unit 12b detects a plurality of points corresponding to moving objects based on the comparison result of the plurality of point cloud data. Furthermore, the moving object detection unit 12b detects moving objects included in the image data based on image recognition using the image data. Details of the detection of moving objects will be described later.

[0055] The alignment unit 12e aligns, for example, the point cloud data of the LiDAR 21 with the image data of the visible light camera 31. That is, the alignment unit 12e identifies the correspondence between each point of the point cloud data and each pixel of the image data. Note that alignment between the point cloud data and image data is also performed in the same manner for the remaining LiDARs 22 to 24 and the visible light cameras 32 to 34.

[0056] The moving object removal unit 12c, for example, removes a plurality of points corresponding to moving objects from at least one of the plurality of point cloud data generated by the LiDAR 21, and removes moving objects from the image data generated by the visible light camera 31. Note that moving objects are also removed appropriately from the remaining point cloud data of the LiDARs 22 to 24 and the image data of the visible light cameras 32 to 34.

[0057] The interpolation unit 12f interpolates a plurality of points in an area from which a plurality of points corresponding to a moving object have been removed in the point cloud data, based on the distribution of points around the area. The interpolation unit 12f also interpolates pixel values ​​(colors) of the pixel area from which the moving object has been removed in the image data, based on an image around the pixel area. Details of the interpolation unit 12f will be described later.

[0058] The 3D information generating unit 12d converts the point cloud data after removing the points corresponding to the moving objects into a mesh model as 3D information. The color information adding unit 12g adds color information to the 3D information. That is, the color information adding unit 12g adds the color of pixels corresponding to each mesh of the mesh model to the mesh in the image data.

[0059] FIG. 7 is a flowchart showing the flow of processing by the information processing device (also see FIG. 6 as appropriate). In step S201, the information processing device 10A acquires point cloud data from the LiDARs 21 to 24 and image data from the visible light cameras 31 to 34 using the data acquisition unit 12a (data acquisition process). Note that a plurality of point cloud data corresponding to a plurality of sensing operations is acquired from the LiDAR 21 (the same applies to the other LiDARs 22 to 24). Meanwhile, one or a plurality of frames of image data are acquired from the visible light camera 31 (the same applies to the other visible light cameras 32 to 34). Note that even when there is only one frame of image data, it is possible to detect a moving object based on image recognition.

[0060] In step S202, the information processing device 10A aligns the point cloud data and image data using the alignment unit 12e. For example, alignment may be performed based on perspective projection transformation between the point cloud data of the LiDAR 21 and the coordinate values ​​of each pixel in the image captured by the visible light camera 31. The same applies to the point cloud data of the remaining LiDARs 22 to 24 and the image data of the visible light cameras 32 to 34. Note that the above-mentioned alignment method is an example and is not limited to this.

[0061] In step S203, the information processing device 10A detects a moving object from the point cloud data and also detects a moving object from the image data using the moving object detection unit 12b (moving object detection process). More specifically, the moving object detection unit 12b detects a plurality of points corresponding to moving objects based on a comparison result of a plurality of point cloud data, and also detects a moving object included in the image data based on image recognition using the image data. The process of detecting a moving object from point cloud data is the same as that in the first embodiment (S102 to S105 in FIG. 3), and therefore description thereof will be omitted.

[0062] Furthermore, for detecting a moving object based on image data, the type of moving object (person, vehicle, etc.) may be specified in advance by a user's operation via the input device 40 (see FIG. 5). In this case, image recognition data for the moving object is assumed to be stored in advance in the storage unit 11 (see FIG. 6) in association with the type of moving object. Note that predetermined programs and data based on machine learning may be included in the image recognition data. Furthermore, when image recognition is performed by template matching, a predetermined template image is assumed to be included in the image recognition data.

[0063] Alternatively, for example, a region in which the difference in pixel values ​​between corresponding pixels in image data of two frames captured at different times is equal to or greater than a predetermined value may be detected as a moving object.Also, AI (Artificial Intelligence) may be used as appropriate to detect moving objects.

[0064] In step S204, the information processing device 10A causes the moving object removal unit 12c to remove a plurality of points corresponding to the moving object from at least one of the plurality of point cloud data, and remove the moving object from the image data (moving object removal process).

[0065] In step S205, the information processing device 10A performs a predetermined interpolation process using the interpolation unit 12f. That is, the interpolation unit 12f interpolates a plurality of points in an area from which a plurality of points corresponding to a moving object have been removed in the point cloud data, based on the distribution of points around the area. Furthermore, the interpolation unit 12f interpolates pixel values ​​(colors) of the pixel area from which the moving object has been removed in the image data, based on an image around the pixel area. This improves visibility when the user checks the three-dimensional information.

[0066] In step S206, the information processing device 10A generates three-dimensional information from the point cloud data using the three-dimensional information generation unit 12d (three-dimensional information generation process). That is, the three-dimensional information generation unit 12d converts the point cloud data from which a plurality of points corresponding to moving objects have been removed into a mesh model as three-dimensional information. In step S207, the information processing device 10A adds color information to the three-dimensional information (mesh model) by the color information adding unit 12g. That is, the color information adding unit 12g adds the color of a pixel corresponding to each mesh of the mesh model to the mesh in the image data. The pixel corresponding to each mesh of the mesh model is identified based on the result of the above-mentioned alignment (S202).

[0067] In step S208, the information processing device 10A transmits the stereoscopic information and the like to the terminal T1 via the communication unit 13. As a result, the three-dimensional stereoscopic information (mesh model) to which the color information has been added is displayed on the display of the terminal T1.

[0068] Alternatively, a mesh model without any added color information or image data from the visible light cameras 31-34 may be displayed on the display of the terminal T1 as appropriate in response to a user's input operation. Furthermore, the dimensions of an edge line of an object specified by the user in the mesh model may be calculated and displayed based on the three-dimensional positions of the start and end points of the edge line. This allows the user to easily grasp the dimensions of the location (e.g., material, equipment, or passageway) that the user specified. After performing the process of step S208, the information processing device 10A ends the series of processes (END).

[0069] <Effects> According to the second embodiment, moving objects are removed from both the point cloud data and the image data, allowing the user to view highly visible three-dimensional information. It is also possible to calculate and display the dimensions of the ridges of an object specified by the user in the three-dimensional information. Recording the three-dimensional information and image data can improve traceability regarding the progress of a construction site, for example. Furthermore, areas from which moving objects have been removed are appropriately interpolated in the point cloud data and image data, making it possible to further improve the visibility of the three-dimensional information and image data compared to the first embodiment.

[0070] <<Variations>> Although the information processing devices 10 and 10A and the information processing method according to the present disclosure have been described in the embodiments above, the present disclosure is not limited to these descriptions and various modifications can be made.

[0071] FIG. 8 is an explanatory diagram showing an example of a construction site in which LiDARs 21 to 24 and visible light cameras 31 to 34 are installed in an information processing device according to a modified example. In the example of FIG. 8, a first marker M1 is attached to a predetermined location on the mixer truck C1. Furthermore, a second marker M2 is attached to a predetermined location on the truck C2. For example, an AR marker (Augmented Reality Marker) is used as the first marker M1 and the second marker M2. The first marker M1 is a mark used to exclude a predetermined object (the mixer truck C1 in the example of FIG. 8) present around the visible light cameras 31 to 34 from the detection targets of moving objects. The second marker M2 is a mark used to include a predetermined object (the truck C2 in the example of FIG. 8) present around the visible light cameras 31 to 34 in the detection targets of moving objects.

[0072] In this configuration, when the moving object detection unit 12b (see FIG. 6) detects a first marker M1 from the image data of the visible light cameras 31 to 34, it detects the mixer truck C1 (target object) to which the first marker M1 is attached based on image recognition. Furthermore, the moving object detection unit 12b excludes the pixel area in the image data in which the mixer truck C1 (target object) is captured from the detection targets for moving objects. This allows the user to confirm, for example, an image in which the movable mixer truck C1 is captured as three-dimensional information. Therefore, it becomes possible to manage the progress of construction work, including the mixer truck C1.

[0073] Furthermore, when the moving object detection unit 12b (see FIG. 6) detects the second marker M2 from the image data of the visible light cameras 31 to 34, it detects the track C2 (object) to which the second marker M2 is attached based on image recognition. Furthermore, the moving object detection unit 12b detects the moving object within the pixel region in which the track C2 (object) appears in the image data. This allows the track C2 to which the second marker M2 is attached to be reliably detected based on image recognition.

[0074] Furthermore, when sensing is repeated multiple times by the LiDARs 21 to 24 and point cloud data is generated for each of the multiple sensing operations, the following processing may be performed. First, the data acquisition unit 12a (see FIG. 6) acquires point cloud data for multiple times from the LiDARs 21 to 24. When there are multiple points that do not correspond to each other in a pair of point cloud data that are adjacent in order when sensing is repeated, the moving object detection unit 12b (see FIG. 6) identifies areas including these multiple points in each point cloud data. Then, the moving object detection unit 12b (see FIG. 6) determines, based on image recognition, whether a moving object exists in a pixel area in the image data that corresponds to the above-mentioned area. In this way, by narrowing down the pixel area when detecting a moving object by image recognition based on the point cloud data, the detection accuracy of the moving object can be improved. Furthermore, it is preferable that the moving object detection unit 12b performs image recognition using image data of a frame acquired near the middle of the exposure time when the LiDAR 21 receives reflected light, among multiple frames captured at different times in the image data of the visible light camera 31. For example, if the exposure time of the LiDAR 21 is 10 seconds, there is a possibility that a person is not within the sensing range immediately after the start of exposure, and that a person may have left the sensing range just before the end of exposure. Therefore, the moving object detection unit 12b detects a moving object based on image data acquired, for example, five seconds after the start of exposure. This makes it possible to prevent missing detection of a moving object. The same can be said for the other LiDARs 22 to 24 and the visible light cameras 32 to 34.

[0075] Furthermore, the user may select, through an operation via the input device 40 (see FIG. 5), whether or not to include each predetermined pixel region included in the image data in a moving object detection target. In this case, the moving object detection unit 12b (see FIG. 6) detects moving objects by targeting those pixel regions selected by the user's operation to be included in a moving object detection target among the plurality of pixel regions. In this way, by allowing the user to select for each pixel region whether or not to include each pixel region in a moving object detection target, the degree of freedom in setting is increased. Note that the plurality of pixel regions may be divided into a matrix in advance, or may be appropriately designated through an input operation by the user.

[0076] In the first embodiment, the case where the LiDARs 21 to 24 are used has been described, and in the second embodiment, the case where the LiDARs 21 to 24 and the visible light cameras 31 to 34 are used has been described, but this is not limiting. For example, instead of the LiDARs 21 to 24, a stereo camera or a depth camera may be used. The number of types of sensors that generate measurement data by sensing the surroundings may be three or more. As the sensors, a stereo camera, a depth camera, etc. may be used as appropriate in addition to a LiDAR or a visible light camera. Furthermore, the AI ​​may generate distance information to each position included in the image data based on one or more frames of image data, and point cloud data may be generated based on this distance information.

[0077] In addition, although each embodiment has been described with respect to the case where two point cloud data sets that are adjacent in the sensing order are compared, this is not limiting. For example, the moving object detection unit 12b may compare three or more point cloud data sets (point cloud data sets that have approximately the same sensing range but different exposure time periods) and remove a moving object from at least one of the plurality of point cloud data sets.

[0078] Furthermore, the moving object detected by the moving object detection unit 12b may be removed by photo retouching software in response to a user's operation via the input device 40 (see FIG. 5). Furthermore, for example, if a moving object is found in both point cloud data based on a comparison result of two point cloud data that are adjacent in the sensing order of the LiDAR 21, new point cloud data may be generated by performing sensing again with the LiDAR 21, and the new point cloud data may be compared with either of the two point cloud data to detect the moving object again. This allows the area including the trajectory of the moving object to be extracted with high accuracy.

[0079] Furthermore, a program for causing a computer to execute the processes (information processing method, etc.) executed by the information processing device 10 can be provided via a communication line, or can be written to a predetermined recording medium and distributed. Furthermore, the functions of the information processing device 10 may be distributed to multiple computers connected in a predetermined manner via a signal line or a network.

[0080] Furthermore, the present disclosure is not limited to the embodiments and includes various modifications. For example, the embodiments have been described in detail to clearly explain the present disclosure, and the present disclosure is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to add, delete, or replace part of the configuration of the embodiments with other configurations.

[0081] Furthermore, the above-mentioned configurations, functions, processing units, processing means, etc. may be partly or entirely implemented in hardware, for example, by designing them as integrated circuits. Furthermore, the above-mentioned configurations, functions, etc. may be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0082] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0083] 10, 10A Information processing equipment 11 Storage section 12, 12A Processing section 12a Data acquisition section 12b Moving object detection unit 12c Moving object removal section 12d 3D information generation section 12e Alignment section 12f Interpolation section 12g Color information addition section 13 Communications Department 21, 22, 23, 24 LiDAR 31, 32, 33, 34 Visible light camera 40 Input Devices 50 Output Device A1 area C1 Mixer truck (object) C2 Truck (Object) M1 First marker M2 Second marker S101, S201 steps (data acquisition process) Steps S102, S103, S104, S105, S203 (moving object detection process) Steps S106 and S204 (moving object removal process) Steps S108 and S206 (3D information generation processing) S205 Step (interpolation processing) T1 terminal

Claims

1. a data acquisition unit that acquires a plurality of point cloud data corresponding to a plurality of sensing operations from the LiDAR that repeatedly senses the surroundings; a moving object detection unit that detects a plurality of points corresponding to moving objects based on a comparison result of the plurality of point cloud data; a moving object removal unit that removes a plurality of points corresponding to the moving object from at least one of the plurality of point cloud data; a stereoscopic information generating unit that generates three-dimensional stereoscopic information based on the point cloud data after the plurality of points corresponding to the moving object have been removed, The LiDAR is installed at a position higher than the height of an object onto which laser light is irradiated from the LiDAR, The moving object detection unit When there are a plurality of points that do not correspond to each other in a pair of point cloud data that are adjacent in order when the sensing is repeated, extracting an area including the plurality of points from each of the point cloud data; An information processing device that, when a plurality of points included in the area of ​​one of a pair of point cloud data have a smaller depth based on the position of the LiDAR than a plurality of points included in the area of ​​the other point cloud data, detects the plurality of points included in the area of ​​the one point cloud data as the moving object.

2. the three-dimensional information generating unit generates new point cloud data by taking a union of the plurality of point cloud data after the plurality of points corresponding to the moving object have been removed, and generates the three-dimensional information based on the new point cloud data.

2. The information processing device according to claim 1,

3. the data acquisition unit acquires a plurality of point cloud data from the LiDAR and acquires image data of at least one frame from a visible light camera; the moving object detection unit detects a plurality of points corresponding to the moving object based on a comparison result of the plurality of point cloud data, and detects the moving object included in the image data based on image recognition using the image data; The moving object removal unit removes a plurality of points corresponding to the moving object from at least one of the plurality of point cloud data, and removes the moving object from the image data.

2. The information processing device according to claim 1,

4. a registration unit that performs registration between the point cloud data and the image data; a color information adding unit that adds color information to the three-dimensional information, the three-dimensional information generation unit converts the point cloud data from which the plurality of points corresponding to the moving object have been removed into a mesh model as the three-dimensional information; The color information adding unit adds a color of a pixel corresponding to each mesh of the mesh model to the mesh in the image data.

4. The information processing device according to claim 3,

5. A data acquisition unit that acquires multiple point cloud data corresponding to multiple sensing operations from a LiDAR that repeatedly senses the surroundings; a moving object detection unit that detects a plurality of points corresponding to moving objects based on a comparison result of the plurality of point cloud data; a moving object removal unit that removes a plurality of points corresponding to the moving object from at least one of the plurality of point cloud data; a stereoscopic information generating unit that generates three-dimensional stereoscopic information based on the point cloud data after the plurality of points corresponding to the moving object have been removed, the data acquisition unit acquires a plurality of point cloud data from the LiDAR and acquires image data of at least one frame from a visible light camera; the moving object detection unit detects a plurality of points corresponding to the moving object based on a comparison result of the plurality of point cloud data, and detects the moving object included in the image data based on image recognition using the image data; the moving object removal unit removes a plurality of points corresponding to the moving object from at least one of the plurality of point cloud data, and removes the moving object from the image data; The moving object detection unit When there are a plurality of points that do not correspond to each other in a pair of point cloud data that are adjacent in order when the sensing is repeated, extracting an area including the plurality of points from each of the point cloud data; determining whether the moving object exists in a pixel region corresponding to the region in the image data based on image recognition; An information processing device that performs the image recognition using image data of a frame acquired at a time near the middle of the exposure time when the LiDAR receives reflected light, from among multiple frames in the image data that have different image capture times.

6. A data acquisition unit that acquires multiple point cloud data corresponding to multiple sensing operations from a LiDAR that repeatedly senses the surroundings; a moving object detection unit that detects a plurality of points corresponding to moving objects based on a comparison result of the plurality of point cloud data; a moving object removal unit that removes a plurality of points corresponding to the moving object from at least one of the plurality of point cloud data; a stereoscopic information generating unit that generates three-dimensional stereoscopic information based on the point cloud data after the plurality of points corresponding to the moving object have been removed, the data acquisition unit acquires a plurality of point cloud data from the LiDAR and acquires image data of at least one frame from a visible light camera; the moving object detection unit detects a plurality of points corresponding to the moving object based on a comparison result of the plurality of point cloud data, and detects the moving object included in the image data based on image recognition using the image data; the moving object removal unit removes a plurality of points corresponding to the moving object from at least one of the plurality of point cloud data, and removes the moving object from the image data; The information processing device further includes an interpolation unit that interpolates a plurality of points in an area from which a plurality of points corresponding to the moving object have been removed in the point cloud data based on a distribution of points surrounding the area, and that interpolates pixel values ​​of the pixel area from which the moving object has been removed in the image data based on an image surrounding the pixel area.

7. a storage unit in which image recognition data for a type of moving object is stored in association with the type of moving object designated by a user through an input device; The moving object detection unit detects the moving object from the image data based on the image recognition using the image recognition data.

4. The information processing device according to claim 3,

8. A first marker is attached to the predetermined object; When the moving object detection unit detects the first marker from the image data, the moving object detection unit detects the object to which the first marker is attached based on image recognition, and further excludes a pixel area in which the object is captured in the image data from detection targets for the moving object.

4. The information processing device according to claim 3,

9. A second marker is attached to the predetermined object; When the moving object detection unit detects the second marker from the image data, the moving object detection unit detects the object to which the second marker is attached based on image recognition, and further detects the moving object within a pixel area in which the object is captured in the image data.

4. The information processing device according to claim 3,

10. a user operates an input device to select whether or not to include each predetermined pixel region included in the image data in a detection target for the moving object; The moving object detection unit detects the moving object by targeting one of the plurality of pixel regions that has been selected by the operation to be included in the detection target of the moving object.

4. The information processing device according to claim 3,

11. a data acquisition process for acquiring a plurality of point cloud data corresponding to a plurality of sensing operations from the LiDAR that repeatedly senses the surroundings; a moving object detection process for detecting a plurality of points corresponding to moving objects based on a comparison result of the plurality of point cloud data; a moving object removal process for removing a plurality of points corresponding to the moving object from at least one of the plurality of point cloud data; a stereoscopic information generation process for generating three-dimensional stereoscopic information based on the point cloud data after the plurality of points corresponding to the moving object have been removed, The LiDAR is installed at a position higher than the height of an object onto which laser light is irradiated from the LiDAR, In the moving object detection process, When there are a plurality of points that do not correspond to each other in a pair of point cloud data that are adjacent in order when the sensing is repeated, extracting an area including the plurality of points from each of the point cloud data; An information processing method in which, when a plurality of points included in the area of ​​one of a pair of point cloud data have a smaller depth based on the position of the LiDAR than a plurality of points included in the area of ​​the other point cloud data, the plurality of points included in the area of ​​the one point cloud data are detected as the moving object.

12. a data acquisition process for acquiring a plurality of point cloud data corresponding to a plurality of sensing operations from the LiDAR that repeatedly senses the surroundings; a moving object detection process for detecting a plurality of points corresponding to moving objects based on a comparison result of the plurality of point cloud data; a moving object removal process for removing a plurality of points corresponding to the moving object from at least one of the plurality of point cloud data; a stereoscopic information generation process for generating three-dimensional stereoscopic information based on the point cloud data after the plurality of points corresponding to the moving object have been removed, In the data acquisition process, a plurality of point cloud data are acquired from the LiDAR, and at least one frame of image data is acquired from a visible light camera; In the moving object detection process, a plurality of points corresponding to the moving object are detected based on a comparison result of the plurality of point cloud data, and the moving object included in the image data is detected based on image recognition using the image data; In the moving object removal process, a plurality of points corresponding to the moving object are removed from at least one of the plurality of point cloud data, and the moving object is removed from the image data; In the moving object detection process, When there are a plurality of points that do not correspond to each other in a pair of point cloud data that are adjacent in order when the sensing is repeated, extracting an area including the plurality of points from each of the point cloud data; determining whether the moving object exists in a pixel region corresponding to the region in the image data based on image recognition; An information processing method in which the image recognition is performed using image data of a frame acquired at a time near the middle of the exposure time when the LiDAR receives reflected light, from among multiple frames in the image data that have different image capture times.

13. a data acquisition process for acquiring a plurality of point cloud data corresponding to a plurality of sensing operations from the LiDAR that repeatedly senses the surroundings; a moving object detection process for detecting a plurality of points corresponding to moving objects based on a comparison result of the plurality of point cloud data; a moving object removal process for removing a plurality of points corresponding to the moving object from at least one of the plurality of point cloud data; a stereoscopic information generation process for generating three-dimensional stereoscopic information based on the point cloud data after the plurality of points corresponding to the moving object have been removed, In the data acquisition process, a plurality of point cloud data are acquired from the LiDAR, and at least one frame of image data is acquired from a visible light camera; In the moving object detection process, a plurality of points corresponding to the moving object are detected based on a comparison result of the plurality of point cloud data, and the moving object included in the image data is detected based on image recognition using the image data; In the moving object removal process, a plurality of points corresponding to the moving object are removed from at least one of the plurality of point cloud data, and the moving object is removed from the image data; The method further includes an interpolation process of interpolating a plurality of points in an area from which a plurality of points corresponding to the moving object have been removed in the point cloud data based on a distribution of points around the area, and interpolating pixel values ​​of the pixel area from which the moving object has been removed in the image data based on an image around the pixel area, An information processing method, wherein the interpolation processing is performed after the moving object removal processing and before the three-dimensional information generation processing.

14. A program for causing a computer to execute the information processing method according to any one of claims 11 to 13.

Images