Information processing device, information processing method, and program

Abstract

This information processing device processes point group data output from a measuring device which comprises: an external sensor that repeatedly scans a surrounding space and acquires division point group data for each scan; and an internal sensor that detects an attitude and acquires attitude detection data. The information processing device is provided with at least one processor. The processor generates synthesized point group data by performing synthesis processing using a plurality of pieces of division point group data acquired during a period in which the attitude detection data satisfies a permission condition, from among a plurality of pieces of division point group data acquired by the external sensor at different times.

Description

[Technical Field]

[0001] The technology disclosed herein relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There is a known mobile system that mounts a measuring device such as LiDAR (Light Detection and Ranging) on ​​a mobile body to acquire point cloud data that represents the coordinates of surrounding structures. In a mobile system, the measuring device repeatedly scans the surrounding space, acquiring point cloud data for each scan, and then synthesizing the acquired multiple point cloud data to create map data with 3D information.

[0003] Japanese Patent Application Laid-Open No. 2016-189184 describes adjusting point cloud data in accordance with the attitude of a LiDAR, etc. This adjustment of the point cloud data is performed when combining point cloud data before and after a change in attitude. Summary of the Invention [Problem to be solved by the invention]

[0004] When the change in the attitude of the LiDAR is slight, it is possible to synthesize the point cloud data before and after the change in attitude by adjusting the point cloud data, as described in JP 2016-189184 A.

[0005] However, if the change in LiDAR attitude is large, point cloud data will be acquired that cannot be synthesized before and after the attitude change, and the information processing device used for the synthesis process will fail to synthesize the point cloud data before and after the attitude change.

[0006] The technology disclosed herein provides an information processing device, an information processing method, and a program that can prevent failures when combining multiple point cloud data. [Means for solving the problem]

[0007] The information processing device disclosed herein is an information processing device that processes segmented point cloud data output from a measurement device that has an external sensor that repeatedly scans the surrounding space and acquires segmented point cloud data for each scan, and an internal sensor that detects posture and acquires posture detection data, and is equipped with at least one processor. The processor generates composite point cloud data by performing a synthesis process using multiple segmented point cloud data that were acquired during a period in which the posture detection data satisfies an acceptable condition, out of multiple segmented point cloud data that were acquired at different times by the external sensor.

[0008] The internal sensor is an inertial measurement sensor having at least one of an acceleration sensor and an angular velocity sensor, and the attitude detection data preferably includes an output value of the acceleration sensor or the angular velocity sensor.

[0009] The permissible condition is preferably that the absolute value of the output value of the acceleration sensor or the angular velocity sensor is less than a first threshold value.

[0010] The permissible condition is preferably that the amount of change over time in the output value of the acceleration sensor or the angular velocity sensor is less than a second threshold value.

[0011] The external sensor preferably includes a first sensor that acquires first segmented point cloud data by scanning a space with laser light, and a second sensor that acquires second segmented point cloud data based on a plurality of camera images, and the segmented point cloud data preferably includes the first segmented point cloud data and the second segmented point cloud data.

[0012] Preferably, the processor generates composite segment point cloud data by combining the first segment point cloud data and the second segment point cloud data, and generates composite point cloud data by combining the generated plurality of composite segment point cloud data.

[0013] Preferably, the processor generates the composite segmented point cloud data by partially selecting data from each of the first segmented point cloud data and the second segmented point cloud data based on features of a structure captured in at least one of the multiple camera images.

[0014] The measuring device is preferably provided on an unmanned vehicle.

[0015] The information processing method disclosed herein is an information processing method for processing segmented point cloud data output from a measurement device equipped with an external sensor that repeatedly scans the surrounding space and acquires segmented point cloud data for each scan, and an internal sensor that detects posture and acquires posture detection data, and generates composite point cloud data by performing a synthesis process using multiple segmented point cloud data acquired during a period in which the posture detection data satisfies an acceptable condition, out of multiple segmented point cloud data acquired at different times by the external sensor.

[0016] The program disclosed herein is a program that causes a computer to process segmented point cloud data output from a measurement device that has an external sensor that repeatedly scans the surrounding space and acquires segmented point cloud data for each scan, and an internal sensor that detects posture and acquires posture detection data, and causes the computer to perform a synthesis process that generates synthetic point cloud data using multiple segmented point cloud data that were acquired during a period in which the posture detection data satisfies an acceptable condition, out of multiple segmented point cloud data that were acquired at different times by the external sensor. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a schematic configuration diagram showing an example of the overall configuration of a mobile body system according to a first embodiment. [Figure 2] FIG. 2 is a schematic perspective view showing an example of detection axes of an acceleration sensor and an angular velocity sensor. [Figure 3] FIG. 2 is a block diagram showing an example of a hardware configuration of a mobile system. [Figure 4] FIG. 2 is a conceptual diagram showing an example of a route along which a moving object moves; [Figure 5] FIG. 10 is a conceptual diagram illustrating an example of segment point cloud data. [Figure 6] FIG. 10 is a conceptual diagram illustrating an example of a synthesis process of a plurality of segment point cloud data. [Figure 7] FIG. 10 is a conceptual diagram illustrating an example of a period in which the permissible condition is not satisfied. [Figure 8] 10 is a flowchart showing an example of the flow of a synthesis process according to the first embodiment. [Figure 9] FIG. 10 is a schematic configuration diagram showing an example of the overall configuration of a mobile body system according to a second embodiment. [Figure 10] FIG. 10 is a block diagram showing an example of a hardware configuration of a mobile body system according to a second embodiment. [Figure 11] FIG. 10 is a conceptual diagram showing an example of a method for acquiring second division point cloud data. [Figure 12] FIG. 10 is a block diagram showing an example of a synthesis processing unit according to the second embodiment. [Figure 13] FIG. 10 is a conceptual diagram showing an example of second synthesized segment point cloud data. [Figure 14] 10 is a flowchart showing an example of the flow of a synthesis process according to the second embodiment. [Figure 15] FIG. 11 is a conceptual diagram showing an example of a period in which the permissible condition is not satisfied in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, examples of an information processing device, an information processing method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] CPU is an abbreviation for "Central Processing Unit". NVM is an abbreviation for "Non-volatile memory". RAM is an abbreviation for "Random Access Memory". IC is an abbreviation for "Integrated Circuit". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor." SLAM is an abbreviation for "Simultaneous Localization and Mapping."

[0021] [First embodiment] As an example, as shown in Fig. 1, a mobile body system 2 is configured with a mobile body 10 and an information processing device 20. A measuring device 30 is mounted on the mobile body 10. In this embodiment, the mobile body 10 is an unmanned airplane (a so-called drone) as an example of an unmanned mobile body. The mobile body 10 and the information processing device 20 communicate wirelessly.

[0022] The moving body 10 includes a main body 12 and four propellers 14 as drive devices. By controlling the rotation direction of each of the four propellers 14, the moving body 10 can fly along any desired path in three-dimensional space.

[0023] The measurement device 30 is attached, for example, to the top of the main body 12. The measurement device 30 has an external sensor 32 and an internal sensor 34 (see FIG. 3 ) built in. The external sensor 32 is a sensor that senses the external environment of the mobile body 10. In this embodiment, the external sensor 32 is a LiDAR, which scans the surrounding space, for example, by emitting a pulsed laser beam L into the surroundings. The laser beam L is, for example, visible light or infrared light.

[0024] The external sensor 32 receives the reflected light of the laser beam L reflected from structures present in the surrounding space, and measures the time from emitting the laser beam L to receiving the reflected light, thereby determining the distance to the reflection point of the laser beam L on the structure. Furthermore, each time the external sensor 32 scans the surrounding space, it outputs point cloud data representing position information (three-dimensional coordinates) of multiple reflection points. The point cloud data is also referred to as a point cloud. The point cloud data is, for example, data expressed in three-dimensional Cartesian coordinates.

[0025] The external sensor 32 emits a laser beam L in a field of view S of, for example, 135 degrees left and right (270 degrees in total) and 15 degrees up and down (30 degrees in total) based on the traveling direction of the mobile object 10. The external sensor 32 emits the laser beam L over the entire field of view S while changing the angle, for example, by 0.25 degrees in either the left and right or up and down directions. The external sensor 32 repeatedly scans the field of view S and outputs point cloud data for each scan. The point cloud data output by the external sensor 32 for each scan will be referred to hereinafter as segmented point cloud data PG.

[0026] The internal sensor 34 includes an acceleration sensor 36 and an angular velocity sensor 38 (see FIG. 3). For example, as shown in FIG. 2, the acceleration sensor 36 detects accelerations acting in the directions of the X-axis Ax, the Y-axis Ay, and the Z-axis Az, which are orthogonal to each other. For example, as shown in FIG. 2, the angular velocity sensor 38 detects angular velocities acting around each of the X-axis Ax, the Y-axis Ay, and the Z-axis Az (i.e., in the directions of roll, pitch, and yaw rotation). In other words, the internal sensor 34 is a six-axis inertial measurement sensor.

[0027] The internal sensor 34 outputs attitude detection data that indicates the attitude of the mobile object 10. The attitude detection data includes an output value S1 of the acceleration sensor 36 and an output value S2 of the angular velocity sensor 38. The output value of the acceleration sensor 36 includes acceleration detection values ​​in three axial directions, namely, the X-axis Ax, the Y-axis Ay, and the Z-axis Az, but for simplicity in this embodiment, these are collectively referred to as the output value S1. Similarly, the output value of the angular velocity sensor 38 includes angular velocity detection values ​​in three rotational directions, namely, roll, pitch, and yaw, but for simplicity in this embodiment, these are collectively referred to as the output value S2.

[0028] The mobile body 10 flies autonomously along a specified route while estimating its own position based on data acquired by the external sensor 32 and the internal sensor 34. The mobile body system 2 uses, for example, SLAM technology to simultaneously estimate the self-position of the mobile body 10 and create an environmental map.

[0029] The information processing device 20 is, for example, a personal computer, and includes a receiving device 22 and a display 24. The receiving device 22 is, for example, a keyboard, a mouse, a touch panel, etc. The information processing device 20 creates an environmental map by combining multiple pieces of segmented point cloud data PG output from the measuring device 30 of the moving body 10. The information processing device 20 displays the created environmental map on the display 24.

[0030] 3, the main body 12 of the moving body 10 is provided with a controller 16, a communication I / F 18, and a motor 14A. The controller 16 is configured, for example, with an IC chip. The controller 16 controls the flight of the moving body 10 by controlling the drive of the motor 14A provided for each of the four propellers 14. The controller 16 also controls the scanning operation of the laser beam L by the external sensor 32, and receives segmented point cloud data PG output from the external sensor 32.

[0031] The controller 16 also receives attitude detection data (the output value S1 of the acceleration sensor 36 and the output value S2 of the angular velocity sensor 38) output from the internal sensor 34. The controller 16 wirelessly transmits the received segmented point cloud data PG and attitude detection data to the information processing device 20 via the communication I / F 18.

[0032] The information processing device 20 includes a CPU 40, an NVM 42, a RAM 44, and a communication I / F 46 in addition to the accepting device 22 and the display 24. The accepting device 22, the display 24, the CPU 40, the NVM 42, the RAM 44, and the communication I / F 46 are connected via a bus 48. The information processing device 20 is an example of a "computer" according to the technology of the present disclosure. The CPU 40 is an example of a "processor" according to the technology of the present disclosure.

[0033] The NVM 42 stores various types of data. Here, examples of the NVM 42 include various types of nonvolatile storage devices such as an EEPROM, an SSD, and / or an HDD. The RAM 44 temporarily stores various types of information and is used as a work memory. Examples of the RAM 44 include a DRAM or an SRAM.

[0034] A program 43 is stored in the NVM 42. The CPU 40 reads the program 43 from the NVM 42 and executes the read program 43 on the RAM 44. The CPU 40 controls the entire mobile system 2 including the information processing device 20 by executing processing in accordance with the program 43. The CPU 40 also functions as a synthesis processing unit 41 by performing processing based on the program 43.

[0035] The communication I / F 46 wirelessly communicates with the communication I / F 18 of the moving object 10, and receives the segmented point cloud data PG and posture detection data output for each scan from the moving object 10. That is, the information processing device 20 receives multiple pieces of segmented point cloud data PG acquired at different times by the external sensor 32, and posture detection data corresponding to each piece of segmented point cloud data PG.

[0036] The synthesis processing unit 41 generates synthesized point cloud data SG by performing synthesis processing to synthesize multiple pieces of segmented point cloud data PG received from the moving object 10. The synthesized point cloud data SG corresponds to the above-mentioned environmental map. The synthesized point cloud data SG generated by the synthesis processing unit 41 is stored in the NVM 42. The synthesized point cloud data SG stored in the NVM 42 is displayed on the display 24 as the environmental map.

[0037] In addition, when performing the synthesis process, the synthesis processing unit 41 generates synthesis point cloud data SG by performing the synthesis process using multiple division point cloud data PG acquired from the moving body 10 during a period in which the posture detection data satisfies the tolerance conditions.

[0038] The acceptable condition is, for example, that the absolute value of the output value S1 of the acceleration sensor 36 is less than the threshold value TH1. This means, for example, that of the acceleration detection values ​​in the three axial directions included in the output value S1 of the acceleration sensor 36, at least one of the acceleration detection values ​​in the axial direction is less than the threshold value TH1. In this case, the synthesis processing unit 41 generates the synthesized point cloud data SG by performing synthesis processing using the multiple segmented point cloud data PG acquired during the period when the absolute value of the output value S1 of the acceleration sensor 36 is less than the threshold value TH1. The threshold value TH1 is an example of a "first threshold value" according to the technology of the present disclosure.

[0039] The permissible condition may be that the absolute value of the output value S2 of the angular velocity sensor 38 is less than a threshold. This means, for example, that of the angular velocity detection values ​​in three rotation directions included in the output value S2 of the angular velocity sensor 38, at least one of the angular velocity detection values ​​in the rotation direction is less than the threshold. In this case, the synthesis processing unit 41 generates the synthesized point cloud data SG by performing synthesis processing using the plurality of segmented point cloud data PG acquired during the period in which the absolute value of the output value S2 of the angular velocity sensor 38 is less than the threshold.

[0040] The acceptable condition may also be that the amount of change over time in the output value S1 of the acceleration sensor 36 is less than a threshold value TH2. This means, for example, that the amount of change over time in the acceleration detection value in at least one of the three axial directions included in the output value S1 of the acceleration sensor 36 is less than the threshold value TH2. The amount of change over time is the absolute value of the amount of change per unit time (e.g., one second). In this case, the synthesis processing unit 41 generates the synthesized point cloud data SG by performing synthesis processing using multiple segmented point cloud data PG acquired during a period in which the amount of change over time in the output value S1 of the acceleration sensor 36 is less than the threshold value TH2. The threshold value TH2 is an example of a "second threshold" according to the technology of the present disclosure. For example, the threshold value TH2 is set to a value 1.5 times the amount of change over time in the output value S1 when the moving object 10 is in a steady state.

[0041] The permissible condition may also be that the amount of change over time in the output value S2 of the angular velocity sensor 38 is less than a threshold value. This means, for example, that the amount of change over time in at least one of the angular velocity detection values ​​in the three rotation directions included in the output value S2 of the angular velocity sensor 38 is less than the threshold value. The amount of change over time is the absolute value of the amount of change per unit time (e.g., one second). In this case, the synthesis processing unit 41 generates the synthesized point cloud data SG by performing synthesis processing using multiple segmented point cloud data PG acquired during a period in which the amount of change over time in the output value S2 of the angular velocity sensor 38 is less than the threshold value.

[0042] The permissible condition may also be a condition that combines two or more values ​​from the output value S1 of the acceleration sensor 36, the output value S2 of the angular velocity sensor 38, the amount of change over time in the output value S1, and the amount of change over time in the output value S2.

[0043] As an example, as shown in FIG. 4, a mobile object 10 moves along a predetermined route KL. A plurality of structures 50 exist around the route KL along which the mobile object 10 moves. While moving along the route KL, the mobile object 10 repeatedly scans the surrounding space with the external sensor 32 of the measurement device 30, and acquires and outputs segmented point cloud data PG for each scan. In other words, the mobile object 10 scans the entire space around the route KL by scanning the entire space in units that are spatially and temporally segmented. In the example shown in FIG. 4, the mobile object 10 scans the field of view S with a laser beam L at each of three positions K1 to K3.

[0044] 5, the moving body 10 acquires and outputs segmented point cloud data PG1 to PG3 at positions K1 to K3 using the external sensor 32. Each point included in the segmented point cloud data PG1 to PG3 represents the position (three-dimensional coordinates) of the reflection point of the laser beam L by the structure 50.

[0045] 6, the synthesis processing unit 41 performs synthesis processing after aligning the segmented point cloud data PG1 to PG3 so that they match each other, thereby generating synthesized point cloud data SG. The synthesis processing unit 41 performs synthesis processing using, for example, a technique used in SLAM.

[0046] While the moving object 10 is flying along the route KL, its attitude may change significantly due to being blown by, for example, a gust of wind. If the attitude of the moving object 10 changes significantly while moving along the route KL in this way, the field of view S changes significantly, making it impossible to match the two pieces of segmented point cloud data PG before and after the attitude change, which may result in a failure of the synthesis process. For this reason, as described above, the synthesis processing unit 41 does not use segmented point cloud data PG acquired during a period in which the attitude detection data does not satisfy the tolerance condition, but instead performs the synthesis process using only segmented point cloud data PG whose attitude detection data satisfies the tolerance condition.

[0047] As an example, as shown in FIG. 7, when the absolute value of the output value S1 during the period in which the division point cloud data PG1 to PG15 and the acceleration sensor 36 are obtained and the division point cloud data PG7 to PG9 are obtained is less than the threshold value TH1, the synthesis processing unit 41 performs synthesis processing using the division point cloud data PG1 to PG6 and PG10 to PG15.

[0048] In addition, the synthesis processing unit 41 may acquire multiple segmented point cloud data PG from the moving body 10 and then synthesize the acquired multiple segmented point cloud data PG, or may perform the synthesis processing each time it acquires segmented point cloud data PG from the moving body 10.

[0049] Next, the operation of the mobile system 2 will be described with reference to FIG.

[0050] Fig. 8 shows a flowchart illustrating an example of the flow of the compositing process executed by the compositing processing unit 41. The flow of the compositing process shown in Fig. 8 is an example of an "information processing method" according to the technology of the present disclosure.

[0051] 8 shows an example in which the synthesis processing unit 41 performs synthesis processing every time it acquires segmented point cloud data PG from the moving object 10. Here, it is assumed that the moving object 10 repeatedly scans the surrounding space with the external sensor 32 and outputs the segmented point cloud data PG to the information processing device 20 for each scan.

[0052] 8, first, in step ST10, the synthesis processing unit 41 acquires the segmented point cloud data PG output from the moving object 10. After step ST10, the synthesis processing proceeds to step ST11.

[0053] In step ST11, the synthesis processing unit 41 acquires the above-mentioned attitude detection data corresponding to the segmented point group data PG acquired in step ST10 from the moving object 10. After step ST11, the synthesis processing proceeds to step ST12.

[0054] In step ST12, the synthesis processing unit 41 determines whether or not the attitude detection data acquired in step ST11 satisfies the above-described tolerance conditions. If the attitude detection data satisfies the tolerance conditions in step ST12, the determination is affirmative, and the synthesis processing proceeds to step ST13. If the attitude detection data does not satisfy the tolerance conditions in step ST12, the determination is negative, and the synthesis processing proceeds to step ST14.

[0055] In step ST13, the synthesis processing unit 41 performs the synthesis processing described above. Specifically, the synthesis processing unit 41 performs synthesis processing to synthesize the segmented point cloud data PG acquired in the previous scan and the segmented point cloud data PG acquired in the current scan. After step ST13, the synthesis processing proceeds to step ST14.

[0056] In step ST14, the synthesis processing unit 41 determines whether or not a condition for terminating the synthesis process (hereinafter referred to as the "termination condition") is satisfied. One example of the termination condition is that an instruction to terminate the synthesis process is accepted by the accepting device 22. In step ST14, if the termination condition is not satisfied, judgementIf the result of the calculation is negative, the synthesis process proceeds to step ST10. If the termination condition is met in step ST14, judgement is affirmative, and the synthesis process ends.

[0057] As described above, the information processing device 20 performs synthesis processing using a plurality of segmented point cloud data PG acquired during a period in which the posture detection data satisfies the tolerance condition, out of a plurality of segmented point cloud data PG acquired at different times by the external sensor 32, to generate synthesized point cloud data SG. This makes it possible to prevent failures when synthesizing a plurality of point cloud data.

[0058] [Second embodiment] In the first embodiment, the external sensor 32 is configured by one sensor (LiDAR), but in the second embodiment, the external sensor 32 is configured by two sensors.

[0059] As an example, as shown in FIG. 9, in a mobile body system 2A according to the second embodiment, a mobile body 10A has a measuring device 30A equipped with multiple cameras 60. The cameras 60 are, for example, digital cameras with CMOS image sensors, and generate and output image data PD. The cameras 60 perform imaging operations at a predetermined frame rate. The image data PD is an example of a "camera image" according to the technology of the present disclosure.

[0060] The multiple cameras 60 each capture a portion of the field of view S, thereby capturing an image of the entire range including the field of view S. The imaging ranges of at least two adjacent cameras 60 among the multiple cameras 60 at least partially overlap. That is, the two adjacent cameras 60 capture a pair of parallax images made up of image data PD.

[0061] 10 as an example, the external sensor 32 of this embodiment has a first sensor 32A and a second sensor 32B. The first sensor 32A is the LiDAR described in the first embodiment, and acquires segmented point cloud data PG by scanning a laser beam L within a field of view range S. Hereinafter, the segmented point cloud data PG acquired by the first sensor 32A will be referred to as first segmented point cloud data PGA.

[0062] The second sensor 32B has the above-mentioned multiple cameras 60, and acquires the segmented point cloud data PG based on the multiple image data PD acquired by the multiple cameras 60. Hereinafter, the segmented point cloud data PG acquired by the second sensor 32B will be referred to as second segmented point cloud data PGB.

[0063] 11, the second sensor 32B extracts corresponding feature points U1 and U2 from a pair of image data PD1 and PD2. Based on the difference in position (parallax) between the extracted feature points U1 and U2, the second sensor 32B calculates the three-dimensional coordinates of point P represented by the corresponding feature points U1 and U2 using the principle of triangulation. Publicly known algorithms such as SIFT, SURF, and AKAZE can be used to extract the feature points.

[0064] 11, only one feature point is shown in each of the image data PD1 and PD2, but the second sensor 32B extracts multiple feature points from each of the image data PD1 and PD2 to calculate three-dimensional coordinates of multiple points P. The second sensor 32B is a distance measurement sensor that uses a so-called stereo camera.

[0065] The second sensor 32B calculates three-dimensional coordinates for a plurality of points P within the field of view range S based on a plurality of image data PD acquired by a plurality of cameras 60, thereby generating second segmented point cloud data PGB.

[0066] The second sensor 32B using a stereo camera extracts feature points corresponding to the texture (pattern, etc.) of the structure being measured from the image data PD, and therefore the distance measurement accuracy depends on the texture of the structure. For example, the second sensor 32B cannot measure distances from a surface without a pattern, because it is difficult to acquire feature points. In contrast, the first sensor 32A using LiDAR measures distances based on the reflected light of the laser beam L from the structure, and therefore the distance measurement accuracy does not depend on the texture of the structure. For this reason, the point cloud density of the second segmented point cloud data PGB generated by the second sensor 32B is lower than the point cloud density of the first segmented point cloud data PGA generated by the first sensor 32A.

[0067] On the other hand, since the image data PD makes it possible to accurately extract the edge portions of the structure as feature points, the second sensor 32B can measure the distance to the edge portions of the structure with high accuracy. In contrast, since the measurement points obtained by scanning with the laser beam L are discrete, the first sensor 32A cannot measure the distance to the edge portions of the structure with high accuracy.

[0068] That is, the first sensor 32A can acquire point cloud data with high accuracy for parts other than the edge parts of the structure, and the second sensor 32B can acquire point cloud data with high accuracy for the edge parts of the structure.

[0069] In this embodiment, the CPU 40 outputs the first segmented point cloud data PGA and the second segmented point cloud data PGB to the information processing device 20 via the communication I / F 18. Furthermore, in this embodiment, the CPU 40 outputs, in addition to the first segmented point cloud data PGA and the second segmented point cloud data PGB, a plurality of image data PD acquired by a plurality of cameras 60 to the information processing device 20 via the communication I / F 18.

[0070] As an example, as shown in FIG. 12, in this embodiment, a synthesis processing unit 41A realized by the CPU 40 is configured with a first synthesis processing unit 70, a second synthesis processing unit 72, and an edge detection unit 74. The first synthesis processing unit 70 detects the moving object 10 for each scan. A The edge detection unit 74 obtains the first segmented point cloud data PGA and the second segmented point cloud data PGB output from the A The plurality of image data PD output from

[0071] The first synthesis processing unit 70 generates synthetic segment point cloud data SPG by partially selecting data from each of the first segment point cloud data PGA and the second segment point cloud data PGB based on the characteristics of structures depicted in at least one of the multiple image data PD.

[0072] The edge detection unit 74 detects the moving object 10 A The edge portion of the structure shown in the image data PD is detected by performing image analysis on at least one of the multiple image data PD acquired from the image data PD. For edge detection, a filtering method, a machine learning method, or the like can be used.

[0073] In this embodiment, the first synthesis processor 70 generates the synthesized segmented point cloud data SPG by partially selecting data from each of the first segmented point cloud data PGA and the second segmented point cloud data PGB, based on area information of the edge portions of the structures detected by the edge detector 74. The generation of the synthesized segmented point cloud data SPG by the first synthesis processor 70 is performed for each scan described above.

[0074] 13, the first synthesis processor 70 selects data corresponding to the edge portions of the structure 50 from the second segmented point cloud data PGB, selects data corresponding to portions other than the edge portions of the structure 50 from the first segmented point cloud data PGA, and synthesizes the selected data to generate the synthesized segmented point cloud data SPG. In other words, the synthesized segmented point cloud data SPG is high-resolution segmented point cloud data in which the edge portions of the structure 50 in the first segmented point cloud data PGA have been supplemented with the second segmented point cloud data PGB.

[0075] The second synthesis processor 72 generates synthesized point cloud data SG by synthesizing the multiple synthesized segment point cloud data SPG generated by the first synthesis processor 70. The synthesized point cloud data SG generated by the second synthesis processor 72 corresponds to the synthesized point cloud data SG of the first embodiment.

[0076] The synthesis processing unit 41A may perform the above-mentioned synthesis processing after acquiring a plurality of first segment point cloud data PGA and second segment point cloud data PGB from the moving body 10A, or may perform the synthesis processing each time a set of first segment point cloud data PGA and second segment point cloud data PGB is acquired from the moving body 10A.

[0077] Next, the operation of the mobile body system 2A according to the second embodiment will be described with reference to FIG.

[0078] Fig. 14 shows a flowchart illustrating an example of the flow of the compositing process executed by the compositing processing unit 41A. Note that the flow of the compositing process shown in Fig. 14 is an example of an "information processing method" according to the technology of the present disclosure.

[0079] 14 shows an example in which the synthesis processing unit 41A performs synthesis processing every time it acquires the first segmented point cloud data PGA and the second segmented point cloud data PGB from the moving object 10A. Here, it is assumed that the moving object 10A repeatedly scans the surrounding space with the external sensor 32 and outputs the first segmented point cloud data PGA and the second segmented point cloud data PGB to the information processing device 20 for each scan.

[0080] 14, first, in step ST20, the synthesis processing unit 41A acquires the first segmented point cloud data PGA and the second segmented point cloud data PGB output from the moving object 10A. Note that in step ST20, the synthesis processing unit 41A acquires a plurality of image data PD output from the moving object 10A in addition to the first segmented point cloud data PGA and the second segmented point cloud data PGB. After step ST20, the synthesis processing proceeds to step ST21.

[0081] In step ST21, the synthesis processing unit 41A acquires, from the moving object 10A, the above-mentioned attitude detection data corresponding to the first segmented point cloud data PGA and the second segmented point cloud data PGB acquired in step ST20. After step ST21, the synthesis processing proceeds to step ST22.

[0082] In step ST22, the synthesis processing unit 41A determines whether or not the attitude detection data acquired in step ST21 satisfies the above-mentioned tolerance conditions. If the attitude detection data satisfies the tolerance conditions in step ST22, the determination is affirmative, and the synthesis processing proceeds to step ST23. If the attitude detection data does not satisfy the tolerance conditions in step ST22, the determination is negative, and the synthesis processing proceeds to step ST26.

[0083] In step ST23, the edge detection unit 74 detects the edge portion of a structure appearing in the image data PD based on at least one image data PD. After step ST23, the synthesis process proceeds to step ST24.

[0084] In step ST24, the first synthesis processor 70 generates the synthesized segmented point cloud data SPG by synthesizing the first segmented point cloud data PGA and the second segmented point cloud data PGB based on the area information of the edge portions of the structures detected by the edge detector 74. After step ST24, the synthesis process proceeds to step ST25.

[0085] In step ST25, the second synthesis processing unit 72 performs the synthesis process described above. Specifically, the first synthesis processing unit 70 synthesizes the synthesized segment point group data SPG generated in the previous cycle and the synthesized segment point group data SPG generated in the current cycle. Generate The synthesized point cloud data SG is generated by synthesizing the synthesized segment point cloud data SPG with the synthesized segment point cloud data SPG. After step ST25, the synthesis process proceeds to step ST26.

[0086] In step ST26, the synthesis processing unit 41A determines whether or not a termination condition for terminating the synthesis process is satisfied. One example of the termination condition is that an instruction to terminate the synthesis process is accepted by the accepting device 22. In step ST26, if the termination condition is not satisfied, judgement If the result of the calculation is negative, the synthesis process proceeds to step ST20. If the termination condition is met in step ST26, judgement is affirmative, and the synthesis process ends.

[0087] As described above, in the second embodiment, the information processing device 20 generates the combined segment point cloud data SPG by combining the first segment point cloud data PGA and the second segment point cloud data PGB, and further generates the combined point cloud data SG by combining a plurality of combined segment point cloud data SPG, thereby obtaining high-resolution combined point cloud data representing an environmental map.

[0088] In the second embodiment, it is assumed that the moving body 10A outputs the first segmented point cloud data PGA and the second segmented point cloud data PGB for each scan (i.e., at the same cycle). Alternatively, as shown in FIG. 15, the moving body 10A may output the first segmented point cloud data PGA and the second segmented point cloud data PGB at different cycles. Even in this case, the synthesis process is performed using the first segmented point cloud data PGA and the second segmented point cloud data PGB acquired during a period in which the attitude detection data satisfies the tolerance condition. In this case, the second synthesis processor 72 may generate the synthesized segmented point cloud data SPG by combining the first segmented point cloud data PGA and the second segmented point cloud data PGB that are closest in time.

[0089] Furthermore, in the second embodiment, the measurement device 30A is provided with multiple cameras 60, but the measurement device 30A may be provided with only one camera 60. Even if the measurement device 30A is provided with only one camera 60, the two image data PD captured at different times have different viewpoints because the moving object 10A is moving. Therefore, the second sensor 32B can generate the second segmented point cloud data PGB based on the two image data PD captured at different times.

[0090] In the first and second embodiments, the program 43 for the synthesis process is stored in the NVM 42 (see FIGS. 3 and 10), but the technology of the present disclosure is not limited to this, and the program 43 may be stored in a non-transitory storage medium such as an SSD or a USB memory. In this case, the program 43 stored in the non-transitory storage medium is installed in the information processing device 20 as a computer, and the CPU 40 executes the above-described synthesis process in accordance with the program 43.

[0091] Alternatively, the program 43 may be stored in a storage device of another computer or server device connected to the information processing device 20 via a communication network (not shown), and the program 43 may be downloaded and installed in the information processing device 20 in response to a request from the information processing device 20. In this case, the synthesis process is executed by the computer in accordance with the installed program 43.

[0092] Furthermore, in the first and second embodiments, the synthesis process is performed by the information processing device 20, but the synthesis process may be performed inside the moving bodies 10 and 10A.

[0093] The hardware resources for executing the above-described synthesis process can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing the synthesis process by executing software, i.e., program 43, as described above. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations designed specifically for executing specific processes. Each processor has a built-in or connected memory, and executes the synthesis process using the memory.

[0094] The hardware resource that executes the synthesis process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the synthesis process may be a single processor.

[0095] Examples of a system configured with a single processor include: first, a system in which one processor is configured with a combination of one or more CPUs and software, as typified by client and server computers, and this processor functions as a hardware resource that executes the synthesis process; second, a system in which a processor is used to realize the functions of the entire system, including multiple hardware resources that execute the synthesis process, on a single IC chip, as typified by SoCs; thus, the synthesis process is realized using one or more of the above-mentioned various processors as hardware resources.

[0096] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0097] The above-described synthesis process is merely an example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of the processes may be changed, without departing from the spirit of the invention.

[0098] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0099] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0100] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0101] The above explanation allows one to understand the following techniques. [Additional note 1] An information processing device that processes segmented point cloud data output from a measurement device that includes an external sensor that repeatedly scans a surrounding space and acquires segmented point cloud data for each scan, and an internal sensor that detects an attitude and acquires attitude detection data, at least one processor; The processor: generating composite point cloud data by performing a synthesis process using a plurality of the segmented point cloud data acquired during a period in which the posture detection data satisfies an allowable condition, among the plurality of segmented point cloud data acquired at different times by the external sensor; Information processing device. [Additional note 2] the internal sensor is an inertial measurement sensor having at least one of an acceleration sensor and an angular velocity sensor; the attitude detection data includes an output value of the acceleration sensor or the angular velocity sensor; Item 1. An information processing device according to item 1. [Additional note 3] The permissible condition is that the absolute value of the output value of the acceleration sensor or the angular velocity sensor is less than a first threshold value. Item 2. An information processing device according to claim 2. [Additional note 4] The permissible condition is that a time change in the output value of the acceleration sensor or the angular velocity sensor is less than a second threshold value. Item 2. An information processing device according to claim 2. [Additional note 5] the external sensor includes a first sensor that acquires first segmented point cloud data by scanning a space with a laser beam, and a second sensor that acquires second segmented point cloud data based on a plurality of camera images; the segmented point cloud data includes the first segmented point cloud data and the second segmented point cloud data; 5. The information processing device according to claim 1. [Additional note 6] The processor: generating composite segment point cloud data by combining the first segment point cloud data and the second segment point cloud data, and generating the composite point cloud data by combining the generated plurality of composite segment point cloud data; Item 5. An information processing device according to item 5. [Additional note 7] The processor: generating the composite segmented point cloud data by partially selecting data from each of the first segmented point cloud data and the second segmented point cloud data based on features of structures captured in at least one of the plurality of camera images; Item 6. An information processing device according to claim 6. [Additional note 8] The measuring device is provided on an unmanned vehicle. 8. An information processing device according to any one of claims 1 to 7. [Additional note 9] An information processing method for processing segmented point cloud data output from a measurement device including an external sensor that repeatedly scans a surrounding space and acquires segmented point cloud data for each scan, and an internal sensor that detects an attitude and acquires attitude detection data, comprising: generating composite point cloud data by performing a synthesis process using a plurality of the segmented point cloud data acquired during a period in which the posture detection data satisfies an allowable condition, among the plurality of segmented point cloud data acquired at different times by the external sensor; Information processing methods. [Additional Note 10] A program that causes a computer to process segmented point cloud data output from a measurement device that includes an external sensor that repeatedly scans a surrounding space and acquires segmented point cloud data for each scan, and an internal sensor that detects an attitude and acquires attitude detection data, a synthesis process for generating synthesized point cloud data using a plurality of the segmented point cloud data acquired during a period in which the posture detection data satisfies an allowable condition, among the plurality of segmented point cloud data acquired at different times by the external sensor; A program to be executed by the computer.

Claims

1. an external sensor including a first sensor that acquires first segmented point cloud data by scanning with a laser beam and a second sensor that acquires second segmented point cloud data based on a plurality of camera images, and that repeatedly scans a surrounding space to acquire segmented point cloud data including the first segmented point cloud data and the second segmented point cloud data for each scan; an internal sensor that detects the posture and acquires posture detection data; An information processing device that processes the segmented point cloud data output from a measurement device comprising: at least one processor; The processor: In a synthesis process for generating synthesized point cloud data using a plurality of segmented point cloud data acquired during a period in which the posture detection data satisfies an allowable condition, among the plurality of segmented point cloud data acquired at different times by the external sensor, generating composite segment point cloud data by partially selecting data from each of the first segment point cloud data and the second segment point cloud data included in the segment point cloud data based on features of a structure captured in at least one of the plurality of camera images, and generating the composite point cloud data by combining the generated plurality of composite segment point cloud data; Information processing device.

2. the internal sensor is an inertial measurement sensor having at least one of an acceleration sensor and an angular velocity sensor; the attitude detection data includes an output value of the acceleration sensor or the angular velocity sensor; The information processing device according to claim 1 .

3. the permissible condition is that the absolute value of the output value of the acceleration sensor or the angular velocity sensor is less than a first threshold value; The information processing device according to claim 2 .

4. the permissible condition is that a time change in the output value of the acceleration sensor or the angular velocity sensor is less than a second threshold value; The information processing device according to claim 2 .

5. the processor detects an edge portion as a feature of the structure, selects data corresponding to the edge portion from the second segmented point cloud data, selects data corresponding to portions other than the edge portion from the first segmented point cloud data, and generates the composite point cloud data by combining the selected data. The information processing device according to any one of claims 1 to 4.

6. The measuring device is provided on an unmanned vehicle. The information processing device according to claim 1 .

7. an external sensor including a first sensor that acquires first segmented point cloud data by scanning with a laser beam and a second sensor that acquires second segmented point cloud data based on a plurality of camera images, and that repeatedly scans a surrounding space to acquire segmented point cloud data including the first segmented point cloud data and the second segmented point cloud data for each scan; an internal sensor that detects the posture and acquires posture detection data; An information processing method for processing the segmented point cloud data output from a measurement device comprising: a processor performing a synthesis process of generating synthetic point cloud data using a plurality of segmented point cloud data acquired during a period in which the posture detection data satisfies an allowable condition, among the plurality of segmented point cloud data acquired at different times by the external sensor, the processor generates composite segmented point cloud data by partially selecting data from each of the first segmented point cloud data and the second segmented point cloud data included in the segmented point cloud data based on features of structures captured in at least one of the plurality of camera images, and generates the composite point cloud data by combining the generated plurality of composite segmented point cloud data; Information processing methods.

8. an external sensor including a first sensor that acquires first segmented point cloud data by scanning with a laser beam and a second sensor that acquires second segmented point cloud data based on a plurality of camera images, and that repeatedly scans a surrounding space to acquire segmented point cloud data including the first segmented point cloud data and the second segmented point cloud data for each scan; an internal sensor that detects the posture and acquires posture detection data; A program for causing a computer to process the segmented point cloud data output from a measurement device comprising: In a synthesis process for generating synthesized point cloud data using a plurality of segmented point cloud data acquired during a period in which the posture detection data satisfies an allowable condition, among the plurality of segmented point cloud data acquired at different times by the external sensor, a process of generating composite segment point cloud data by partially selecting data from each of the first segment point cloud data and the second segment point cloud data included in the segment point cloud data based on features of a structure captured in at least one of the plurality of camera images, and generating the composite point cloud data by combining the generated plurality of composite segment point cloud data; A program to be executed by the computer.

Images