Information processing device, method for controlling the information processing device, and control program for the information processing device

Abstract

This invention provides a technology that enables the acquisition of detailed three-dimensional shape information of a measurement target, regardless of the distance from the measuring unit to the measurement target. [Solution] The information processing device comprises a shape acquisition unit capable of acquiring three-dimensional shape information of a processing target, an image acquisition unit capable of acquiring image data including the processing target, and a density enhancement unit capable of gradually increasing the density of the three-dimensional shape information based on the image data.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for an information processing apparatus, and a control program for an information processing apparatus.

Background Art

[0002] In recent years, three-dimensional shape information obtained from three-dimensional ranging devices such as LiDAR (Light Detection And Ranging) sensors has been widely used in autonomous driving, object recognition, or generation of 3D models. In Non-Patent Document 1, a method for generating a high-density point cloud by combining an image with a point cloud based on reflected light reflected by a measurement target using a LiDAR sensor is described. Further, in Patent Document 1, a method for generating a high-density three-dimensional shape by repeatedly applying the visual volume intersection method to voxel grids with different densities is described. Further, in Patent Document 2, a method for inferring a high-density three-dimensional shape of a road surface in autonomous driving using a neural network structure is described.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, LiDAR sensors emit light rays radially from the sensor center and acquire 3D distance information based on the light rays reflected by the object being measured. Therefore, if the object being measured is located far away, the acquired 3D distance information will be sparse. Similarly, if a low-performance LiDAR is used, the acquired 3D distance information will also be sparse. Thus, in such cases, it is difficult to obtain detailed 3D shape information of the object being measured using the technology described in Non-Patent Document 1. Furthermore, the viewing volume cross-section method in Patent Document 1 is a method based solely on images, so the accuracy of the resulting 3D shape is limited. Moreover, the method described in Patent Document 2 is a method that recovers the surface shape from a point cloud acquired from a LiDAR sensor, so if the point cloud itself is sparse, it is difficult to obtain detailed 3D shape information. Furthermore, its application is limited to road surfaces.

[0006] Therefore, in one aspect, the present invention aims to provide a technology that can acquire detailed three-dimensional shape information of a measurement target regardless of the distance information from the measuring unit to the measurement target. [Means for solving the problem]

[0007] One aspect of the present invention is, A shape acquisition unit capable of acquiring 3D shape information of the object to be processed, An image acquisition unit capable of acquiring image data including the aforementioned processing target, The system includes a densification unit capable of progressively increasing the density of the three-dimensional shape information based on the aforementioned image data, The image acquisition unit acquires first image data including at least a portion of the processing target, and second image data including a portion of the region included in the first image data, or in which the image included in the first image data has been given higher resolution. The aforementioned high-density section is First high-density shape information is generated by increasing the density of the three-dimensional shape information corresponding to the processing target contained in the first image data, and second high-density shape information is generated by increasing the density of the information corresponding to the processing target contained in the second image data from the first high-density shape information. It is an information processing device. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a technology that can acquire detailed three-dimensional shape information of a measurement target regardless of the distance from the measurement position to the measurement target. [Brief explanation of the drawing]

[0009] [Figure 1] Block diagram of a three-dimensional information processing device according to one embodiment. [Figure 2] Flowchart of 3D information processing according to one embodiment [Figure 3] Diagram illustrating 3D information processing according to one embodiment. [Figure 4] Block diagram of a three-dimensional information processing device according to one embodiment. [Figure 5] Flowchart of 3D information processing according to one embodiment [Figure 6] Block diagram of a three-dimensional information processing device according to one embodiment. [Figure 7] Flowchart of 3D information processing according to one embodiment [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] The blocks of the three-dimensional information processing device 100 according to this embodiment will be explained using Figure 1. The three-dimensional information processing device 100 is composed of a three-dimensional distance measuring unit 110, a wide-angle imaging unit 120, a telephoto imaging unit 130, a control unit 140, a non-volatile memory 150, and a system memory 160. In Figure 1, the arrow lines connecting the blocks indicate the flow of data from one block to another, and this data flow is executed by the control unit 140. The control unit 140 is connected to the wide-angle imaging unit 120, the telephoto imaging unit 130, the non-volatile memory 150, and the system memory 160, and controls the operation of these blocks.

[0012] The 3D distance measuring unit 110 (an example of a "shape acquisition unit capable of acquiring 3D shape information") is, for example, a LiDAR sensor, and measures the distance from the 3D distance measuring unit 110 to the object to be measured (an example of a "processing target"). The LiDAR sensor includes a light ray output unit that outputs a light ray and irradiates the surface of the object with the ray, and a receiving unit that receives the reflected light ray reflected from the surface of the object to be measured. The 3D distance measuring unit 110 calculates the distance to the surface of the object to be measured in the direction of irradiation using the time from irradiation of the light ray to reception of the reflected light ray, or the phase difference between the irradiated light ray and the reflected light ray. The 3D distance measuring unit 110 then acquires 3D shape information as a 3D point cloud using the calculated distance and the information of the direction of irradiation. The control unit 140 then stores the 3D point cloud in the 3D point cloud storage unit 152. Note that the 3D shape information is not limited to a 3D point cloud, and may be represented by known methods such as voxels, meshes, or implicit function representations. Furthermore, the 3D distance measuring unit 110 is not limited to a LiDAR sensor, but may also acquire 3D shape information of the object to be measured using, for example, a laser scanner, stereo vision, a moving image of the object to be measured, or computer graphics (CG).

[0013] The wide-angle imaging unit 120 and the telephoto imaging unit 130 (each an example of an "image acquisition unit") are, for example, digital still cameras, and can photograph the same measurement target. The wide-angle imaging unit 120 and the telephoto imaging unit 130 each include a lens unit with a different angle of view, an imaging device that converts an optical image into an electrical signal, and an A / D converter that converts an analog signal into a digital signal. Then, in the wide-angle imaging unit 120 and the telephoto imaging unit 130, an optical image is input to the imaging device via the lens unit, and an image is acquired by converting the electrical signal converted by the imaging device into a digital signal. The telephoto lens unit 131 of the telephoto imaging unit 130 has a narrower angle of view than the wide-angle lens unit 121 of the wide-angle imaging unit 120, and can acquire an image with higher resolution instead of a limited imaging area. Also, the control unit 140 causes the image data obtained from the wide-angle imaging unit 120 and the telephoto imaging unit 130 to be stored in the image storage unit 153, respectively.

[0014] The control unit 140 is configured to include at least one processor and controls the entire three-dimensional information processing apparatus 100. Such a control unit 140 realizes each process described later by executing a program recorded in the system storage unit 151. The control unit 140 can include a high-density point cloud generation unit 141 and a three-dimensional point cloud extraction unit 142. The high-density point cloud generation unit 141 generates a high-density three-dimensional point cloud from the three-dimensional point cloud recorded in the three-dimensional point cloud storage unit 152 and the image recorded in the image storage unit 153 using, for example, a method disclosed in Non-Patent Document 1. Then, the high-density point cloud generation unit 141 stores the generated high-density three-dimensional point cloud in the three-dimensional point cloud storage unit 152. Note that the high-density point cloud generation unit 141 is an example of a "densification unit capable of gradually densifying three-dimensional shape information".

[0015] The 3D point cloud extraction unit 142 extracts 3D point clouds from the 3D point clouds recorded in the 3D point cloud storage unit 152 that are included in the region of interest. The region of interest is defined as the region corresponding to the field of view of the telephoto imaging unit 130. The region of interest is set based on the relative position information of the telephoto imaging unit 130 with respect to the 3D distance measuring unit 110. Furthermore, the 3D point cloud extraction unit 142 uses the difference in field of view between the wide-angle imaging unit 120 and the telephoto imaging unit 130 to convert the position information of the extracted 3D point clouds into a coordinate system based on the position of the telephoto imaging unit 130. More specifically, the 3D point cloud extraction unit 142 calculates the focal length of the wide-angle imaging unit 120 from its size and field of view. Similarly, the 3D point cloud extraction unit 142 calculates the focal length of the telephoto imaging unit 130 from its size and field of view. The 3D point cloud extraction unit 142 then uses these focal lengths to calculate the ratio of the size of the measurement target included in the telephoto image (described later) to the size of the measurement target included in the wide-angle image (described later), and uses this ratio to perform an affine transformation on the positional information of the 3D point cloud. Note that the method of coordinate transformation is not limited to using focal lengths. For example, the size of the measurement target included in the images acquired by the wide-angle imaging unit 120 and the telephoto imaging unit 130 may be directly derived. Then, the above ratio may be calculated from that size, and an affine transformation may be performed using this ratio. The 3D point cloud extraction unit 142 then stores the positional information of the sparsely transformed 3D point cloud in the 3D point cloud storage unit 152.

[0016] The non-volatile memory 150 includes a system memory unit 151, a three-dimensional point group memory unit 152, and an image memory unit 153. The non-volatile memory 150 is an electrically erasable and recordable memory, such as an EEPROM or the like. In the system memory unit 151, operation programs of each block of the control unit 140, constants for operation, and the like are stored. Also, the system memory unit 151 stores the relative position information of the telephoto imaging unit 130 with respect to the three-dimensional ranging unit 110. Here, the program mentioned here includes programs for executing various flowcharts described later. The three-dimensional point group memory unit 152 stores the three-dimensional point group acquired by the three-dimensional ranging unit 110 and the position information of the three-dimensional point group converted by the three-dimensional point group extraction unit 142. The image memory unit 153 stores the image data captured by the wide-angle imaging unit 120 and the telephoto imaging unit 130, respectively. The system memory 160 is a rewritable volatile memory such as a DRAM or the like. In the system memory 160, constants, variables, and data read from the non-volatile memory 150 during the operation of the control unit 140 are expanded.

[0017] <Processing Example 1> Using FIGS. 2 and 3, a process of generating a high-density three-dimensional point group indicating a measurement target will be described. FIG. 2 shows a flowchart of this process, and FIG. 3 is a schematic diagram of a captured image and a three-dimensional point group indicating the measurement target included in the captured image. In FIG. 3, the dotted line indicates the wide-angle image, and the two-dot chain line indicates the telephoto image. Such a process is realized by the control unit 140 loading a program stored in the non-volatile memory 150 into the system memory 160 and executing it to control each component and the like.

[0018] In S201, a 3D point cloud and various images are acquired. Specifically, the 3D distance measuring unit 110 measures the distance from the 3D distance measuring unit 110 to the measurement target and outputs a 3D point cloud 320 of the measurement area using the measured distance information. The control unit 140 then stores the 3D point cloud 320 in the 3D point cloud storage unit 152. The wide-angle imaging unit 120 and the telephoto imaging unit 130 each photograph the measurement area and output a wide-angle image 310 (an example of "first image data") and a telephoto image 340 (an example of "second image data"). The control unit 140 then stores the wide-angle image 310 and the telephoto image 340 in the image storage unit 153. In Figure 3(A), the dashed-dot frame 311 included in the wide-angle image 310 indicates the range photographed by the telephoto imaging unit 130. Note that the measurement target measured by the 3D distance measuring unit 110 may include measurement targets not included in the wide-angle image 310. In other words, the wide-angle image 310 only needs to be an image that includes at least a portion of the object to be measured by the 3D distance measuring unit 110.

[0019] In S202, the high-density point cloud generation unit 141 combines the 3D point cloud 320 recorded in the 3D point cloud storage unit 152 with the wide-angle image 310 recorded in the image storage unit 153 to generate a wide-angle high-density point cloud 330 (an example of "first high-density shape information"). More specifically, as shown in Figure 3(A), the high-density point cloud generation unit 141 generates the wide-angle high-density point cloud 330 by adding points to the 3D point cloud 320 (an example of "3D shape information corresponding to the processing target") so as to match the measurement target included in the wide-angle image 310. Point addition is performed by methods such as Depth Completion. The high-density point cloud generation unit 141 then stores the generated wide-angle high-density point cloud 330 in the 3D point cloud storage unit 152. Note that the wide-angle high-density point cloud 330 is not limited to being generated from the wide-angle image 310 and the 3D point cloud 320. The high-density point cloud generation unit 141 may generate a wide-angle high-density point cloud 330 from the 3D point cloud 320 without using the wide-angle image 310 by, for example, other deep learning methods such as CNNs or diffusion generation models, or non-deep learning methods such as bicubic methods. The high-density point cloud generation unit 141 may, before generating the wide-angle high-density point cloud 330 from the 3D point cloud 320, use the relative position information of the wide-angle imaging unit 120 with respect to the 3D distance measuring unit 110 to translate the 3D point cloud 320, for example, in a direction parallel to or perpendicular to the image plane. Such a translation is an example of "converting the first high-density shape information from a coordinate system based on the shape acquisition unit to a coordinate system based on the image acquisition unit."

[0020] In S203, the 3D point cloud extraction unit 142 sets the region of interest 331 using the position information of the dotted frame 311 included in the wide-angle image 310 and the relative position information of the telephoto imaging unit 130 with respect to the 3D distance measuring unit 110. The 3D point cloud extraction unit 142 then extracts the point clouds included in the region of interest 331 from the wide-angle high-density point cloud 330 recorded in the 3D point cloud storage unit 152 as the region of interest point cloud 350. Figure 3(A) illustrates such a region of interest 331 and region of interest point cloud 350. The 3D point cloud extraction unit 142 then uses the difference in field of view between the wide-angle imaging unit 120 and the telephoto imaging unit 130 to convert the position information of the extracted region of interest point cloud 350 into the coordinate system of the telephoto imaging unit 130. When the position information of the region of interest point cloud 350 is converted in this way, a sparse region of interest point cloud 351 is generated, as shown in Figure 3(B).

[0021] The region of interest 331 is set using relative position information of the telephoto imaging unit 130 relative to the 3D distance measuring unit 110, which is stored in the system memory unit 151 in advance, but the setting method is not limited to this. The region of interest 331 may also be set using, for example, the relative position information of the telephoto imaging unit 130 relative to the 3D distance measuring unit 110, which is dynamically calculated by comparing the feature quantities of the wide-angle high-density point cloud 330 and the telephoto image 340. Feature quantities include, for example, SIFT (Scale-Invariant Feature Transform) features or features extracted using CNN (Convolutional Neural Network). Using such features allows for correct matching of the feature quantities of measurement targets contained in images with different scales, such as the wide-angle image 310 and the telephoto image 340, making it possible to calculate relative position information between measurement targets. Alternatively, the region of interest 331 may be set from the features of the telephoto image 340.

[0022] In S204, the high-density point cloud generation unit 141 generates a higher-density telephoto high-density point cloud 360 by combining the point cloud of the region of interest 350, which was extracted in S203 and coordinate-transformed to the coordinate system of the telephoto imaging unit 130, with the telephoto image 340 recorded in the image storage unit 153. More specifically, the high-density point cloud generation unit 141 generates the telephoto high-density point cloud 360 by adding points to the point cloud of interest 350 to match the measurement target included in the telephoto image 340. The generation method is, for example, the same as the method in S202. Figure 3(B) shows an example of the telephoto high-density point cloud 360 generated in this way. After that, the processing of this flowchart is terminated. Note that the telephoto high-density point cloud 360 is an example of "second high-density shape information".

[0023] <One aspect of action / effect> When distance information from the 3D distance measuring unit 110 to the measurement target is acquired sparsely, the 3D point cloud 320 of the measurement area becomes sparse, as shown in Figure 3(A). However, the 3D information processing device 100 makes it possible to acquire a telephoto high-density point cloud 360 by gradually fitting such a 3D point cloud 320 to the measurement target included in the wide-angle image 310 and the telephoto image 340. Therefore, detailed 3D shape information of the measurement target can be acquired.

[0024] (First variation) The 3D information processing device 100 according to the first modified example is configured to include a single imaging unit equipped with a zoom lens (an example of an "optical lens"). Furthermore, in the 3D information processing device 100, the same imaging unit captures the measurement target multiple times, and the captured images are aligned. These points differ from the 3D information processing device 100 according to the embodiment. In the following, the same configuration as in the embodiment will be omitted from the description, and the focus will be on the parts that differ from the embodiment.

[0025] Using Figure 4, the blocks of the 3D information processing device 100 according to the first modified example will be explained. The 3D information processing device 100 according to the first modified example includes an imaging unit 420 as a substitute for the wide-angle imaging unit 120 and telephoto imaging unit 130 according to the embodiment. The 3D information processing device 100 according to the first modified example also includes a 3D distance measuring unit 110, a control unit 140, a non-volatile memory 150, and a system memory 160, similar to the 3D information processing device 100 according to the embodiment.

[0026] The imaging unit 420 is a digital still camera that includes, for example, a zoom drive mechanism (an example of a "zoom mechanism that allows the lens to move in the optical axis direction so that the processing target can be imaged at two different angles of view"). The imaging unit 420 also includes a zoom lens unit 421 in which the shooting angle of view can be changed according to user operation. That is, the zoom lens unit 421 consists of a zoom lens and a zoom mechanism that drives the zoom lens. The zoom function is realized when the lens drive unit moves the zoom lens in the optical axis direction.

[0027] The control unit 140 includes a position difference calculation unit 443 in addition to the high-density point cloud generation unit 141 and the 3D point cloud extraction unit 142 according to the embodiment. The position difference calculation unit 443 extracts matching features between the two images by comparing the feature quantities of the wide-angle image 310 and the telephoto image 340 recorded in the image storage unit 153. Then, using the position information of the extracted feature quantities, it calculates the position difference of the imaging unit 420 between one shooting and another shooting. The feature quantities are, as described above, for example, SIFT features or features extracted using a CNN. The method for calculating the position difference is not limited to this. For example, if the 3D information processing device 100 is equipped with a GPS or IMU (Inertial Measurement Unit), the position difference may be calculated using the information output from these. Alternatively, if time information of when the measurement target was imaged, or feature quantities on the 3D point cloud are recorded in the image data, the position difference may be calculated using this information.

[0028] <Processing Example 2> Figures 5 and 6 illustrate another example of the process for generating a high-density 3D point cloud representing the measurement target. This process is achieved by the control unit 140 according to the first modified example loading a program stored in the non-volatile memory 150 into the system memory 160 and executing it, thereby controlling each component. S202, S203, and S204 are the same as in the embodiment and are therefore omitted from explanation.

[0029] In S501, the 3D distance measuring unit 110 measures the distance from the 3D distance measuring unit 110 to the object to be measured and outputs a 3D point cloud 320 of the measurement area using the measured distance information. The control unit 140 then stores the 3D point cloud 320 in the 3D point cloud storage unit 152. The imaging unit 420 also photographs the object to be measured without using the zoom function and stores the captured image as a wide-angle image 310 in the image storage unit 153. The imaging unit 420 also photographs the object to be measured using the zoom function and stores the captured image as a telephoto image 340 in the image storage unit 153. The subsequent processing proceeds in the order of S202 and S502. In S502, the zoom lens unit 421 of the imaging unit 420 drives the zoom lens to photograph the object to be measured with a narrower field of view than in step S501. The control unit 140 then stores the captured image as a telephoto image 340 in the image storage unit 153.

[0030] In S503, the position difference calculation unit 443 calculates the difference between the position captured by the imaging unit 420 at a narrow angle and the position captured by the imaging unit 420 at a wide angle, based on the feature quantities of the wide-angle image 310 and the telephoto image 340 recorded in the image storage unit 153. Then, using this position difference, the telephoto image 340 is aligned with the wide-angle image 310. The position difference calculation unit 443 also stores the calculated position difference in the system storage unit 151 as relative position information of the telephoto imaging unit 130 with respect to the 3D distance measuring unit 110 according to this embodiment. The process then proceeds to S203. In S203, the 3D point cloud extraction unit 142 converts the position information of the extracted point cloud of interest 350 into the coordinate system of the telephoto image 340 captured at a high zoom magnification. The other processing contents are the same as in S203 according to this embodiment. The process then proceeds to S204, ending the processing in this flowchart.

[0031] <One aspect of action / effect> According to the 3D information processing device 100 of the first modified example, the positioning of a telephoto image taken at a different timing is performed with respect to a wide-angle image. By performing this process, even when the wide-angle imaging unit and the telephoto imaging unit are not provided as separate units, detailed 3D shape information of the object to be measured can be obtained in the same way as in the 3D information processing device 100 of the embodiment.

[0032] In the first modified example, the imaging unit 420 equipped with a zoom lens unit generates the wide-angle image 310 and the telephoto image 340. However, for example, a camera with interchangeable lenses may be used, and lenses with different angles of view may be changed each time a picture is taken. The wide-angle image 310 and the telephoto image 340 can also be generated by such a camera with interchangeable lenses.

[0033] (Second variation) The 3D information processing device 100 according to the second modified example differs from the 3D information processing device 100 according to the embodiment and the first modified example in that a telephoto image is captured before a wide-angle image is captured. Then, the feasibility of increasing the density of the point cloud is determined using the telephoto image, and based on the determination result, the capture of the wide-angle image is performed. In the following, the same configuration as in the embodiment or the first modified example will be omitted from the explanation, and the explanation will focus on the differences.

[0034] Using Figure 6, the blocks of the 3D information processing device 100 according to the second modified example will be explained. The 3D information processing device 100 according to the second modified example includes a control unit 140, which includes a high-density point cloud generation unit 141, a 3D point cloud extraction unit 142, and a position difference calculation unit 443 according to the first modified example, as well as a generation determination unit 644 and a zoom control unit 645. The generation determination unit 644 determines whether or not a high-density point cloud can be generated by matching the 3D point cloud 320 recorded in the 3D point cloud storage unit 152 to the measurement target included in the telephoto image 340 recorded in the image storage unit 153. Details of the determination method will be described later. The zoom control unit 645 controls the zoom lens unit 421 based on the determination result of the generation determination unit 644. Furthermore, the 3D information processing device 100 according to the second modified example includes a 3D distance measuring unit 110, an imaging unit 420, a non-volatile memory 150, and a system memory 160, similar to the first modified example.

[0035] <Processing Example 3> Figure 7 illustrates another example of the process for generating a high-density 3D point cloud representing the measurement target. This process is achieved by the control unit 140 loading a program stored in the non-volatile memory 150 into the system memory 160 and executing it, thereby controlling each component. S203, S204, S502, and S503 are the same as in the embodiment or the first modified example, so their explanation is omitted.

[0036] In S701, the 3D distance measuring unit 110 measures the distance from the 3D distance measuring unit 110 to the object to be measured and outputs a 3D point cloud 320 of the measurement area using the measured distance information. The control unit 140 then stores the 3D point cloud 320 in the 3D point cloud storage unit 152. The zoom lens unit 421 of the imaging unit 420 drives the zoom lens to photograph the object to be measured with a narrow field of view. The control unit 140 then stores the captured image data as a telephoto image 340 in the image storage unit 153.

[0037] In S702, the generation determination unit 644 determines whether or not a telephoto high-density point cloud 360 can be generated by matching the 3D point cloud 320 recorded in the 3D point cloud storage unit 152 with the measurement targets included in the telephoto image 340 recorded in the image storage unit 153. More specifically, the generation determination unit 644 calculates the ratio of the number of pixels in the telephoto image 340 to the number of points in the 3D point cloud that represent the measurement targets included in the telephoto image 340. This ratio is calculated based on the angular resolution of the 3D distance measuring unit 110 that measured the 3D point cloud 320, the resolution of the telephoto image 340, and the field of view of the imaging unit 420 that captured the telephoto image 340. The generation determination unit 644 then determines whether or not this ratio is below a predetermined threshold. If the generation determination unit 644 determines that this ratio is below the threshold, it determines that a telephoto high-density point cloud 360 can be generated, and the process proceeds to S703; otherwise, it proceeds to S704.

[0038] In S703, the 3D point cloud extraction unit 142, in response to the determination in S702 that a telephoto high-density point cloud 360 can be generated, sets the region of interest 331 using the relative position information of the telephoto imaging unit 130 with respect to the 3D distance measuring unit 110. This relative position information is stored in the system storage unit 151. Then, from the 3D point cloud 320 recorded in the 3D point cloud storage unit 152, points included in the region of interest 331 are extracted as the region of interest point cloud 350. After that, the process proceeds to S204.

[0039] On the other hand, if it is determined in S702 that a telephoto high-density point cloud 360 cannot be generated, in S704 the zoom control unit 645 drives the zoom lens unit 421 to move the lens. The imaging unit 420 then photographs the object to be measured with a wide angle of view. The control unit 140 then stores the captured image data as a wide-angle image 310 in the image storage unit 153. The subsequent processing proceeds in the order of S502, S503, S203, and S204. This completes the processing of this flowchart.

[0040] <One aspect of action / effect> The flowchart according to the first modified example (Figure 5) has an increased number of steps compared to the flowchart according to the embodiment (Figure 2), resulting in increased processing time. Therefore, in order to shorten the processing time, the 3D information processing device 100 according to the second modified example determines in advance whether or not high-density 3D point cloud can be created before capturing the wide-angle image 310. If it is determined that high-density 3D point cloud can be created, a telephoto high-density point cloud 360 is generated without capturing the wide-angle image 310. With such a 3D information processing device 100, detailed 3D shape information of the measurement target can be efficiently acquired.

[0041] (Other variations) In this embodiment, the 3D point cloud is densified twice using two images, a wide-angle image 310 and a telephoto image 340. However, the number of images and the number of densification steps are not limited to those described above. For example, three or more images may be used to perform densification three or more times. In this embodiment, sets of the telephoto image 340 and the wide-angle image 310 used for the two densification steps are captured, but these sets may be generated from a single image. For example, the 3D information processing device 100 may include an image extraction unit that creates a cropped image by cutting out a portion of the wide-angle image output (provided) by the wide-angle imaging unit 120. That is, the control unit 140 may acquire the wide-angle image from the wide-angle imaging unit 120 and transmit it to the image extraction unit. The control unit 140 may then acquire the cropped image created by the image extraction unit.

[0042] Alternatively, the 3D information processing device 100 may include an image processing unit that generates an image of the same field of view with reduced resolution by processing the telephoto image. That is, the control unit 140 may acquire the telephoto image from the telephoto imaging unit 130 and transmit it to the image processing unit. The control unit 140 may then acquire the low-resolution image created by the image processing unit. The above set may be generated by a combination of a wide-angle image and a cropped image (a substitute for the telephoto image 340), or by a combination of a telephoto image and its low-resolution image (a substitute for the wide-angle image 310). The various controls described above, which are performed by the control unit 140, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0043] Furthermore, a stereo camera may be used in the 3D distance measuring unit 110. Also, the imaging unit may be provided separately from the 3D information processing unit 100. In addition, the object to be measured may not be photographed, and the wide-angle image 310 or telephoto image 340 may be an image artificially synthesized using CG or the like, or the 3D information processing unit may be connected to a network and acquired from other devices via the network. Furthermore, the resolution of the telephoto image 340 does not have to be low.

[0044] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0045] The present invention can also be realized by performing the following process: that is, software (program) that realizes the functions of the above-described embodiment is supplied to a system or device via a network or various storage media. The computer (or CPU, MPU, etc.) of that system or device then reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0046] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or recording medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0047] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0048] The disclosures herein include the following information processing devices, methods for controlling the information processing devices, and control programs for the information processing devices. [Item 1] A shape acquisition unit capable of acquiring 3D shape information of the object to be processed, An image acquisition unit capable of acquiring image data including the aforementioned processing target, The system includes a densification unit capable of progressively increasing the density of the three-dimensional shape information based on the aforementioned image data, The image acquisition unit acquires first image data including at least a portion of the processing target, and second image data including a portion of the region included in the first image data, or in which the image included in the first image data has been given higher resolution. The aforementioned high-density section is First high-density shape information is generated by increasing the density of the three-dimensional shape information corresponding to the processing target contained in the first image data, and second high-density shape information is generated by increasing the density of the information corresponding to the processing target contained in the second image data from the first high-density shape information. Information processing device. [Item 2] The image acquisition unit is equipped with two optical lenses with different angles of view. The first image data is acquired using an optical lens with a wide field of view. The second image data is acquired using an optical lens with a narrow field of view. The information processing device described in item 1. [Item 3] The image acquisition unit, Equipped with a zoom mechanism, The object to be processed is imaged at a first zoom magnification in the zoom mechanism, and the imaged data is acquired as the first image data. The object to be processed is imaged at a second zoom magnification higher than the first zoom magnification of the first image data, and the imaged data is acquired as the second image data. The information processing device described in item 1. [Item 4] Using the ratio of the size of the processing target included in the second image data to the size of the processing target included in the first image data, the first high-density shape information is converted from a coordinate system based on the shape acquisition unit to a coordinate system based on the image acquisition unit. An information processing device described in any one of items 1 to 3. [Item 5] The system further includes an image extraction unit that extracts a portion from the first image data to obtain the second image data, The image acquisition unit, The first image data is acquired and provided to the image extraction unit. The image extraction unit acquires the second image data it has extracted. An information processing device described in any one of items 1 through 4. [Item 6] It further includes an image processing unit that can change the image resolution, The image acquisition unit acquires the second image data and provides it to the image processing unit. The image processing unit generates the first image data by reducing the resolution of the second image data. The image acquisition unit acquires the first image data generated by the image processing unit. An information processing device described in any one of items 1 through 5. [Item 7] The system further includes a determination unit that determines whether or not it is possible to generate second high-density shape information by increasing the density of the three-dimensional shape information corresponding to the processing target included in the second image data, If the determination unit determines that it is possible to generate, The image acquisition unit does not acquire the first image data. The densification unit generates second high-density shape information by densifying the three-dimensional shape information corresponding to the processing target contained in the second image data, without using the first image data. An information processing device described in any one of items 1 through 6. [Item 8] The aforementioned 3D shape information is 3D point cloud information, The determination unit determines that the second high-density shape information can be generated if the ratio of the number of pixels in the second image data to the number of points in the three-dimensional point cloud corresponding to the processing target included in the second image data is less than or equal to a predetermined threshold. The information processing device described in item 7. [Item 9] A method for controlling an information processing device, The shape acquisition unit includes a shape acquisition process that can acquire 3D shape information of the object to be processed, The image acquisition unit includes an image acquisition step that can acquire image data including the processing target, The densification unit includes a densification step that can progressively increase the density of the three-dimensional shape information based on the image data, The image acquisition unit acquires first image data including at least a portion of the processing target, and second image data including a portion of the region included in the first image data, or in which the image included in the first image data has been given higher resolution. The aforementioned high-density section is First high-density shape information is generated by increasing the density of the three-dimensional shape information corresponding to the processing target contained in the first image data, and second high-density shape information is generated by increasing the density of the information corresponding to the processing target contained in the second image data from the first high-density shape information. A method for controlling an information processing device. [Item 10] A program for causing a computer to execute each step in a control method for an information processing device, wherein the control method is: The shape acquisition unit includes a shape acquisition process that can acquire 3D shape information of the object to be processed, The image acquisition unit includes an image acquisition step that can acquire image data including the processing target, The densification unit includes a densification step that can progressively increase the density of the three-dimensional shape information based on the image data, The image acquisition unit acquires first image data including at least a portion of the processing target, and second image data including a portion of the region included in the first image data, or in which the image included in the first image data has been given higher resolution. The aforementioned high-density section is First high-density shape information is generated by increasing the density of the three-dimensional shape information corresponding to the processing target contained in the first image data, and second high-density shape information is generated by increasing the density of the information corresponding to the processing target contained in the second image data from the first high-density shape information. program. [Explanation of Symbols]

[0049] 100: 3D information processing unit, 110: 3D distance measuring unit, 120: Wide-angle imaging unit, 130: Telephoto imaging unit, 140: Control unit, 141: High-density point cloud generation unit, 142: 3D point cloud extraction unit, 150: Non-volatile memory, 151: System memory unit, 152: 3D point cloud memory unit, 153: Image memory unit, 160: System memory, 310: Wide-angle image, 320: 3D point cloud, 330: Wide-angle high-density point cloud, 331: Area of ​​interest, 340: Telephoto image, 350, 351: Point cloud of area of ​​interest, 360: Telephoto high-density point cloud, 420: Imaging unit, 421: Zoom lens unit, 443: Position difference calculation unit, 644: Generation determination unit, 645: Zoom control unit

Claims

1. A shape acquisition unit capable of acquiring 3D shape information of the object to be processed, An image acquisition unit capable of acquiring image data including the aforementioned processing target, The system includes a densification unit capable of progressively increasing the density of the three-dimensional shape information based on the aforementioned image data, The image acquisition unit acquires first image data including at least a part of the object to be processed, and second image data including a part of the region included in the first image data, or in which the image included in the first image data has been given higher resolution. The aforementioned high-density section is First high-density shape information is generated by increasing the density of the three-dimensional shape information corresponding to the processing target included in the first image data, and second high-density shape information is generated by increasing the density of the information corresponding to the processing target included in the second image data from the first high-density shape information. Information processing device.

2. The image acquisition unit is equipped with two optical lenses with different angles of view. The first image data is acquired using an optical lens with a wide field of view. The second image data is acquired using an optical lens with a narrow field of view. The information processing apparatus according to claim 1.

3. The image acquisition unit, Equipped with a zoom mechanism, The object to be processed is imaged at a first zoom magnification in the zoom mechanism, and the imaged data is acquired as the first image data. The object to be processed is imaged at a second zoom magnification higher than the first zoom magnification of the first image data, and the imaged data is acquired as the second image data. The information processing apparatus according to claim 1.

4. Using the ratio of the size of the processing target included in the second image data to the size of the processing target included in the first image data, the first high-density shape information is converted from a coordinate system based on the shape acquisition unit to a coordinate system based on the image acquisition unit. The information processing apparatus according to claim 1.

5. The system further includes an image extraction unit that extracts a portion from the first image data to obtain the second image data, The image acquisition unit, The first image data is acquired and provided to the image extraction unit. The image extraction unit acquires the second image data it has extracted. The information processing apparatus according to claim 1.

6. It further includes an image processing unit that can change the image resolution, The image acquisition unit acquires the second image data and provides it to the image processing unit. The image processing unit generates the first image data by reducing the resolution of the second image data. The image acquisition unit acquires the first image data generated by the image processing unit. The information processing apparatus according to claim 1.

7. The system further includes a determination unit that determines whether or not it is possible to generate second high-density shape information by increasing the density of the three-dimensional shape information corresponding to the processing target included in the second image data, If the determination unit determines that it is possible to generate, The image acquisition unit does not acquire the first image data. The densification unit generates the second high-density shape information by densifying the three-dimensional shape information corresponding to the processing target contained in the second image data, without using the first image data. An information processing apparatus according to any one of claims 1 to 6.

8. The aforementioned three-dimensional shape information is three-dimensional point cloud information, The determination unit determines that the second high-density shape information can be generated if the ratio of the number of pixels in the second image data to the number of points in the three-dimensional point cloud corresponding to the processing target included in the second image data is less than or equal to a predetermined threshold. The information processing apparatus according to claim 7.

9. A method for controlling an information processing device, The shape acquisition unit includes a shape acquisition process that can acquire three-dimensional shape information of the object to be processed, The image acquisition unit includes an image acquisition step that can acquire image data including the processing target, The densification unit includes a densification step that enables the densification of the three-dimensional shape information in stages based on the image data, The image acquisition unit acquires first image data including at least a part of the object to be processed, and second image data including a part of the region included in the first image data, or in which the image included in the first image data has been given higher resolution. The aforementioned high-density section is First high-density shape information is generated by increasing the density of the three-dimensional shape information corresponding to the processing target included in the first image data, and second high-density shape information is generated by increasing the density of the information corresponding to the processing target included in the second image data from the first high-density shape information. A method for controlling an information processing device.

10. A program for causing a computer to execute each step in a control method for an information processing device, wherein the control method is: The shape acquisition unit includes a shape acquisition process that can acquire three-dimensional shape information of the object to be processed, The image acquisition unit includes an image acquisition step that can acquire image data including the processing target, The densification unit includes a densification step that enables the densification of the three-dimensional shape information in stages based on the image data, The image acquisition unit acquires first image data including at least a part of the object to be processed, and second image data including a part of the region included in the first image data, or in which the image included in the first image data has been given higher resolution. The aforementioned high-density section is First high-density shape information is generated by increasing the density of the three-dimensional shape information corresponding to the processing target included in the first image data, and second high-density shape information is generated by increasing the density of the information corresponding to the processing target included in the second image data from the first high-density shape information. program.

Images