Three-dimensional information processing apparatus and three-dimensional information processing method

Abstract

To provide a three-dimensional information processing apparatus which can efficiently generate accurate three-dimensional shape data.SOLUTION: The three-dimensional information processing apparatus acquires three-dimensional shape data representing a shape of a three-dimensional object and data of an image of an imaging range including the three-dimensional object. The three-dimensional information processing apparatus generates a depth map corresponding to the imaging range from the three-dimensional shape data and increases a resolution of a region corresponding to a region of interest set for the image in the depth map. Then, the three-dimensional information processing apparatus generates three-dimensional shape data on the basis of the depth map in which the resolution of the region corresponding to the region of interest is increased.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a three-dimensional information processing device and a three-dimensional information processing method. [Background technology]

[0002] In recent years, shape data of three-dimensional objects (three-dimensional shape data) has come into widespread use. When expressing the shape of a three-dimensional object as a set of three-dimensional coordinate data (point cloud data), the higher the density of the three-dimensional coordinates, the more accurately the shape can be expressed.

[0003] However, increasing the density of 3D coordinates increases the processing load when generating and using 3D shape data. The volume of 3D shape data also increases. Furthermore, devices for measuring 3D shapes with high accuracy (such as LiDAR (Light Detection and Ranging) sensors) are generally large and expensive.

[0004] As a method for reducing the amount of calculation required to obtain three-dimensional shape data, Patent Document 1 proposes a method in which the shape of a three-dimensional object is understood as a collection of surfaces, and the contour is estimated as existing at the boundaries of the surfaces. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2012-13660 A Summary of the Invention [Problem to be solved by the invention]

[0006] In order to improve the accuracy of 3D shape data using the method proposed in Patent Document 1, it is necessary to estimate the contour line with high accuracy. In order to estimate the contour line with high accuracy, it is necessary to improve the accuracy of surface detection. As a result, it is necessary to acquire 3D object point cloud data for detecting surfaces at a high density overall, and problems such as an increase in the processing load required for generating point cloud data and an increase in the size and cost of the measuring device cannot be solved.

[0007] In view of the problems with the conventional techniques, one aspect of the present invention provides a three-dimensional information processing apparatus and a three-dimensional information processing method capable of efficiently generating accurate three-dimensional shape data. [Means for solving the problem]

[0008] In one aspect, the present invention provides a three-dimensional information processing device comprising: a first acquisition means for acquiring three-dimensional shape data representing the shape of a three-dimensional object; a second acquisition means for acquiring image data of an imaging range including the three-dimensional object; a conversion means for generating a depth map corresponding to the imaging range from the three-dimensional shape data; a resolution conversion means for increasing the resolution of a region of the depth map that corresponds to a region of interest set for the image; and an inverse conversion means for generating three-dimensional shape data based on the depth map in which the resolution of the region corresponding to the region of interest has been increased. Effect of the Invention

[0009] According to one aspect of the present invention, it is possible to provide a three-dimensional information processing apparatus and a three-dimensional information processing method capable of efficiently generating accurate three-dimensional shape data. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of a three-dimensional information processing system according to a first embodiment; [Diagram 2] Flowchart for three-dimensional shape data generation processing in the first embodiment [Diagram 3] Schematic diagram of the process of generating 3D shape data in the first embodiment. [Figure 4] Schematic diagram of the process of generating 3D shape data in the first embodiment. [Diagram 5] Flowchart for attention area setting processing in the first embodiment [Figure 6] FIG. 11 is a block diagram showing an example of the functional arrangement of a three-dimensional information processing system according to a second embodiment. [Figure 7] Flowchart of 3D shape data generation process in the second embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0012] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in a personal computer (PC). However, the present invention can be implemented in any electronic device that uses a microprocessor. Such electronic devices include computer devices (tablet computers, media players, PDAs, etc.), smartphones, game consoles, robots, drones, drive recorders, etc. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0013] ●(First embodiment) 1 is a block diagram showing an example of a functional configuration of a three-dimensional information processing system 1 according to a first embodiment of the present invention. The three-dimensional information processing system 1 includes a three-dimensional information processing device 100, a distance measuring device 10, and an imaging device 11.

[0014] The three-dimensional information processing apparatus 100 includes a non-volatile memory 110, a system memory 120, and a control unit 150. The three-dimensional information processing apparatus 100 may be, for example, a personal computer.

[0015] The distance measuring device 10 measures data related to the three-dimensional shape of a target object. The distance measuring device 10 is, for example, a LiDAR sensor, and performs distance measurement based on the time difference (ToF) between emitting a light beam and detecting the reflected light while changing the direction of the light beam, thereby generating a group of distance data having a predetermined spatial resolution. The distance data can be converted into three-dimensional coordinates based on the direction of the light beam. A set of three-dimensional coordinates of the object surface (point cloud data) represents the three-dimensional shape of the object surface, and can therefore be treated as three-dimensional shape data. Note that the distance may be measured using the phase difference between the emitted light and the reflected light.

[0016] In this embodiment, the three-dimensional shape data is handled in the form of point cloud data. However, the point cloud data may be handled in another form, such as voxels, meshes, or implicit functions.

[0017] Furthermore, the distance measuring device 10 is not limited to a configuration using a LiDAR sensor, and may be a configuration using, for example, a laser scanner, stereo vision, 3D reconstruction from a moving image, or the like.

[0018] The imaging device 11 is, for example, a camera, and outputs image data representing an image of a field of view including the distance measurement range of the distance measurement device 10. The imaging device 11 has a lens unit, an imaging element that converts an optical image formed on an imaging surface by the lens unit into an analog image signal, and an A / D converter that converts the analog image signal into a digital image signal (image data). Note that the image data output by the imaging device 11 may be in a state before color interpolation processing (RAW format) or in a state after color interpolation processing.

[0019] The distance measuring device 10 and the imaging device 11 are communicatively connected through an interface of the three-dimensional information processing device 100. The operations of the distance measuring device 10 and the imaging device 11 are controlled by a control unit 150. The three-dimensional information processing device 100 may include at least one of the distance measuring device 10 and the imaging device 11.

[0020] Note that the position and orientation information of the distance measuring device 10 and the imaging device 11, and information for identifying the imaging range of the imaging device 11 (information relating to the focal length of the lens unit and the optical axis direction) can be stored in advance in the non-volatile memory 110 (system storage unit 113). Basically, the imaging range and imaging direction of the imaging device 11 are set so as to capture an image of a range that includes the distance measuring range of the distance measuring device 10. Alternatively, the distance measuring device 10 may be set so as to generate point cloud data within the imaging range of the imaging device 11.

[0021] The distance measuring device 10 measures point cloud data at a first density (spatial resolution) that is determined in advance for the entire measurement range. When using a LiDAR sensor, for example, the density of the point cloud data is determined by the spatial resolution of the irradiation position of the light beam.

[0022] The nonvolatile memory 110 is electrically rewritable. A three-dimensional point cloud storage unit 111, an image storage unit 112, and a system storage unit 113 shown as components of the nonvolatile memory 110 may each be a part of the memory space of the nonvolatile memory 110.

[0023] The three-dimensional point cloud storage unit 111 stores the point cloud data acquired from the distance measuring device 10. The image storage unit 112 stores the image data acquired from the imaging device 11.

[0024] The system storage unit 113 stores the programs (OS, applications) executed by the control unit 150, various setting values, GUI data, and the like.

[0025] The system memory 120 is a rewritable volatile memory such as a DRAM, etc. The system memory 120 temporarily stores programs executed by the control unit 150, constants and variables used by the programs being executed, data read from the non-volatile memory 101, and the like.

[0026] When a user interactively operates the three-dimensional information processing device 100, the three-dimensional information processing system includes a display device and a user interface device connected to or built into the three-dimensional information processing device 100. The user interface device may be a keyboard, a mouse, a touch pad, or the like. When the display device is a touch display, the touch display may also function as the user interface device.

[0027] The control unit 150 is, for example, a processor (CPU, MPU, microprocessor, etc.) capable of executing a program. The control unit 150 loads a program stored in the system storage unit 113 into the system memory 120 and executes the program, thereby realizing the functions of the three-dimensional information processing device 100, including the generation process of three-dimensional shape data, which will be described later.

[0028] In addition, the image extraction unit 151, the 3D point cloud extraction unit 152, the high density point cloud generation unit 153, and the 3D point cloud replacement unit 154 described in the figure as components of the control unit 150 indicate functions realized by the control unit 150 executing a program. Therefore, the operations performed by the image extraction unit 151, the 3D point cloud extraction unit 152, the high density point cloud generation unit 153, or the 3D point cloud replacement unit 154 are actually executed by the control unit 150.

[0029] Among the functions realized by the control unit 150 executing a program, for example, processing with a large computation load may be performed using a hardware circuit. For example, computation processing related to image processing or machine learning may be performed using a hardware circuit (GPU, NPU, etc.) suitable for these processes.

[0030] The image extraction unit 151 extracts an area of ​​interest from an image stored in the image storage unit 112, for example, based on the feature amount of the image. The image extraction unit 151 stores data of the extracted area of ​​interest in the image storage unit 112. The feature amount may be, for example, a feature amount for detecting an area of ​​a predetermined type of subject. For example, the feature amount may be stored in the non-volatile memory 110 in association with the type of area of ​​interest, and the image extraction unit 151 may select and use the feature amount to be used according to, for example, a setting.

[0031] The three-dimensional point cloud extraction unit 152 extracts point cloud data included in the region of interest from the point cloud data stored in the three-dimensional point cloud storage unit 111. The three-dimensional point cloud extraction unit 152 can specify the point cloud data included in the region of interest using the position information of the distance measuring device 10 and the imaging device 11 and the image coordinates of the region of interest stored in the system storage unit 113. The three-dimensional point cloud extraction unit 152 stores the extracted point cloud data in the three-dimensional point cloud storage unit 111.

[0032] The three-dimensional point cloud extraction unit 152 may specify the point cloud data included in the region of interest by another method. For example, the point cloud data included in the region of interest may be specified based on the feature amount used by the image extraction unit 151.

[0033] The high density point cloud generating unit 153 (resolution conversion means) generates point cloud data of a second density higher than the first density based on a known method from the region of interest extracted by the image extracting unit 151 and the point cloud data of the first density extracted by the 3D point cloud extracting unit 152. The operation of the high density point cloud generating unit 153 will be described in detail later.

[0034] The 3D point cloud replacing unit 154 synthesizes the second density point cloud data generated by the high density point cloud generating unit 153 with the first density point cloud data generated by the distance measuring device 10. The synthesis may be, for example, replacing, among the first density point cloud data, point cloud data included in the range of the second density point cloud data generated by the high density point cloud generating unit 153 with the second density point cloud data.

[0035] In this manner, in this embodiment, among the point cloud data of the first density measured by the distance measuring device 10, the density of the point cloud data in the characteristic image region is increased to the second density. Therefore, it is not necessary to obtain point cloud data of the second density for the entire measurement range, and point cloud data of the second density is obtained for the region of interest. Therefore, by configuring the region requiring high-density point cloud data to be extracted as the region of interest, it becomes possible to efficiently generate point cloud data while suppressing an increase in the processing load and data amount.

[0036] The above-mentioned operation of generating three-dimensional shape data will be further explained below. 3 is a diagram showing a schematic diagram of a generation process of three-dimensional shape data in this embodiment. An original image 310 is an original image obtained from the imaging device 11, and an attention area 311 is an attention area extracted from the original image 310 by the image extraction unit 151. Also, point cloud data 320 shows a schematic diagram of point cloud data obtained by the distance measuring device 10 within the field of view of the imaging device 11. Two black dots 321 show distance information for close distances, and seven white dots 322 show distance information for long distances.

[0037] The depth image 330 is an image (depth map) in which the point cloud data 320 is projected onto a two-dimensional space corresponding to the original image 310, and each pixel indicates a distance in the imaging direction of the imaging device 11. Although the depth image 330 is referred to as a "depth image" here for convenience, the data forming the depth image 330 does not have to be in a form that can be particularly viewed, and may be a data group in which distance values ​​and data corresponding to the distance values ​​are arranged in correspondence with the original image 310. The area 331 indicates an area corresponding to the attention area 311 in the depth image. The depth image 340 is an area in which the resolution (number of pixels) of the area 331 in the depth image 330 is increased. The point cloud data 350 indicates a state in which the depth image 340 is reversely projected onto a three-dimensional space. The data of the point cloud data 320 that is not projected onto the area 331 and the point cloud data 350 are combined to obtain three-dimensional shape data.

[0038] Fig. 4(a) shows a state in which multiple attention regions 701 are set in the original image. In this case, the resolution of the depth image is increased for each attention region. As a result, as shown in Fig. 4(b), point cloud data 702 with a second density is generated for each attention region.

[0039] The process of generating three-dimensional shape data will be further described with reference to the flowchart of Fig. 2. The process shown in Fig. 2 is performed by the control unit 150 executing a program, for example, a three-dimensional shape data generation application program. Note that the acquisition of point cloud data by the distance measuring device 10 and the acquisition of image data by the imaging device 11 do not need to be performed when generating three-dimensional shape data, and data acquired in advance and stored in the non-volatile memory 110 may be used. Here, a case where the original image 310 and point cloud data 320 shown in Fig. 3 are used will be described.

[0040] In S201, image extraction unit 151 (first acquisition means) acquires data of original image 310 stored in image storage unit 112. Then, image extraction unit 151 sets an attention area in original image 310. The attention area is an area for generating high-density point cloud data, and can be set as a main subject area in an image, such as a person. Details regarding setting of the attention area will be described later. Here, it is assumed that the area of ​​the person shown in FIG. 3 is set as the attention area.

[0041] In S202, the image extraction unit 151 extracts data of the attention area set in S201 from the data of the original image 310, and stores it in the system memory 120. The image extraction unit 151 extracts, for example, data of a rectangular area circumscribing the attention area as data of the attention area from the data of the original image 310 stored in the image storage unit 112. Note that the data is not limited to the rectangular area circumscribing the attention area, and data of a range along the contour of the attention area may be extracted, data of a part of the attention area may be extracted, or data of a fixed-size area including the attention area may be output. In addition, an image may be displayed and the range to be extracted may be determined by the user through a user interface device. Although it has been described above, in reality, it is not limited to this. Here, it is assumed that data of the attention area 311 shown in FIG. 3 is extracted.

[0042] In S203, the three-dimensional point cloud extraction unit 152 (second acquisition means) acquires point cloud data 320 corresponding to the imaging range of the original image 310 acquired in S201 from among the point cloud data 320 corresponding to the original image 310 stored in the three-dimensional point cloud storage unit 111. Then, the three-dimensional point cloud extraction unit 152 (conversion means) generates data of a depth image 330 corresponding to the original image 310 acquired in S201. The three-dimensional point cloud extraction unit 152 can generate data of the depth image 330 by projecting the extracted point cloud data onto a plane perpendicular to the optical axis of the imaging device 11 based on the positional relationship between the distance measuring device 10 and the imaging device 11. The three-dimensional point cloud extraction unit 152 stores the generated data of the depth image 330 in the system memory 120.

[0043] In the depth image 330, the value of each pixel constituting the image represents the distance in the depth direction at the pixel position, and is also called a depth map. The depth image 330 can be converted to and from the point cloud data 320. Note that the depth image 330 and the original image 310 have the same imaging range, but the resolution (number of pixels) of the depth image 330 can be lower than that of the original image 310. Note that in this embodiment, the point cloud data 320 is converted to the depth image 330 using the position information of the distance measuring device 10 and the imaging device 11. However, the point cloud data 320 may be converted to the depth image 330 using other known methods such as matching of point cloud features and image features or a calibration method.

[0044] In S204, the three-dimensional point cloud extraction unit 152 extracts data of a region corresponding to the attention region 311 extracted in S202 from the data of the depth image 330 generated in S203 as data of the attention point cloud region 331. The three-dimensional point cloud extraction unit 152 stores the data of the attention point cloud region 331 in the system memory 120.

[0045] In S205, the high density point cloud generation unit 153 uses the data of the attention area 311 extracted in S202 and the data of the attention point cloud area 331 extracted in S204 to increase the density of the point cloud data included in the attention point cloud area 331. This corresponds to increasing (upscaling) the number of pixels included in the attention point cloud area 331, which is a partial area of ​​the depth image 330, and generating the depth image 340. This makes it possible to increase the density of the point cloud data in the part of the original image 310 that corresponds to the attention area 311.

[0046] The method of increasing the density of point cloud data using the corresponding two-dimensional image data can be a known method using a convolutional neural network (CNN), also called Depth Completion. For details, see the following literature, for example. Xinjing Cheng et al., “Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network”, ECCV 2018, pp.108-125, September 8, 2018 Jinsun Park, et al., “Non-Local Spatial Propagation Network for Depth Completion”, ECCV 2020, pp.120-136, July 20, 2020

[0047] Therefore, the high density point cloud generator 153 can increase the density of the point cloud data using a trained CNN using known techniques such as those disclosed in these documents. The high density point cloud generator 153 stores the generated depth image 340 in the system memory 120.

[0048] In S206, the high density point cloud generating unit 153 (inverse conversion means) converts (inversely projects) the composite depth image obtained by combining the depth image 330 and the depth image 340 into point cloud data. This conversion can be executed as the inverse conversion of the conversion performed by the 3D point cloud extracting unit 152 in S203. As a result, point cloud data for the entire original image 310, in which the density of the point cloud data in the attention area 311 is high, is obtained. This corresponds to the point cloud data obtained by combining the point cloud data 350 and the point cloud data 320 in FIG. 3.

[0049] Instead of converting the composite depth image into point cloud data, in S206, only the depth image 340 may be converted into point cloud data 350, and in S207, the point cloud data 350 may be composited with the point cloud data 320 by the three-dimensional point cloud replacement unit 154 (composite means).

[0050] (Area of ​​interest setting process) Details of the attention area setting process in S201 will be described with reference to the flowchart of FIG. 5. Here, a case will be described where an area of ​​a human subject, which is an example of a specific subject, is set as the attention area. However, the type of subject set as the attention area is not limited to a person. For example, when an imaging mode for imaging a specific type of subject is set when the original image 310 is captured, an area of ​​a subject of a type according to the settings at the time of imaging may be set as the attention area. In addition, the user may be allowed to set the type of subject. In addition, the attention area may be set according to conditions different from the type of subject, such as setting a subject area including a focus detection area at the time of imaging or an area having a specific color as the attention area.

[0051] In S401, the image extraction unit 151 applies human subject region detection processing to the original image 310. The human subject region can be detected by any known method, such as a method using template matching or a method using a trained neural network. When two or more human subject regions are detected, the image extraction unit 151 selects one or more regions based on a predetermined condition, such as a region whose detection position is closest to the image center or the largest region.

[0052] In S402, image extraction unit 151 determines whether or not a human subject region was detected in S401, and if it is determined that a human subject region was detected, executes S403, and if it is not determined that a human subject region was detected, executes S404.

[0053] In S403, the image extraction unit 151 sets the human subject region selected in S401 as a region of interest, and ends the process.

[0054] In S404, the image extraction unit 151 detects an in-focus area in the original image 310. The in-focus area can be detected by a known method such as applying a wavelet transform to the original image 310. If information on the position of the focus detection area at the time of image capture is known, for example, by being recorded in the data file of the original image 310, the in-focus area may be detected based on the position of the focus detection area.

[0055] In S405, the image extraction unit 151 obtains the ratio of the in-focus area detected in S404 to the entire original image 310, and determines whether the ratio is equal to or smaller than a predetermined threshold. If the ratio is equal to or smaller than the threshold, the image extraction unit 151 executes S406, and if the ratio is not equal to or smaller than the threshold, the image extraction unit 151 executes S407.

[0056] In S406, the image extraction unit 151 sets the in-focus area detected in S404 as an area of ​​interest, and ends the process.

[0057] In S407, the image extraction unit 151 applies edge detection processing to the data of the original image 310. The edge detection processing can be executed using any known method, such as a magnitude determination using a Histogram of Oriented Gradient (HoG) feature amount.

[0058] In S408, the image extraction unit 151 sets the area surrounded by the contour detected in S407 as the area of ​​interest, and ends the process.

[0059] In this example, the human subject region, the in-focus region, or the region surrounded by the detected outline is set as the region of interest. However, the region of interest may be set from the human subject region, the in-focus region, or the region surrounded by the outline. In addition, a feature region may be set by combining multiple conditions, such as setting the region of interest that is most in-focus among the human subject regions.

[0060] In this embodiment, among the point cloud data measured at a first density or resolution, the point cloud data included in the region of interest is increased to a second density higher than the first density, thereby generating more detailed 3D shape data for the region of interest than other regions. Therefore, it is not necessary to measure high-density point cloud data from the beginning. In addition, since the density of the point cloud data can be increased without measurement, it is not necessary to use a large and expensive sensor capable of high-density measurement. In addition, the volume of point cloud data obtained by measurement can be suppressed.

[0061] On the other hand, detailed point cloud data can be obtained for the region of interest. Therefore, by setting the region requiring detailed shape data as the region of interest, it is possible to efficiently generate useful 3D shape data while suppressing the processing load.

[0062] ●(Second embodiment) Next, a second embodiment of the present invention will be described. This embodiment differs from the first embodiment in that a plurality of regions of interest are set. The following will focus on the differences from the first embodiment. In this embodiment, after a region of interest is set, a high-resolution image of the region of interest is obtained and used to increase the density of point cloud data.

[0063] Fig. 6 is a block diagram showing an example of a functional configuration of a three-dimensional information processing system 1' according to the second embodiment. The three-dimensional information processing system 1 includes a three-dimensional information processing device 100', a distance measuring device 10, and an imaging device 11'. The same reference numerals are used for configurations common to the first embodiment, and duplicated explanations will be omitted. Although not shown in Fig. 6, the three-dimensional information processing device 100' may have a three-dimensional point group replacement unit 154 similar to the first embodiment.

[0064] The imaging device 11' has an imaging magnification setting section 801. The imaging magnification setting section 801 may be, for example, a zoom lens. The angle of view (focal length) of the zoom lens can be controlled by the control section 150. After the magnification at the time of imaging is set based on control or a setting value on the hardware, the imaging device 11 captures an image. Note that, when an image with an increased resolution of the region of interest is obtained without imaging, the imaging device 11' does not need to have the imaging magnification setting section 801.

[0065] Here, it is assumed that the original image is captured at a first magnification, which is lower than the maximum magnification that can be set in the imaging device 11'.

[0066] Fig. 7 is a flow chart of the three-dimensional shape data generation process in this embodiment. In Fig. 7, the same reference numerals as in Fig. 2 are used for steps in which the same processes as in the first embodiment are executed, and the description thereof will be omitted.

[0067] In S901, the control unit 150 acquires an image with a higher resolution than the original image for the attention area set by the image extraction unit 151 in S202. The control unit 150 controls, for example, the imaging magnification setting unit 801 (for example, a zoom lens) of the imaging device 11' to perform imaging at a second magnification higher than that at the time of capturing the original image. Then, the control unit 150 can acquire an image with a higher resolution than the original image for the attention area by cutting out the attention area from the image data stored in the image storage unit 112. Note that the control unit 150 can control the imaging magnification setting unit 801 to capture the attention area at the maximum magnification that can be realized without changing the optical axis direction of the imaging device 11' and within a range in which the attention area does not exceed the imaging range. Therefore, the magnification of the attention area at the time of re-imaging may differ depending on the size and position of the attention area.

[0068] Alternatively, the control unit 150 may obtain an image of the area of ​​interest having a higher resolution than the original image by increasing the resolution of the original image by known image processing. In this case, imaging by the imaging device 11' is not necessary. There is also no restriction on the magnification as in imaging. When data of the area of ​​interest with increased resolution is generated by using image processing, the control unit 150 stores the generated image data in the system memory 120.

[0069] In S902, the three-dimensional point cloud extraction unit 152 converts the point cloud data corresponding to the original image stored in the three-dimensional point cloud storage unit 111 into a depth image in the same manner as in S203. Note that the point cloud data corresponding to the imaging range when re-capturing in S901 may be converted into a depth image. The three-dimensional point cloud extraction unit 152 stores the generated depth image data in the system memory 120.

[0070] In S903, the high density point cloud generating unit 153 uses the data of the area of ​​interest at the second magnification (high resolution) acquired in S901 and the area of ​​interest extracted in S204 to increase the density of the point cloud data included in the area of ​​interest. The image of the area of ​​interest acquired in S901 has a higher resolution (more pixels) than the area of ​​interest in the original image. Therefore, the accuracy of the point cloud data obtained by densification can be improved compared to the case where the data of the area of ​​interest in the original image is used.

[0071] As described above, in this embodiment, the resolution of the attention area used in the densification process of the point cloud data is set higher than the resolution of the attention area in the original image. Therefore, in addition to the effect of the first embodiment, the accuracy of the point cloud data obtained based on the high-resolution depth image can be improved.

[0072] (Other embodiments) The above-described embodiments may be combined in whole or in part as long as no contradiction occurs.

[0073] Furthermore, densification of point cloud data (increasing the resolution of depth images) can be performed using any technique that improves the resolution of images.

[0074] The attention area may be set without using information on the image acquired by the imaging device 11. For example, the attention area may be set based on information different from the image feature amount, such as shape information acquired by an ultrasonic sensor, temperature information acquired by a temperature sensor, or a combination of these. The attention area may be set by using information different from the image feature amount and the image feature amount in combination.

[0075] The operations described as being performed by control unit 150 may be executed by a single piece of hardware, or may be executed by a plurality of pieces of hardware (eg, a plurality of processors or circuits) working together.

[0076] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0077] The disclosure of the present embodiment includes the following three-dimensional information processing device, three-dimensional information processing system, three-dimensional information processing method, and program. (Item 1) A first acquisition means for acquiring three-dimensional shape data representing a shape of a three-dimensional object; A second acquisition means for acquiring image data of an imaging range including the three-dimensional object; A conversion means for generating a depth map corresponding to the imaging range from the three-dimensional shape data; a resolution conversion means for increasing a resolution of a region of the depth map corresponding to a region of interest set for the image; an inverse transformation means for generating three-dimensional shape data based on the depth map having an increased resolution of an area corresponding to the area of ​​interest; A three-dimensional information processing device comprising: (Item 2) The inverse conversion means generates three-dimensional shape data based on data of the area of ​​the depth map where the resolution has been increased, The three-dimensional information processing device further includes a synthesis unit that synthesizes the three-dimensional shape data acquired by the first acquisition unit and the three-dimensional shape data generated by the inverse conversion unit. 2. The three-dimensional information processing device according to item 1, (Item 3) 3. The three-dimensional information processing device according to item 1 or 2, wherein the resolution conversion means uses data of the region of interest when increasing the resolution. (Item 4) 4. The three-dimensional information processing device according to item 3, wherein the resolution of the region of interest used when increasing the resolution is higher than the resolution of the region of interest in the image acquired by the second acquisition means. (Item 5) 5. The three-dimensional information processing device according to item 4, characterized in that data of the area of ​​interest used when increasing the resolution is obtained by capturing an image including the area of ​​interest at an imaging magnification higher than the imaging magnification of the image captured by the second acquisition means. (Item 6) 5. The three-dimensional information processing device according to item 4, characterized in that the resolution of the area of ​​interest of the image acquired by the second acquisition means is increased by image processing, thereby acquiring data of the area of ​​interest used when increasing the resolution. (Item 7) The image forming apparatus further includes a setting unit for setting the region of interest, The setting means sets an area of ​​a specific subject included in the image as the attention area. 7. The three-dimensional information processing device according to any one of items 1 to 6, (Item 8) 8. The three-dimensional information processing device according to item 7, wherein the setting means sets a focused area or an area surrounded by a contour of the image as the attention area when the area of ​​the specific subject is not detected in the image. (Item 9) 9. The three-dimensional information processing device according to any one of items 1 to 8, wherein the three-dimensional shape data is point cloud data. (Item 10) 10. A three-dimensional information processing device according to any one of items 1 to 9, A measuring device that measures three-dimensional shape data representing the shape of a three-dimensional object; an imaging device for acquiring image data of an imaging range including the three-dimensional object; A three-dimensional information processing system comprising: (Item 11) A three-dimensional information processing method executed by an information processing device, comprising: acquiring three-dimensional shape data representative of a shape of a three-dimensional object; acquiring image data of an imaging range including the three-dimensional object; generating a depth map corresponding to the imaging range from the three-dimensional shape data; Increasing the resolution of a region of the depth map that corresponds to a region of interest established for the image; generating three-dimensional shape data based on the depth map with increased resolution of an area corresponding to the area of ​​interest; A three-dimensional information processing method comprising the steps of: (Item 12) A program for causing a computer to function as each of the means possessed by the three-dimensional information processing device according to any one of items 1 to 9.

[0078] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0079] 10: distance measuring device, 11: imaging device, 100: three-dimensional information processing device, 110: non-volatile memory, 120: system memory, 150: control unit

Claims

1. a first acquisition means for acquiring three-dimensional shape data representing a shape of a three-dimensional object; A second acquisition means for acquiring image data of an imaging range including the three-dimensional object; A conversion means for generating a depth map corresponding to the imaging range from the three-dimensional shape data; a resolution conversion means for increasing a resolution of a region of the depth map corresponding to a region of interest set for the image; an inverse transformation means for generating three-dimensional shape data based on the depth map in which the resolution of the area corresponding to the area of ​​interest is increased; A three-dimensional information processing apparatus comprising:

2. The inverse conversion means generates three-dimensional shape data based on data of the area of ​​the depth map where the resolution has been increased, the three-dimensional information processing apparatus further comprises a synthesis means for synthesizing the three-dimensional shape data acquired by the first acquisition means and the three-dimensional shape data generated by the inverse conversion means; 2. The three-dimensional information processing apparatus according to claim 1 .

3. 2. The three-dimensional information processing apparatus according to claim 1, wherein said resolution conversion means uses data of said region of interest when increasing said resolution.

4. 4. The three-dimensional information processing apparatus according to claim 3, wherein the resolution of the region of interest used when increasing the resolution is higher than the resolution of the region of interest in the image acquired by the second acquisition means.

5. The three-dimensional information processing device according to claim 4, characterized in that data of the area of ​​interest used when increasing the resolution is obtained by capturing an image including the area of ​​interest at an imaging magnification higher than the imaging magnification of the image acquired by the second acquisition means.

6. 5. The three-dimensional information processing apparatus according to claim 4, wherein the resolution of the area of ​​interest of the image acquired by the second acquisition means is increased by image processing, thereby acquiring data of the area of ​​interest used when increasing the resolution.

7. The image forming apparatus further includes a setting unit for setting the region of interest, The setting means sets an area of ​​a specific subject included in the image as the attention area.

2. The three-dimensional information processing apparatus according to claim 1 .

8. 8. The three-dimensional information processing apparatus according to claim 7, wherein said setting means sets a focused area or an area surrounded by an outline of said image as said attention area when said specific subject area is not detected in said image.

9. 2. The three-dimensional information processing apparatus according to claim 1, wherein the three-dimensional shape data is point cloud data.

10. A three-dimensional information processing apparatus according to any one of claims 1 to 9, A measuring device that measures three-dimensional shape data representing a shape of a three-dimensional object; an imaging device for acquiring image data of an imaging range including the three-dimensional object; A three-dimensional information processing system comprising:

11. A three-dimensional information processing method executed by an information processing device, comprising: acquiring three-dimensional shape data representing a shape of a three-dimensional object; acquiring image data of an imaging range including the three-dimensional object; generating a depth map corresponding to the imaging range from the three-dimensional shape data; Increasing the resolution of a region of the depth map that corresponds to a region of interest established for the image; generating three-dimensional shape data based on the depth map with increased resolution of an area corresponding to the area of ​​interest; A three-dimensional information processing method comprising the steps of:

12. A program for causing a computer to function as each of the means included in the three-dimensional information processing apparatus according to any one of claims 1 to 9.

Images