Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program that appropriately determine an imaging apparatus to be used as a stereo camera, and correct three-dimensional shape data of an object using images captured by the determined imaging apparatus.SOLUTION: An information processing system having an imaging apparatus group including a plurality of imaging apparatuses, an information processing apparatus, and a control apparatus for controlling each imaging apparatus, the information processing apparatus 100 is configured to: acquire, by an image acquisition unit 201, three-dimensional shape data indicating a three-dimensional shape of an object; determine, by a determination unit 204, an imaging apparatus to be used as a stereo camera from among the plurality of imaging apparatuses based on the three-dimensional shape data; and correct, by a correction unit 206, the three-dimensional shape data using captured images obtained by capturing using the imaging apparatus determined to be used as the stereo camera.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a technique for determining an imaging device to be used as a stereo camera. [Background technology]

[0002] There is a technology for acquiring three-dimensional shape data representing the three-dimensional shape of an object using multiple captured images (hereinafter referred to as "multi-view images") obtained by capturing an object from multiple imaging devices each capturing an image from different directions. The three-dimensional shape data is represented, for example, by a polygon mesh or a point cloud. A widely known method for estimating three-dimensional shape data using multi-view images is, for example, the volume intersection method. However, since the volume intersection method cannot properly restore concave regions of an object, the difference between the three-dimensional shape of a specified object and the actual three-dimensional shape of the object (also referred to as "shape estimation error") may be large. Patent Document 1 discloses a technology for correcting three-dimensional shape data corresponding to an object acquired by the volume intersection method using information indicating the distance to the surface of the object calculated based on captured images obtained by a stereo camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-015863 Summary of the Invention [Problem to be solved by the invention]

[0004] However, depending on the positional relationship between the object and the stereo camera, the stereo camera may not be able to capture an image of a concave region in the object, or the target object may be hidden behind another object.

[0005] Therefore, an object of the present disclosure is to provide a technique for appropriately determining an imaging device to be used as a stereo camera. [Means for solving the problem]

[0006] The information processing device of the present invention includes a shape acquisition means for acquiring three-dimensional shape data indicating the three-dimensional shape of an object, a determination means for determining an imaging device to be used as a stereo camera from among a plurality of imaging devices based on the three-dimensional shape data, and a correction means for correcting the three-dimensional shape data using an image obtained by imaging with the imaging device determined to be used as the stereo camera. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to appropriately determine an imaging device to be used as a stereo camera. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an information processing system according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of a functional configuration of an information processing device according to a first embodiment. [Figure 3] FIG. 4 is a diagram for explaining an example of a determination process of the imaging device according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining an example of a determination process of the imaging device according to the first embodiment. [Figure 5] FIG. 4 is a diagram for explaining an example of a determination process of the imaging device according to the first embodiment. [Figure 6] FIG. 4 is a diagram for explaining an example of a determination process of the imaging device according to the first embodiment. [Figure 7] FIG. 4 is a diagram for explaining an example of a determination process of the imaging device according to the first embodiment. [Figure 8] FIG. 4 is a diagram for explaining an example of a determination process of the imaging device according to the first embodiment. [Figure 9]FIG. 4 is a diagram for explaining an example of a determination process of the imaging device according to the first embodiment. [Figure 10] 10 is a flowchart showing an example of the flow of a determination process of the stereo camera according to the first embodiment. [Figure 11] 4 is a flowchart showing an example of a processing flow of the information processing device according to the first embodiment. [Figure 12] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to a first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings. However, the following embodiments do not necessarily limit the means for solving the problems according to the present disclosure, and not all of the combinations of features described in the following embodiments are necessarily essential to the means for solving the problems according to the present disclosure. In each drawing, the same configuration or element is designated by the same reference numeral, and redundant description will be omitted or simplified.

[0010] [Embodiment 1] 1 is a block diagram showing an example of the configuration of an information processing system 1 according to a first embodiment. The information processing system 1 includes an imaging device group 10 including multiple imaging devices C1 to C100, an information processing device 100, and a control device 11 that controls each imaging device Cn (n is any integer from 1 to 100). Each imaging device Cn is assigned information for identifying each other (hereinafter referred to as a "device ID (identifier)"). In this embodiment, the imaging device group will be described as including 100 imaging devices Cn, and the multiple imaging devices Cn are arranged to surround a predetermined imaging area, but the number and arrangement of the imaging devices Cn are not limited to this.

[0011] Each imaging device Cn is configured as a digital still camera, a digital video camera, or the like, and captured image data obtained by imaging by the imaging device Cn is transmitted to the information processing device 100. The control device 11 is configured as a personal computer, a server device, or the like, and controls the imaging device Cn by transmitting a control signal. The information processing device 100 is configured as a personal computer, a server device, or the like. The configuration and operation of the information processing device 100 will be described below with reference to the drawings.

[0012] FIG. 2 is a block diagram showing an example of the functional configuration of the information processing device 100 according to the first embodiment. The information processing device 100 has, as functional blocks, an image acquisition unit 201, a foreground acquisition unit 202, a shape acquisition unit 203, a determination unit 204, a distance acquisition unit 205, and a correction unit 206. The image acquisition unit 201 acquires captured image data transmitted from each image capture device Cn. The captured image data acquired by the image acquisition unit 201 is transmitted to the foreground acquisition unit 202 and the distance acquisition unit 205. In this embodiment, the image acquisition unit 201 is described as acquiring captured image data transmitted from each image capture device Cn, but the source of the captured image data is not limited to the image capture device Cn. For example, the image acquisition unit 201 may acquire captured image data from an external device such as a storage device that stores captured image data in advance.

[0013] The foreground acquisition unit 202 acquires the data of the plurality of captured images transmitted from the image acquisition unit 201 and extracts a foreground region from each captured image. Here, the foreground region is an image region in the captured image that includes an image of an object (hereinafter simply referred to as an "object") that is in the foreground. For example, the foreground acquisition unit 202 extracts the foreground region from the captured image by comparing the captured image to be processed with a background image that has been generated in advance based on the captured image. There are various comparison methods, and for example, the foreground acquisition unit 202 determines that a pixel is included in the foreground region when the difference in RGB values ​​of corresponding pixels in the background image and the captured image is equal to or greater than a predetermined threshold. The above-mentioned comparison method may be any method that compares images.

[0014] The foreground acquisition unit 202 generates information indicating the acquired foreground region (hereinafter referred to as "foreground information") and transmits this to the shape acquisition unit 203. Here, the foreground information is, for example, image data of the same image size as the captured image, and is binary image data in which the foreground region is represented in white for each pixel and the region other than the foreground region (referred to as "background region") is represented in black. The foreground information may be represented by any method as long as it can indicate the foreground region.

[0015] In this embodiment, as described above, the foreground acquisition unit 202 is described as extracting a foreground region from a captured image, but the foreground acquisition unit 202 may acquire information indicating the foreground region corresponding to each captured image from an external device such as another information processing device. In this case, the external device extracts the foreground region from each captured image and transmits the information indicating the foreground region corresponding to each captured image to the foreground acquisition unit 202, and the foreground acquisition unit 202 receives this information to acquire the foreground region in each captured image. Note that the external device may be included in the imaging device Cn.

[0016] The shape acquisition unit 203 acquires three-dimensional shape data indicating the three-dimensional shape of the object. Specifically, for example, the shape acquisition unit 203 acquires the three-dimensional shape data by generating three-dimensional shape data corresponding to the object using foreground information acquired from the foreground acquisition unit 202 and information indicating the position and orientation of each image capture device Cn that is stored in advance or acquired in advance. Here, the information indicating the position and orientation of the image capture device Cn includes, for example, external parameters of the image capture device Cn, internal parameters of the lens of the image capture device Cn, such as the focal length, angle of view, and aperture value, and distortion parameters of the lens. Hereinafter, these external parameters, internal parameters, and distortion parameters will be collectively referred to as camera parameters. The three-dimensional shape data acquired by the shape acquisition unit 203 is transmitted to the determination unit 204 and the correction unit 206.

[0017] For example, the shape acquisition unit 203 generates three-dimensional shape data by estimating the shape of an object using a volume intersection method (shape from silhouette). The three-dimensional shape data is represented, for example, by a point cloud representing the shape of the object. Furthermore, the shape acquisition unit 203 processes each point included in the point cloud using a general three-dimensional labeling method based on the presence or absence of adjacent points, thereby dividing the point cloud information into point cloud information for each object. Each point included in the point cloud is assigned an object ID, which is the result of labeling using the three-dimensional labeling method. By specifying the object ID, point cloud information for the object corresponding to the object ID can be obtained. Note that the method for generating three-dimensional shape data is not limited to the above-mentioned method. Furthermore, the representation method for three-dimensional shape data is not limited to a point cloud, and may be, for example, a polygon mesh or voxel.

[0018] In this embodiment, as described above, the shape acquisition unit 203 is described as generating three-dimensional shape data using foreground information and camera parameters, but the shape acquisition unit 203 may acquire the three-dimensional shape data from an external device such as another information processing device. In this case, the external device generates three-dimensional shape data using the foreground information and camera parameters and transmits it to the foreground acquisition unit 202, and the shape acquisition unit 203 acquires the three-dimensional shape data by receiving it. The external device may be included in the control device 11, for example.

[0019] Based on the three-dimensional shape data acquired from the shape acquisition unit 203, the determination unit 204 determines, from among the image capture devices C1 to C100, an image capture device Cn to be used as a stereo camera in order to obtain distance information for correcting the three-dimensional shape data. The method by which the determination unit 204 determines the image capture device Cn to be used as a stereo camera will be described later with reference to FIGS. 3 to 9. The determination unit 204 transmits information related to the determined image capture device Cn to be used as a stereo camera (hereinafter referred to as "stereo camera information") to the distance acquisition unit 205. The determination process by the determination unit 204 to determine the image capture device Cn to be used as a stereo camera is performed for each piece of three-dimensional shape data to be corrected, and the image capture device Cn to be used as a stereo camera to obtain distance information for correcting each piece of three-dimensional shape data is determined for each piece of three-dimensional shape data.

[0020] The stereo camera information is, for example, information in which the object ID of the three-dimensional shape data and the device ID of the image capture device Cn used as the stereo camera are associated with each other. The three-dimensional shape data for each object ID is associated with the number of pairs of stereo cameras and the device ID corresponding to each of the stereo cameras. If there is no stereo camera that can acquire distance information suitable for correcting the three-dimensional shape data, the determination unit 204 may transmit, for example, information indicating that no stereo camera information exists (hereinafter referred to as "information without corresponding pair") to the distance acquisition unit 205.

[0021] 3 to 9 are diagrams illustrating an example of a process performed by the determination unit 204 according to the first embodiment to determine an image capture device Cn to be used as a stereo camera. Note that the same reference numerals are used in FIGS. 3 to 9 for the same configurations or elements. FIG. 3 illustrates an example of the arrangement of the image capture devices C1 to C100 and the relative positions of the image capture devices C1 to C100 and the image capture area 300 according to the first embodiment. Hereinafter, as an example, the image capture devices C1 to C50 are described as being arranged at a higher position than the image capture devices C51 to C100. Note that the number and arrangement of the image capture devices Cn are not limited to those illustrated in FIG. 3. For ease of explanation, the present embodiment will be described assuming that the image capture area 300 is rectangular and the determination unit 204 determines the stereo camera in four directions, i.e., imaging directions 301 to 304. First, the following three processes (1-1), (1-2), and (1-3) will be described as an example of a process performed by the determination unit 204 to determine an image capture device Cn to be used as a stereo camera.

[0022] (1-1) Stereo camera decision processing 4 and 5, the stereo camera determination process, which is the basis of this embodiment, will be described. First, the determination unit 204 selects, from among the image capture devices C1 to C100, only the image capture device Cn that captures the entire object corresponding to the three-dimensional shape data as a candidate image capture device Cn to be used as a stereo camera, and excludes the other image capture devices Cn. For example, the determination unit 204 determines whether the image capture device Cn captures the entire object by three-dimensionally projecting each point within a bounding box of the three-dimensional shape corresponding to the object onto the captured image using the camera parameters of the image capture device Cn. Here, the determination unit 204 selects, as a candidate image capture device Cn to be used as a stereo camera, an image capture device Cn in which all points included in the point cloud corresponding to the three-dimensional shape 401 are projected onto the captured image, i.e., which captures the entire object corresponding to the three-dimensional shape data.

[0023] Fig. 4 shows a three-dimensional shape 401 and the image capture devices Cn selected in the above-described selection process. Note that in Fig. 4, as in Fig. 3, the image capture devices Cn shown in black are those selected as candidates for the image capture devices Cn to be used as a stereo camera, and the image capture devices Cn shown in gray are those excluded from the candidates for the image capture devices Cn to be used as a stereo camera.

[0024] Next, the determination unit 204 narrows down and selects (hereinafter referred to as "narrowed selection") the image capture devices Cn selected as candidates to be used as a stereo camera, based on the following conditions: Specifically, the determination unit 204 selects an image capture device Cn such that the angle formed between a line that is perpendicular to each side of each imaging area and passes through the position of an object corresponding to the three-dimensional shape data and a line that passes through the position and the image capture device Cn is close to a predetermined angle. Here, the predetermined angle is, for example, 30°, and this angle can be determined based on the minimum angle of the angle of view of each image capture device Cn so that the object fits within the angle of view.

[0025] FIG. 5 shows an example of an image capture device Cn selected by the determination unit 204 through the above-described narrowing-down selection process when an object corresponding to the three-dimensional shape 401 is present in the image capture area 300. In FIG. 5, the image capture device Cn surrounded by a solid ellipse is the image capture device Cn selected by the determination unit 204 through the above-described narrowing-down selection process. Furthermore, a line indicated by a dashed dotted line (hereinafter referred to as a "reference line") is a line that intersects each side of the image capture area 300 at a right angle and passes through a position in real space corresponding to the position of the three-dimensional shape 401. Furthermore, a line indicated by a broken line is a line (hereinafter referred to as a "stereo line") that forms an angle of 30° with the reference line and passes through a position in real space corresponding to the position of the three-dimensional shape 401. As shown in FIG. 5, for example, the determination unit 204 narrows down and selects (narrows down and selects) the image capture device Cn surrounded by an ellipse that is located closest to the stereo line from among the image capture devices Cn indicated in black and selected as candidates through the above-described selection process.

[0026] Next, the determination unit 204 determines, from the imaging devices Cn selected by the above-described narrowing-down selection process, for example, two imaging devices Cn selected on the left and right of a vertical plane passing through the reference line as stereo cameras. Specifically, in the example shown in Fig. 5, four sets of stereo cameras are determined: a stereo camera made up of imaging devices C56 and C58, a stereo camera made up of imaging devices C67 and C70, a stereo camera made up of imaging devices C79 and C34, and a stereo camera made up of imaging devices C91 and C96.

[0027] In the above-described narrowing-down selection process, the angle between the reference line and the stereo line is described as 30°. However, 30° is an example of an ideal angle, and the angle is not limited to 30°. For example, the optimal angle may be determined based on the installation conditions of the image capture device Cn. Furthermore, in the above-described narrowing-down selection process, in addition to the angle between the reference line and the stereo line, the distance between the object corresponding to the three-dimensional shape data and the image capture device Cn may also be added as a condition.

[0028] Through the processing up to this point, the image capture devices Cn to be used as each of the four sets of stereo cameras described above are determined for the three-dimensional shape 401. In this embodiment, the upper limit number of sets of stereo cameras determined by the determination unit 204 is four. In this case, the upper limit number of sets of image capture devices Cn to be used as stereo cameras is eight. The upper limit number of sets of stereo cameras may differ depending on the information processing system 1. For example, the upper limit number of sets of stereo cameras may be logically determined based on the imaging target of the information processing system 1, the required accuracy, the processing capacity of each device included in the information processing system 1, the total number of image capture devices Cn, etc. For example, the higher the processing capacity, the more image capture devices Cn can be used to perform stereo processing, which can improve the accuracy of the three-dimensional shape.

[0029] (1-2) Stereo camera decision process when multiple 3D shape data exist With reference to FIG. 6, the stereo camera determination process when multiple pieces of three-dimensional shape data exist will be described. When multiple pieces of three-dimensional shape data exist, occlusion between the three-dimensional shape data must be taken into consideration. When viewed from a certain image capture device Cn, an object corresponding to the target three-dimensional shape data (hereinafter referred to as a "target object") may be hidden behind an object corresponding to another piece of three-dimensional shape data (hereinafter referred to as an "other object"). In such a case, for example, first, the determination unit 204 excludes, from among the image capture devices Cn to be used as the stereo camera determined in (1-1), an image capture device Cn in which the target object is hidden behind another object as viewed from the image capture device Cn. Next, the determination unit 204 identifies an image capture device Cn to be used as a replacement for the excluded image capture device Cn, and adds the identified image capture device Cn to the image capture devices Cn to be used as the stereo camera.

[0030] 6 shows an example of an image capture device Cn to be used as a stereo camera, determined by the determination unit 204 when a target object corresponding to the three-dimensional shape 401 and another object corresponding to the three-dimensional shape 601 are present in the image capture area 300. First, the determination unit 204 excludes, from the candidates for the image capture device Cn to be used as a stereo camera, image capture devices Cn in which the target object is hidden by another object, from among the image capture devices Cn selected as candidates by the above-described selection process and indicated in black in FIG. 4. Specifically, the determination unit 204 excludes image capture devices C34, C36, C39, C84, C86, and C89 indicated in dark gray in FIG. 6 from the candidates for the image capture device Cn to be used as a stereo camera.

[0031] Next, the determination unit 204 performs the above-described narrowing-down selection process on the remaining image capture devices Cn as candidates for use as the stereo camera, and determines the image capture devices Cn surrounded by the solid-line ellipse in Fig. 6 as the image capture devices Cn to be used as the stereo camera. Specifically, in the example shown in Fig. 6, four sets of stereo cameras are determined: a stereo camera made up of image capture devices C56 and C58, a stereo camera made up of image capture devices C67 and C70, a stereo camera made up of image capture devices C79 and C82, and a stereo camera made up of image capture devices C91 and C96. That is, in the example shown in Fig. 6, the image capture device C34 determined to be used as the stereo camera of the image capture device C79 in the process described in (1-1) is excluded, and therefore the image capture device C82 is determined as the stereo camera of the image capture device C79.

[0032] (1-3) Stereo camera determination process when there are no candidates for the imaging device Cn to be used as the stereo camera 7, a stereo camera determination process in determination unit 204 when there is no candidate image capture device Cn suitable for use as a stereo camera will be described. When there is no image capture device suitable for use as a stereo camera for the three-dimensional shape data corresponding to the target object, determination unit 204 transmits information indicating that no stereo camera information exists (no match information) to distance acquisition unit 205.

[0033] FIG. 7 illustrates an example in which the determining unit 204 transmits no-match information to the distance obtaining unit 205. FIG. 7 illustrates a state in which, in addition to the target object corresponding to the three-dimensional shape 401, other objects corresponding to each of the three-dimensional shapes 712 to 714 exist in the imaging area 300. In the state illustrated in FIG. 7, the target object corresponding to the three-dimensional shape 401 is hidden behind the other objects corresponding to the three-dimensional shapes 712 to 714 when viewed from the imaging device Cn arranged in the imaging direction 302. Therefore, the determining unit 204 excludes, from the imaging devices Cn candidates to be used as a stereo camera, imaging devices Cn in which the target object is hidden by the other objects, among the imaging devices Cn selected as candidates by the above-described selection process and indicated in black in FIG. 4. Specifically, in the example illustrated in FIG. 7, imaging devices C25 to C28 and C20, and imaging devices C65 to C68 and C70 are excluded from the imaging devices Cn candidates to be used as a stereo camera.

[0034] Therefore, among the remaining image capture devices Cn as candidates for use as a stereo camera, the image capture devices C59 and C75 have the smallest angle formed between the reference line corresponding to the image capture direction 302 and a line passing through the position of the target object and the position of the image capture device Cn. Hereinafter, the image capture device Cn that can be selected as a stereo camera must have an angle formed between the reference line, the line passing through the position of the target object, and the position of the image capture device Cn within a predetermined angle 700 or 701, such as 45°. Hereinafter, this predetermined angle will be referred to as the "threshold angle." Both the angle 702 formed between the reference line, the line passing through the position of the target object, and the position of the image capture device C59, and the angle 703 formed between the reference line, the line passing through the position of the target object, and the position of the image capture device C75 exceed the threshold angle. Therefore, the determination unit 204 determines that there is no suitable image capture device to be used as a stereo camera in the image capture direction 302, and transmits no-match information to the distance acquisition unit 205.

[0035] Through the processing up to this point, the image capturing devices to be used for each of the stereo cameras are determined, or it is determined that there is no image capturing device suitable for use as a stereo camera. In the above description, it has been described that the no-pair information is transmitted when there is no image capturing device suitable for use as a stereo camera for the preset reference line, but the processing in this case is not limited to the above. When there is no image capturing device suitable for use as a stereo camera for the preset reference line, for example, the determination unit 204 may set a new reference line by changing the direction of the reference line and perform the above processing again using the new reference line. In this case, for example, the determination unit 204 may, when setting the new reference line, identify a direction in which no other object exists between the target object and the image capturing device Cn, and set the new reference line based on that direction.

[0036] 10 is a flowchart showing an example of the flow of the stereo camera determination process in the determination unit 204 according to the first embodiment, and is a flowchart showing an example of the flow of the stereo camera determination process described in (1-1) to (1-3) above. In the following description, the symbol "S" means a step (process). First, in S1001, the determination unit 204 selects an arbitrary image capture device Cn from the multiple image capture devices C1 to C100 included in the image capture device group 10.

[0037] Next, in S1002, the determination unit 204 uses the camera parameters of the image capture device Cn selected in S1001 to project each point within the bounding box of the three-dimensional shape 401 onto a captured image obtained by image capture by the image capture device Cn. Next, in S1003, the determination unit 204 determines whether all points after projection in S1002 fit within the captured image. If it is determined in S1003 that some or all of the points after projection do not fit within the captured image, the determination unit 204 executes processing of S1006. If it is determined in S1003 that all points after projection fit within the captured image, the determination unit 204 executes processing of S1004.

[0038] In S1004, the determination unit 204 determines whether or not another object exists in front of the target object as viewed from the imaging device Cn selected in S1001, causing the target object to be hidden behind the other object. If it is determined in S1004 that a part or all of the target object is hidden behind the other object, the determination unit 204 executes the process of S1006. If it is determined in S1004 that the target object is not hidden behind the other object, the determination unit 204 selects the imaging device Cn selected in S1001 as a candidate imaging device to be used as a stereo camera in S1005. After S1005, the determination unit 204 executes the process of S1006.

[0039] In S1006, the determination unit 204 determines whether all of the imaging devices C1 to C100 included in the imaging device group 10 have been selected in S1001. If it is determined in S1006 that at least some of the imaging devices Cn included in the imaging device group 10 have not been selected, the determination unit 204 returns to the processing of S1001 and selects an imaging device Cn that has not yet been selected from the imaging devices Cn included in the imaging device group 10. Thereafter, the determination unit 204 repeats the processing from S1001 to S1006 until it is determined in S1006 that all of the imaging devices Cn have been selected. If it is determined in S1006 that all of the imaging devices Cn have been selected, the determination unit 204 executes the processing of S1007.

[0040] In S1007, the determination unit 204 sets a predetermined number of reference lines. Note that in a later process by the determination unit 204, a number of stereo cameras corresponding to the number of reference lines are determined. The number of reference lines may be determined by the determination unit 204 based on the imaging target of the information processing system 1, etc. The reference lines set by the determination unit 204 are, for example, lines that are perpendicular to each side of the imaging area as described above and that pass through the position of the target object. Next, in S1008, the determination unit 204 selects an arbitrary reference line from the predetermined number of reference lines set in S1007. Next, in S1009, the determination unit 204 sets a stereo line that indicates the optimal direction for the stereo camera with respect to the reference line selected in S1008.

[0041] Next, in S1010, the determination unit 204 determines whether or not an image capture device Cn selected as a candidate for the image capture device Cn to be used as a stereo camera in S1005 is present at a position that is within the threshold angle with respect to the reference line selected in S1008. If it is determined that an image capture device Cn exists in S1010, in S1011, the determination unit 204 determines the image capture device Cn that is closest to the stereo line set in S1009 from among the image capture devices Cn selected as candidates in S1005 as the image capture device Cn to be used as a stereo camera. Information (stereo camera information) related to the image capture device Cn to be used as a stereo camera determined in S1011 is transmitted to the distance acquisition unit 205. If it is determined that an image capture device Cn does not exist in S1010, the determination unit 204 determines that no suitable image capture device Cn to be used as a stereo camera exists and transmits no-pair information to the distance acquisition unit 205 in S1012.

[0042] After S1011 or S1012, in S1013, the determination unit 204 determines whether or not all of the reference lines set in S1007 have been selected in S1008. If it is determined in S1013 that at least some of the reference lines have not been selected, the determination unit 204 repeatedly executes the processes from S1008 to S1013 as appropriate until it is determined in S1013 that all of the reference lines have been selected. If it is determined in S1013 that all of the reference lines have been selected, the determination unit 204 ends the process of the flowchart shown in FIG.

[0043] According to the above-described stereo camera determination process, it is possible to appropriately determine the imaging device to be used as the stereo camera. The determination unit 204 may execute the stereo camera determination process shown in (2-1) or (2-2) below instead of or in addition to the above-described (1-1), (1-2), and (1-3).

[0044] (2-1) Stereo camera decision process that prioritizes a specific imaging direction The determination unit 204 may determine the image capture device Cn to be used as the stereo camera based on a priority image capture direction set in advance by a user instruction, giving priority to the image capture direction. For example, when image capture direction 301 is set as the priority image capture direction, in the example shown in FIG. 5, the determination unit 204 determines the pair of image capture devices C56 and C58 as the stereo camera with the highest priority. Furthermore, when there are multiple pieces of three-dimensional shape data to be processed, the determination unit 204 may determine the image capture device Cn to be used as the stereo camera based on the positions of objects corresponding to each piece of three-dimensional shape data and the image capture direction set by a user instruction. The determination process shown in (2-1) is performed, for example, by setting a reference line parallel to the specific image capture direction set by the user in the reference line setting process of S1007 in the flowchart shown in FIG. 10.

[0045] (2-2) Stereo camera determination process based on the pose of 3D shape data In (2-1), a method for determining a preferred direction when determining an image capture device Cn to be used as a stereo camera based on an image capture direction set by a user instruction was described. However, the preferred direction may also be determined based on the orientation of the object to be processed. When determining an image capture device Cn to be used as a stereo camera, if the preferred direction is determined based on the orientation of the object to be processed, for example, the determination unit 204 performs the following process. First, the determination unit 204 analyzes three-dimensional shape data and acquires the facial orientation of the object corresponding to the three-dimensional shape data. Next, the determination unit 204 sets a line extending from the position of the object's face toward the outside of the image capture area 300 in the direction of the acquired facial orientation as a reference line. Next, the determination unit 204 sets a stereo line that forms an angle with the set reference line of, for example, 30°, and determines the image capture device Cn closest to the set stereo line as the image capture device to be used as the stereo camera.

[0046] Fig. 8 shows an example of a facial orientation 802 of an object corresponding to a three-dimensional shape 801 to be processed, and an image capture device Cn to be used as a stereo camera in this case. Note that in Fig. 8, the straight line indicated by the dashed dotted line indicates a reference line set based on the facial orientation 802 of the object corresponding to the three-dimensional shape 801, and the straight line indicated by the broken line indicates a stereo line set based on the set reference line. In the example shown in Fig. 8, for example, the determination unit 204 determines image capture devices C9 and C66, which are located closest to the set stereo line, as the image capture devices to be used as the stereo camera.

[0047] Furthermore, when determining the image capture device Cn to be used as a stereo camera based on the posture of the object, the determination unit 204 may perform the following process. For example, the determination unit 204 analyzes three-dimensional shape data, and when the object corresponding to the three-dimensional shape data is in a low posture, such as bending forward, the determination unit 204 prioritizes an image capture device Cn located in a low position to determine the image capture device Cn to be used as a stereo camera. Similarly, when the line of sight of the object is low, the determination unit 204 prioritizes an image capture device Cn located in a low position to determine the image capture device Cn to be used as a stereo camera. Conversely, when the line of sight of the object is high, such as when the object is facing upward, the determination unit 204 prioritizes an image capture device Cn located in a high position to determine the image capture device Cn to be used as a stereo camera.

[0048] FIG. 9 shows an example of an image capture device Cn to be used as a stereo camera, which is determined when an object corresponding to a three-dimensional shape 901 to be processed is in a low posture. In FIG. 9, the dashed line indicates a reference line set based on a facial orientation 902 of the object corresponding to the three-dimensional shape 901, and the dashed line indicates a stereo line set based on the set reference line. As described above, when the image capture devices C1 to C50 are located at a higher height than the image capture devices C51 to C100, for example, the determination unit 204 determines the image capture devices C59 and C66 as the stereo camera. In the determination of the image capture device Cn to be used as a stereo camera based only on the facial orientation of the object shown as an example in FIG. 8, the image capture device C9 located at a higher position was determined as one of the stereo cameras. In contrast, in the example shown in FIG. 9, the image capture device C59 located at a lower position is prioritized and determined as one of the stereo cameras.

[0049] In the present embodiment, the method for setting the reference line shown in (1-1) to (1-3) and the method for setting the reference line shown in (2-1) and (2-2) have been described separately. However, the methods described in (1-1) to (1-3) and (2-1) and (2-2) may be combined as appropriate. Furthermore, in the setting of the reference line shown in (1-1) to (1-3), as an example, the image capture device Cn to be used as a stereo camera is determined in four directions divided in 90-degree increments. However, this is not limited to this. For example, division in 45-degree or 30-degree increments is also possible. Furthermore, in the present embodiment, the stereo line is set so that the angle formed with the reference line is 30 degrees. However, this is not limited to this. For example, the stereo line may be set so that the angle formed with the reference line is 15 degrees or 10 degrees. Note that the reference lines are preferably installed at approximately equal intervals based on the required accuracy.

[0050] In the above description, the determining unit 204 has been described as selecting an image capture device Cn whose angle of view includes the entire object corresponding to the three-dimensional shape data to be processed as a candidate image capture device Cn to be used as a stereo camera. However, this is not limited to this. For example, the determining unit 204 may also select an image capture device Cn whose angle of view includes only a portion of the object as a candidate image capture device Cn to be used as a stereo camera. Specifically, for example, the determining unit 204 may select an image capture device Cn whose angle of view includes at least a predetermined percentage, such as at least 70% of the bounding box of the three-dimensional shape, as a candidate image capture device Cn to be used as a stereo camera. Furthermore, the determining unit 204 may divide the bounding box of the three-dimensional shape and select an image capture device Cn whose angle of view includes some or all of the divided bounding boxes as a candidate image capture device Cn to be used as a stereo camera.

[0051] The distance acquisition unit 205 estimates the distance from the image capture device Cn to the object based on the stereo camera information received from the determination unit 204, and acquires distance information indicating the distance as an estimation result. The distance information acquired by the distance acquisition unit 205 is transmitted to the correction unit 206. Specifically, for example, first, the distance acquisition unit 205 identifies data of a captured image obtained by image capture by the image capture device Cn indicated by the stereo camera information from data of a plurality of captured images received from the image acquisition unit 201. Next, the distance acquisition unit 205 estimates the distance from the image capture device Cn to the object using the identified captured image data and camera parameters for the image capture device Cn indicated by the stereo camera information. It is assumed that the distance acquisition unit 205 stores camera parameters corresponding to each image capture device Cn in advance.

[0052] A common image processing method may be used to estimate the distance to an object included in an image captured by an image capture device Cn used as a stereo camera. Specifically, for example, the distance acquisition unit 205 calculates image correlation such as Sum of Absolute Distance (SAD) and calculates disparity by identifying pixels with high correlation as corresponding points. The focal length of the image capture device Cn used as a stereo camera and the distance (baseline length) between the image capture devices are known from the camera parameters. Therefore, the distance acquisition unit 205 can calculate the distance for each corresponding point. By calculating the distances for all corresponding points, the distance acquisition unit 205 can generate what is known as a distance image in a stereo camera. Note that, when receiving no-match information from the determination unit 204, the distance acquisition unit 205 transmits no-distance information indicating that no distance information exists to the correction unit 206.

[0053] The correction unit 206 corrects the three-dimensional shape data (hereinafter referred to as "initial shape data") received from the shape acquisition unit 203 using the distance information received from the distance acquisition unit 205. The three-dimensional shape data (hereinafter referred to as "corrected shape data") corrected by the correction unit 206 is output to, for example, an external device not shown in FIG. 1. Specifically, for example, the correction unit 206 corrects the initial shape data by performing the following processing.

[0054] First, the correction unit 206 calculates the distance d from the image capture device Cn to each point by projecting, onto each image capture device Cn, points in real space corresponding to each point constituting the initial shape data for each image capture device Cn indicated by the stereo camera information, using the camera parameters. Specifically, for example, the correction unit 206 converts the world coordinate Xw of each point into a coordinate Xc in the image capture coordinate system by multiplying it by an extrinsic matrix Te. Here, the extrinsic matrix Te is a transformation matrix configured based on the extrinsic parameters of the image capture device Cn. If the position of the image capture device Cn is the origin and the optical axis direction of the image capture device Cn is the positive direction of the z axis in the image capture coordinate system, the distance d is calculated as the z coordinate of the coordinate Xc in the image capture coordinate system.

[0055] Next, the correction unit 206 calculates the image coordinate Xi corresponding to the coordinate Xc in the imaging coordinate system, thereby calculating the coordinates of the distance image corresponding to, for example, the left imaging device Cn of the two imaging devices Cn used as the stereo camera. The image coordinate Xi is calculated by multiplying the normalized imaging coordinates, obtained by normalizing the coordinate Xc in the imaging coordinate system by the z-coordinate value, by an internal matrix Ti. Here, the internal matrix Ti is a matrix configured based on the internal parameters of the imaging device Cn. Based on the distance information received from the distance acquisition unit 205, the correction unit 206 acquires the distance ds from the pixel at the image coordinate Xi to the stereo camera configured by the imaging device Cn used as the stereo camera.

[0056] Next, the correction unit 206 compares the distance d with the distance ds to determine whether or not to delete each point from the initial shape data. Since the correction process in the correction unit 206 aims to restore the concave region in the initial shape data, the correction unit 206 determines whether the distance d for each point is shorter than the distance ds, and deletes points determined to be shorter from the initial shape data as being included in the concave region. On the other hand, the correction unit 206 does not delete points determined to be included inside the object as being included in the initial shape data as determined by the correction unit 206 as being equal to or longer than the distance ds. If the correction unit 206 determines not to delete any point included in the initial shape data for all image capture devices Cn indicated by the stereo camera information, the point is not deleted.

[0057] The correction unit 206 can correct the shape data by performing the above-described process of determining whether or not to delete a point (hereinafter referred to as the "deletion determination process") for all arbitrary points. Here, the distance threshold used in the deletion determination process can be changed depending on the size of the imaging area, the size of the object to be imaged, etc. For example, if the imaging area is 8 meters square and the object to be imaged is a natural person, the distance threshold used in the deletion determination process can be set to 50 mm (millimeters). Note that when the correction unit 206 receives no distance information indicating that there is no distance information from the distance acquisition unit 205, the correction unit 206 may output the initial shape data received from the shape acquisition unit 203 as corrected shape data to an external device, etc., without correcting the initial shape data.

[0058] Note that, as described above, in the present embodiment, the information processing device 100 has been described as having the distance acquisition unit 205 and the correction unit 206 as functional components. However, these functional components are not essential for the information processing device 100. Specifically, for example, an external device, such as an information processing device other than the information processing device 100, may have functional components corresponding to the distance acquisition unit 205 and the correction unit 206. In this case, for example, the external device first acquires, from the information processing device 100, initial shape data, stereo camera information, and captured images and camera parameters of the image capturing device Cn indicated by the stereo camera information. Next, the external device calculates the distance from the image capturing device Cn to the object to be processed using the stereo camera information, and the captured images and camera parameters of the image capturing device Cn indicated by the stereo camera information. Next, the external device corrects the initial shape data using the calculated distance from the image capturing device Cn to the object to be processed.

[0059] 11 is a flowchart showing an example of a processing flow of the information processing device 100 according to the first embodiment. The processing of this flowchart is repeatedly executed each time the information processing device 100 receives new captured image data from the multiple image capturing devices C1 to C100 included in the image capturing device group 10. First, in S1101, the image acquiring unit 201 acquires captured image data by receiving the captured image data transmitted from the image capturing devices C1 to C100. The captured image data acquired in S1101 is transmitted to the foreground acquiring unit 202 and the distance acquiring unit 205. Next, in S1102, the foreground acquiring unit 202 acquires the foreground region by extracting, as the foreground region, an image region including an image of an object to be processed from the captured image acquired in S1102. Foreground information indicating the foreground region acquired in S1102 is transmitted to the shape acquiring unit 203.

[0060] Next, in S1103, the shape acquisition unit 203 estimates the three-dimensional shape of the object to be processed based on the foreground region acquired in S1102, thereby acquiring three-dimensional shape data corresponding to the object. The three-dimensional shape data acquired in S1103 is transmitted to the determination unit 204 and the correction unit 206. Next, in S1104, the determination unit 204 determines an image capture device Cn to be used as a stereo camera from among the multiple image capture devices C1 to C100 included in the image capture device group 10, based on the three-dimensional shape data acquired in S1103. Specifically, for example, the determination unit 204 executes the stereo camera determination process shown in the flowchart of FIG. 10 as the process of S1104. Next, in S1105, the determination unit 204 determines whether or not a pair of image capture devices Cn to be used as a stereo camera exists in S1104.

[0061] If it is determined in S1105 that a pair of image capture devices Cn to be used as a stereo camera exists, in S1106, the determination unit 204 transmits stereo camera information indicating the image capture devices Cn determined in S1105 to the distance acquisition unit 205 and the correction unit 206. After S1106, in S1107, the distance acquisition unit 205 acquires the distance from the image capture device Cn to the object by calculating it using the captured image and camera parameters of the image capture device Cn indicated in the stereo camera information transmitted in S1106. The distance information indicating the distance acquired in S1107 is transmitted to the correction unit 206. After S1107, in S1108, the correction unit 206 corrects the three-dimensional shape data (initial shape data) transmitted in S1103 based on the distance information transmitted in S1107. The three-dimensional shape data (corrected shape data) obtained as a result of the correction process in S1108 is transmitted to an external device or the like.

[0062] On the other hand, if it is determined in S1105 that there is no pair of image capturing devices Cn to be used as a stereo camera, in S1109 the determination unit 204 transmits no corresponding pair information to the distance acquisition unit 205. After S1109, in S1110 the distance acquisition unit 205 receives the no corresponding pair information transmitted in S1109 and transmits no distance information to the correction unit 206. After S1110, in S1111 the correction unit 206 receives the no distance information transmitted in S1110 and transmits the initial shape data as corrected shape data to an external device or the like without correcting the three-dimensional shape data (initial shape data) transmitted in S1103. After S1108 or S1111, the information processing device 100 ends the processing of the flowchart shown in FIG. 11.

[0063] 12 is a block diagram showing an example of the hardware configuration of the information processing device 100 according to embodiment 1. The information processing device 100 has a CPU 1201, a ROM 1202, a RAM 1203, an HDD 1204, a communication I / F 1205, an operation device 1206, and a display device 1207. The units that the information processing device 100 has as its hardware configuration are connected to each other via a system bus 1200 so as to be able to communicate with each other.

[0064] The CPU 1201 is a processor such as a central processing unit, and controls the entire information processing device 100 using computer programs and various data stored in the ROM 1202, RAM 1203, HDD 1204, etc. In other words, the CPU 1201 executes computer programs to realize the various units shown as the functional configuration in FIG. 2. The ROM 1202 is a read-only memory that stores various data such as setting data for the information processing device 100 and computer programs such as a boot program. The RAM 1203 is a readable and writable memory that temporarily stores computer programs and various data loaded from the ROM 1202, as well as data obtained from the outside via the communication I / F 1205. The RAM 1203 also functions as a work area used by the CPU 1201 when executing computer programs. In other words, the RAM 1203 can, for example, provide part of its memory area as a frame memory or provide various other memory areas as appropriate.

[0065] The operation device 1206 is configured with a keyboard, a mouse, or the like, and receives operations by a user and inputs various instructions corresponding to the operations to the CPU 1201. The display device 1207 is configured with a liquid crystal display, or the like, and displays processing results, etc., by the CPU 1201. The HDD 1204 is a large-capacity storage device. The HDD 1204 stores an operating system (OS) and computer programs, etc., that cause the CPU 1201 to realize the functions of each unit shown as the functional configuration in FIG. 2. The HDD 1204 may also store various image data, etc., such as data of captured images to be processed, acquired by the information processing device 100. The computer programs and various data stored in the HDD 1204 are loaded into the RAM 1203 as appropriate under control of the CPU 1201, and used for processing by the CPU 1201.

[0066] The communication I / F 1205 is an interface for connecting to a network such as a LAN (Local Area Network) or the Internet, and to an external device such as a projection device or a display device. The information processing device 100 can acquire various information from an external device or output various information to an external device via the communication I / F 1205. The operation of each of the above-mentioned hardware components is controlled by the CPU 1201. Furthermore, some or all of the processing by the CPU 1201 may be performed by a dedicated processing circuit such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0067] According to the information processing device 100 configured as above, it is possible to appropriately determine the imaging device to be used as the stereo camera.

[0068] [Other embodiments] The above-described functions can also be realized by providing a system with a storage medium containing computer program code for implementing the functions described as being possessed by the information processing device 100 in the above-described embodiments, and the system reading and executing the code. In this case, the computer program code itself read from the storage medium implements the above-described functions. Therefore, the storage medium storing the computer program code can be included as part of the technology of the present disclosure. Furthermore, the above-described functions can be realized by an OS or the like running on a computer performing part or all of the processing using the above-described hardware configuration based on instructions in the computer program code. Furthermore, the above-described functions can be realized in the following manner. For example, first, the computer program code read from the storage medium is written to memory provided in a function expansion card inserted into a computer or a function expansion unit connected to a computer. Next, a processor such as a CPU or a dedicated processing circuit provided in the function expansion card or function expansion unit executes part or all of the processing based on instructions in the code. This may realize the above-described functions according to the present embodiment. In this case, the computer program code for implementing the above-described functions is stored in the storage medium.

[0069] The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (such as an ASIC) that realizes one or more functions.

[0070] It should be noted that within the scope of the present disclosure, the embodiments may be freely combined, any component of the embodiments may be modified, or any component of the embodiments may be omitted.

[0071] [Configuration of the present disclosure] The present disclosure includes the following configurations, methods, and programs.

[0072] <Configuration 1> a shape acquisition means for acquiring three-dimensional shape data indicating a three-dimensional shape of an object; a determination means for determining an image capturing device to be used as a stereo camera from among a plurality of image capturing devices based on the three-dimensional shape data; a correction means for correcting the three-dimensional shape data using captured images obtained by imaging using the imaging device determined to be used as the stereo camera; An information processing device comprising:

[0073] <Configuration 2> the determining means determines an imaging device to be used as the stereo camera based on a position of the three-dimensional shape data; 2. The information processing device according to configuration 1,

[0074] <Configuration 3> the determining means identifies a facial orientation of the object corresponding to the three-dimensional shape data based on the three-dimensional shape data, and determines an imaging device to be used as the stereo camera based on the facial orientation of the object; 3. The information processing device according to configuration 1 or 2, characterized in that:

[0075] <Configuration 4> the determining means identifies a posture of the object corresponding to the three-dimensional shape data based on the three-dimensional shape data, and determines an imaging device to be used as the stereo camera based on the posture of the object; 4. The information processing device according to any one of configurations 1 to 3, characterized in that:

[0076] <Configuration 5> the determining means determines an imaging device that captures an image of at least a part of the object as an imaging device to be used as the stereo camera, from among the plurality of imaging devices; 5. The information processing device according to any one of configurations 1 to 4, characterized in that:

[0077] <Configuration 6> the determination means determines, as the imaging devices to be used as the stereo camera, two imaging devices among the plurality of imaging devices, such that the size of an angle formed by two straight lines passing through the respective positions of the two imaging devices and the position of the object corresponding to the three-dimensional shape data is close to a predetermined angle; 7. The information processing device according to any one of configurations 1 to 6,

[0078] <Configuration 7> a foreground acquisition means for acquiring foreground information indicating a foreground region corresponding to the image of the object in the image; and the shape acquisition means acquires the three-dimensional shape data by generating the three-dimensional shape data based on the foreground information; 7. The information processing device according to any one of configurations 1 to 6,

[0079] <Configuration 8> acquiring the foreground information by extracting the foreground area from each of a plurality of captured images obtained by capturing images using the plurality of imaging devices; 8. The information processing device according to configuration 7,

[0080] <Configuration 9> a distance acquisition means for acquiring a distance between the determined imaging device and the object corresponding to the three-dimensional shape data, using a captured image obtained by imaging using the determined imaging device; and the correcting means corrects the three-dimensional shape data based on the acquired distance; 9. The information processing device according to any one of configurations 1 to 8,

[0081] <Configuration 10> when it is determined that there is no imaging device to be used as the stereo camera that satisfies a predetermined condition, the determination means outputs no-candidate information indicating that there is no candidate imaging device to be used as the stereo camera; 10. The information processing device according to any one of configurations 1 to 9, characterized in that:

[0082] <Configuration 11> a distance acquisition means for acquiring a distance between the determined imaging device and the object corresponding to the three-dimensional shape data; and the distance acquisition means outputs, when receiving the no-candidate information, no-distance information information indicating that there is no distance information indicating the distance; 11. The information processing device according to configuration 10,

[0083] <Configuration 12> the correction means corrects the three-dimensional shape data based on the acquired distance, and does not correct the three-dimensional shape data when receiving the information that no distance information is available; 12. The information processing device according to configuration 11,

[0084] <Method> a shape acquisition step of acquiring three-dimensional shape data indicating a three-dimensional shape of an object; a determination step of determining an imaging device to be used as a stereo camera from among a plurality of imaging devices based on the three-dimensional shape data; a correction step of correcting the three-dimensional shape data using captured images obtained by capturing images using the imaging device determined to be used as the stereo camera; An information processing method comprising:

[0085] <Program> 13. A program for causing a computer to function as the information processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0086] 100 Information processing device 203 Shape acquisition section 204 Decision Section

Claims

1. a shape acquisition means for acquiring three-dimensional shape data indicating a three-dimensional shape of an object; a determination means for determining an image capturing device to be used as a stereo camera from among a plurality of image capturing devices based on the three-dimensional shape data; a correction means for correcting the three-dimensional shape data using captured images obtained by imaging using the imaging device determined to be used as the stereo camera; An information processing device comprising:

2. the determining means determines an imaging device to be used as the stereo camera based on a position of the three-dimensional shape data; 2. The information processing device according to claim 1,

3. the determining means identifies a facial orientation of the object corresponding to the three-dimensional shape data based on the three-dimensional shape data, and determines an imaging device to be used as the stereo camera based on the facial orientation of the object; 2. The information processing device according to claim 1,

4. the determining means identifies a posture of the object corresponding to the three-dimensional shape data based on the three-dimensional shape data, and determines an imaging device to be used as the stereo camera based on the posture of the object; 2. The information processing device according to claim 1,

5. the determining means determines an imaging device that captures an image of at least a part of the object as an imaging device to be used as the stereo camera, from among the plurality of imaging devices; 2. The information processing device according to claim 1,

6. the determination means determines, as the imaging devices to be used as the stereo camera, two imaging devices among the plurality of imaging devices, such that the size of an angle formed by two straight lines passing through the respective positions of the two imaging devices and the position of the object corresponding to the three-dimensional shape data is close to a predetermined angle; 2. The information processing device according to claim 1,

7. a foreground acquisition means for acquiring foreground information indicating a foreground region corresponding to the image of the object in the image; and the shape acquisition means acquires the three-dimensional shape data by generating the three-dimensional shape data based on the foreground information; 2. The information processing device according to claim 1,

8. acquiring the foreground information by extracting the foreground area from each of a plurality of captured images obtained by capturing images using the plurality of imaging devices; 8. The information processing device according to claim 7,

9. a distance acquisition means for acquiring a distance between the determined imaging device and the object corresponding to the three-dimensional shape data, using a captured image obtained by imaging using the determined imaging device; and the correcting means corrects the three-dimensional shape data based on the acquired distance; 2. The information processing device according to claim 1,

10. when it is determined that there is no imaging device to be used as the stereo camera that satisfies a predetermined condition, the determination means outputs no-candidate information indicating that there is no candidate imaging device to be used as the stereo camera; 2. The information processing device according to claim 1,

11. a distance acquisition means for acquiring a distance between the determined imaging device and the object corresponding to the three-dimensional shape data; and the distance acquisition means outputs, when receiving the no-candidate information, no-distance information information indicating that there is no distance information indicating the distance; The information processing device according to claim 10 ,

12. the correction means corrects the three-dimensional shape data based on the acquired distance, and does not correct the three-dimensional shape data when receiving the information that no distance information is available; The information processing device according to claim 11 .

13. a shape acquisition step of acquiring three-dimensional shape data indicating a three-dimensional shape of an object; a determination step of determining an imaging device to be used as a stereo camera from among a plurality of imaging devices based on the three-dimensional shape data; a correction step of correcting the three-dimensional shape data using captured images obtained by capturing images using the imaging device determined to be used as the stereo camera; An information processing method comprising:

14. A program for causing a computer to function as the information processing device according to any one of claims 1 to 12.

Images