Shape estimation device, processing method, and program

Abstract

To accurately create a shape model while reducing an influence of a shape of a subject.SOLUTION: A shape estimation device estimates a first element group representing a three-dimensional shape of a subject on the basis of picked-up images obtained picking up images of the subject from different directions by using a plurality of imaging apparatuses, and for each of elements included in the first element group, on the basis of a color in the picked-up images corresponding to the element, determines whether to delete the element from the first element group. For an element A of elements constituting a surface of a shape represented by the first element group, when a distance between the element A and an element different from the element A and corresponding to a position where a straight line connecting the element A and the imaging apparatus picking up an image of the element A intersects the surface, does not exceed a predetermined threshold, the shape estimation device does not take the element A as a target of deletion, regardless of the color.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for generating a three-dimensional shape model of a subject using images of the subject captured from multiple directions. [Background technology]

[0002] There is known a technology that uses captured images obtained by capturing an object from different directions using multiple cameras to recreate the object in a virtual space, and outputs an image that can be observed from an arbitrary viewpoint in the virtual space with an arbitrary line-of-sight direction and magnification, etc. Hereinafter, the viewpoint and line-of-sight direction, etc. in the virtual space will be referred to as a virtual viewpoint, and an image observed at the virtual viewpoint will be referred to as a virtual viewpoint image.

[0003] To generate a high-quality virtual viewpoint image, it is important to generate a shape model that represents the three-dimensional shape of the subject with high accuracy. The shape model is represented, for example, by a mesh that represents the three-dimensional shape of the subject using a polyhedron, or by a point cloud that indicates the shape of the subject using a large number of vertices. When a mesh is used, the shape model uses triangles or quadrangles as elements, and when a point cloud is used, the shape model uses points as elements. Patent Document 1 describes a technology for improving the accuracy of a shape model by using multiple captured images captured by multiple cameras that can capture elements included in the shape model without obstructions, and determining whether to delete the element based on the variance value of the color corresponding to the element. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-208759 Summary of the Invention [Problem to be solved by the invention]

[0005] In the method for improving the accuracy of shape models described in Patent Document 1, depending on the shape of the subject, it may not be possible to accurately select a camera that can capture the elements of the shape model without any obstructions, and the accuracy of the shape model may not be sufficiently improved.

[0006] The present invention provides a technique for generating a shape model with high accuracy while reducing the influence of the shape of the subject. [Means for solving the problem]

[0007] A shape estimation device according to one aspect of the present invention comprises: an estimation means for estimating a first group of elements representing a three-dimensional shape of a subject based on captured images obtained by capturing images of the subject from different directions using a plurality of imaging devices; a determination means for determining, for each element included in the first group of elements, whether or not to delete the element from the first group of elements based on the color in the captured image corresponding to the element; and a generation means for generating a second group of elements by deleting the elements determined to be deleted from the first group of elements, wherein the determination means determines that, for element A among the elements constituting the surface of the shape represented by the first group of elements, element A should not be deleted from the first group of elements, regardless of its color, if the distance between element A and an element different from element A that corresponds to a position where a straight line connecting element A and the imaging device that captured element A intersects with the surface does not exceed a predetermined threshold. [Effects of the Invention]

[0008] According to the present invention, it is possible to generate a shape model with high accuracy while reducing the influence of the shape of the subject. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a system configuration according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a coordinate system of camera parameters. [Figure 3]FIG. 2 is a diagram illustrating an example of a hardware configuration of a shape estimation device. [Figure 4] FIG. 10 is a diagram illustrating the visibility of elements of a shape model by a camera. [Figure 5] FIG. 10 is a diagram illustrating an example of the flow of a shape estimation process according to the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of the flow of shape precision improvement processing according to the first embodiment. [Figure 7] FIG. 10 illustrates an example of a system configuration according to a second embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of the flow of a shape estimation process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] (First embodiment) In this embodiment, a first model representing the three-dimensional shape of the subject is generated based on a captured image, and the first model is then refined. In this refinement process, for example, when the general shape of the subject is represented by a point cloud, a process is performed to delete points from the point cloud that should not be used as a three-dimensional shape model of the subject. This deletion determination is performed, for example, based on whether or not the color variance observed by multiple cameras (cameras in which the point is visible) for each point included in the point cloud is large, i.e., whether or not the color as seen from multiple cameras varies. If the variance is large and the color as seen from multiple cameras varies, the point may be determined to be a point that should be deleted. This allows, for example, points in areas where the color appears different depending on the observation angle, such as a recess in the subject, to be deleted, thereby forming a three-dimensional shape model of the subject with higher accuracy. Meanwhile, it is possible that points referenced as the same point on the subject in the refinement process actually correspond to two different points, for example, on an elongated or thin shape included in the subject. In this case, for example, if the color of the front and back surfaces of a thin shape differs significantly, the color variance may be determined to be large, and the point may be deleted. That is, if the distance between two points is small, the two points may be referenced as the same point, and if the color variance is large, the point may be mistakenly deleted even though it should not be deleted. This embodiment provides a technology that prevents such points from being mistakenly deleted based on distance images of the object as seen from each camera.

[0012] (System Configuration) 1 shows an example of the configuration of a system for generating a virtual viewpoint image in this embodiment. This system includes, for example, an imaging device 101 and a shape estimation device 102.

[0013] As shown in FIG. 1, the imaging device 101 includes, for example, a camera 111 and an image processing unit 112 as its functions. Note that these configurations are merely examples, and a configuration different from the configuration shown in FIG. 1 may be used. For example, when a virtual viewpoint image is generated based on an image captured by a scanner, the imaging device 101 does not need to be included in the system. Also, for example, the function of the image processing unit 112 may be included in the shape estimation device 102. In other words, although the imaging device 101 and the camera 111 will be hereinafter referred to as different components, the imaging device 101 and the camera 111 may also refer to the same component.

[0014] Although FIG. 1 shows only one imaging device 101 for simplicity, the present system is configured to provide multiple imaging devices 101, each equipped with a camera 111 that captures an image of a subject from a different direction. For example, the multiple cameras 111 are installed in different positions surrounding the subject, facing in different directions (e.g., facing the subject). For example, when imaging a stadium for soccer, rugby, or the like, several tens to several hundred imaging devices 101 may be installed around the stadium. Note that the cameras 111 do not necessarily need to capture images of the subject from all directions surrounding the subject. For example, if there are limitations on the installation location, the cameras 111 may be arranged so that images are captured only from some directions. Note that, in one example, an imaging device 101 may be used in which one image processing unit 112 is provided for one camera 111, or one image processing unit 112 may be provided for multiple cameras 111. That is, in one example, one or a small number of imaging devices 101 including an image processing unit 112 may be present, and the one or a small number of imaging devices 101 may control a large number of cameras 111 to capture images. Furthermore, at least some of the multiple cameras 111 may be cameras with different angles of view, such as telephoto cameras and wide-angle cameras. The multiple cameras 111 are configured to synchronize with a single piece of real-world time information, for example, by the imaging device 101 accessing a time server to acquire time information. It is assumed that imaging time information is assigned to each frame of an image in a video captured by the multiple cameras 111. It is also assumed that identification information for identifying each of the multiple cameras 111 is assigned to each of the multiple cameras 111.

[0015] The image processing unit 112 performs predetermined image processing on the captured image obtained by capturing an image using the camera 111. The image processing unit 112 can be realized by, for example, a general-purpose computer (processor) executing a program, but may also be realized by, for example, dedicated hardware. The imaging device 101 may have a hardware configuration similar to that of the shape estimation device 102 described below, and the functions of the image processing unit 112 may be realized using that hardware configuration. The image processing unit 112 performs, as the predetermined image processing, a process of generating a foreground image by extracting a foreground portion from the captured image. The foreground image is an image generated by extracting an area in which a subject appears in the captured image (this area is referred to as the "foreground area"). The subject extracted as the foreground area refers to a dynamic subject (moving object) that is moving (whose position or shape may change) when, for example, image capturing is performed continuously over a certain period of time with the same angle of view. Dynamic subjects include, for example, people such as players and referees on a field where a sporting event is being played, or a ball in a ball game. Furthermore, the dynamic subject may include, for example, a singer, musician, performer, or presenter in a concert or entertainment event. The image processing unit 112 supplies the result of the image processing (for example, a foreground image) to the shape estimation device 102.

[0016] The image processing unit 112 may have a function to detect abnormalities in the camera. Abnormality information indicating the presence or absence of an abnormality is sent to the shape estimation device 102. An abnormality in the camera can be detected, for example, by evaluating the values ​​of general sensors such as heat and vibration.

[0017] Furthermore, the image processing unit 112 may retain state information of the camera 111, including, for example, one or more of the camera's position, attitude (orientation, imaging direction), focal length, optical center, distortion, F-number, etc. Hereinafter, the state of the camera 111 will be referred to as "camera parameters." The camera parameters may be controlled by the image processing unit 112. However, without being limited to this, at least one of the camera parameters may be controlled by a component other than the image processing unit 112, such as a camera platform to which the camera 111 is attached, controlling the position and attitude of the camera 111. In other words, the camera parameters may include parameters controlled by another device such as a camera platform. Hereinafter, the camera parameters related to the position and attitude of the camera 111 will be referred to as "external parameters" because they vary due to factors external to the camera 111. On the other hand, the parameters related to the focal length, image center, and distortion of the camera 111 will be referred to as "internal parameters" because they are internally set parameters of the camera 111.

[0018] Here, the coordinate system related to the camera 111 will be described with reference to FIGS. 2(A) and 2(B). FIG. 2(A) shows a state in which cameras 111a to 111d that capture an object 201 are arranged in a three-dimensional space. The positions and orientations of the cameras 111a to 111d are expressed by a single world coordinate system represented by a three-dimensional coordinate origin 202, an Xw axis 203, a Yw axis 204, and a Zw axis 205. That is, the positions at which the cameras 111a to 111d are arranged are represented by three-dimensional coordinates, and the directions in which the cameras 111a to 111d are facing are indicated by three-dimensional vectors. FIG. 2(B) also shows a camera image coordinate system (hereinafter referred to as the image coordinate system) for an image 211 captured by the camera 111a. 2(B), in the image coordinate system, an origin 212 of two-dimensional coordinates, an Xi axis 214, and a Yi axis 215 are set, and the pixel at coordinates (0,0) is defined as pixel 213. Note that the image coordinate systems of the other cameras 111b to 111d are defined in a similar manner.

[0019] As described above, for example, the function of the image processing unit 112 may be included in the shape estimation device 102. In this case, an image captured by the camera 111 is input to the shape estimation device 102, and the processing performed by the image processing unit 112 described above is executed in the shape estimation device 102.

[0020] Next, the shape estimation device 102 will be described. The shape estimation device 102 can be implemented by, for example, a general-purpose computer. As shown in Fig. 3, the shape estimation device 102 includes, for example, a hardware configuration including a CPU 301, a ROM 302, a RAM 303, an auxiliary storage device 304, a display unit 305, an operation unit 306, and a communication interface 307. These pieces of hardware are connected to each other via, for example, a bus 308. CPU is an abbreviation for Central Processing Unit, ROM is an abbreviation for Read Only Memory, and RAM is an abbreviation for Random Access Memory.

[0021] The CPU 301 controls the entire shape estimation device 102 using computer programs and data stored in the ROM 302 and RAM 303, and realizes the functions of the shape estimation device 102 shown in FIG. 1 , as described below. The shape estimation device 102 may have one or more dedicated hardware components different from the CPU 301, and at least some of the functions described below may be realized by the dedicated hardware components. The dedicated hardware components may include, for example, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a digital signal processor (DSP). The ROM 302 stores programs and data that do not require modification. The RAM 303 temporarily stores programs and data supplied from the auxiliary storage device 304, as well as data supplied from an external device via the communication interface 307. The auxiliary storage device 304 is, for example, a hard disk drive, and stores various data such as image data and audio data.

[0022] The display unit 305 is configured with, for example, a liquid crystal display or LEDs, and displays a GUI (Graphical User Interface) or the like for the user to operate the shape estimation device 102. The operation unit 306 is configured with, for example, a keyboard, a mouse, a joystick, a touch panel, and the like, and receives operations by the user, generates various instruction signals in response to the user operations, and provides them to the CPU 301. The CPU 301 operates as a display control unit that controls the display unit 305 and an operation control unit that controls the operation unit 306. The communication interface 307 is an interface used for communication between the shape estimation device 102 and an external device. For example, when the shape estimation device 102 is connected to an external device via a wired connection, a communication cable is connected to the communication interface 307. When the shape estimation device 102 has a function of wirelessly communicating with an external device, the communication interface 307 includes an antenna.

[0023] The shape estimation device 102 may acquire information indicating user operations, etc., from, for example, an external user client device (not shown) via the communication interface 307 and execute processing accordingly. In this case, the operation unit 306 may be omitted. Similarly, the shape estimation device 102 may cause an external device to display various information, in which case the display unit 305 in the shape estimation device 102 may be omitted. Furthermore, the shape estimation device 102 may be configured not to display information or accept user operations, in which case the display unit 305 and operation unit 306 may also be omitted. The imaging device 101 may also have at least a part of the configuration shown in FIG. 3. For example, the imaging device 101 may implement the image processing unit 112 using a CPU 301, a ROM 302, a RAM 303, etc. The image processing unit 112 may also be implemented by dedicated hardware such as an ASIC.

[0024] Returning to FIG. 1 , the functions of the shape estimation device 102 will be described. As described above, these functions are realized, for example, by the CPU 301 executing a computer program stored in the ROM 302 or the RAM 303. As shown in FIG. 1 , the shape estimation device 102 includes, for example, a foreground information acquisition unit 131, a general shape estimation unit 132, a surface shape generation unit 133, a distance image generation unit 134, a visibility determination unit 135, and a shape precision improvement unit 136 as its functions. Note that these configurations are merely examples, and a configuration different from that shown in FIG. 1 may be used as long as it can realize the processing functions described below. In one example, as described above, the image processing unit 112 may be included in the shape estimation device 102.

[0025] The foreground information acquisition unit 131 acquires, from the image processing unit 112, a foreground image in a captured image captured by the camera 111 and camera parameters of the camera 111. The foreground information acquisition unit 131 may be realized, for example, as a function that acquires the foreground image and camera parameters transmitted from the imaging device 101 via the communication interface 307. The acquired information may be stored in the RAM 303 or the auxiliary storage device 304 under the control of the CPU 301. Note that the camera parameters may be acquired synchronously with the acquisition of the captured image, may be acquired in a preparation stage, or may be acquired asynchronously with the acquisition of the captured image as necessary. The foreground information acquisition unit 131 may also acquire camera abnormality information from the imaging device 101. Note that this is just one example, and the foreground information acquisition unit 131 may determine the camera parameters to be used. For example, the foreground information acquisition unit 131 extracts feature points from an image acquired by the camera 111 capturing an image of a camera calibration marker (e.g., a checkerboard) in advance. Then, the foreground information acquisition unit 131 associates the actual position of the marker with the position of the marker on the image. The foreground information acquisition unit 131 may determine camera parameters by projecting corresponding points on the image onto each camera and calibrating each camera while optimizing the camera so that the error between the projected positions of the corresponding points and the positions where the corresponding points should be is minimized. The calibration here may be performed by any existing method. When the foreground information acquisition unit 131 determines the camera parameters, for example, the shape estimation device 102 may be directly connected to the camera 111 via the communication interface 307. The shape estimation device 102 may then directly control the camera 111 using the camera parameters determined by the foreground information acquisition unit 131.

[0026] The general shape estimation unit 132 performs processing to estimate the general shape of the subject based on the foreground image and camera parameters, and generates a general shape model of the subject. The general shape estimation unit 132 can be implemented, for example, as a function in which the CPU 301 executes a computer program stored in the ROM 302 or the like, and performs image processing on the foreground image acquired by the foreground information acquisition unit 131 and expanded in the RAM 303. The general shape model is input to the shape precision improvement processing, i.e., is shape information that is the target of the normal precision improvement processing, and is a model that indicates the approximate shape of the subject. In this embodiment, the shape model is configured by a set of point elements having three-dimensional coordinates (hereinafter referred to as a point cloud). The general shape estimation unit 132 estimates the general shape model of the subject using, for example, a volume intersection method (shape from silhouette). The generated general shape model is stored, for example, in the RAM 303 or the auxiliary storage device 304.

[0027] The surface shape generation unit 133 generates a surface shape from the point cloud constituting the general shape model, excluding points present inside the subject. The surface shape generation unit 133 may be implemented, for example, as a function in which the CPU 301 executes a computer program stored in the ROM 302 or the like to perform processing on the general shape model generated by the general shape estimation unit 132 and loaded into the RAM 303. That is, the surface shape generation unit 133 generates a surface shape from the point cloud constituting the general shape model, leaving only the surface points visible when the subject is observed from the outside and excluding the other points. Whether each point included in the point cloud constituting the general shape model is an internal point can be determined based on whether other points exist in the vicinity of that point. That is, a point in the general shape model that has other points nearby is retained as a surface shape point, while a point that has no other points nearby (or has only a few other points nearby) is determined not to be a surface shape point and is removed. The generated point cloud representing the surface shape is stored in, for example, the RAM 303 or the auxiliary storage device 304.

[0028] The distance image generation unit 134 generates distance images indicating the distance from each camera to the surface shape using the surface shape and camera parameters. The distance image generation unit 134 can be implemented, for example, by the CPU 301 executing a computer program stored in the ROM 302 or the like. For example, the distance image generation unit 134 can be implemented as a function that processes data indicating the surface shape generated by the surface shape generation unit 133 and stored in the RAM 303 using the camera parameters output by the foreground information acquisition unit 131. In this embodiment, the distance image generation unit 134 generates a minimum distance image indicating the minimum distance from the camera to the surface of the subject and a maximum distance image indicating the maximum distance. In one example, the minimum distance image is a distance image of the surface shape of the subject that is visible from the camera, and the maximum distance image is a distance image of the surface shape of the subject that is invisible from the camera. The minimum distance image and the maximum distance image are generated for each of the multiple cameras 111. The generated minimum distance image and maximum distance image are stored, for example, in the RAM 303 or the auxiliary storage device 304.

[0029] The visibility determination unit 135 determines the visibility from each camera for each point in the general shape model. The visibility determination unit 135 can be implemented, for example, by the CPU 301 executing a computer program stored in the ROM 302 or the like. The visibility determination unit 135 determines points observable by the camera 111 from among the points included in the general shape model generated by the general shape estimation unit 133, for example, based on the shortest distance image and the longest distance image generated by the distance image generation unit 134. The distance image generation unit 134 may perform this determination based on camera parameters acquired by the foreground information acquisition unit 131 and stored in the RAM 303 or the like. Here, visibility refers to information indicating whether each point in the shape model is visible from each camera. The visibility determination will be described with reference to the schematic diagram of FIG. 4. In FIG. 4, the camera 111 is assumed to be facing in a direction 401. Furthermore, the surface 402 of the object includes points A 403 and B 404, which are part of the point cloud of the shape model. 4, when point A403 and point B404 are located on an extension of direction 401, the visibility determination unit 135 determines that point A403, which is closer to the camera 111, is a point visible from the camera 111. On the other hand, with respect to point B404, which is farther from the camera 111 than point A403 and is hidden by point A403, the visibility determination unit 135 determines that the point is not visible from the camera 111. For example, the visibility determination unit 135 can store information indicating a group of visible points for each camera 111 as visibility information in the RAM 303 or the auxiliary storage device 304.

[0030] The shape precision improving unit 136 improves the precision of the general shape model and generates a 3D shape model of the subject. The shape precision improving unit 136 can be implemented, for example, by the CPU 301 executing a computer program stored in the ROM 302 or the like. The shape precision improving unit 136 identifies a camera to be used to improve the precision of the general shape model, for example, based on the difference between the longest distance image and the shortest distance image generated by the distance image generation unit 134 and the visibility information generated by the visibility determination unit 135. The shape precision improving unit 136 then uses the identification result to improve the precision of the general shape model estimated by the general shape estimation unit 132. The improved general shape model can be stored, for example, in the RAM 303 or the auxiliary storage device 304. The improved general shape model is then provided as the final 3D shape model of the subject to an external device (for example, a server that renders virtual viewpoint images, a client device, etc.) via the communication interface 307.

[0031] (Processing flow) An example of the flow of processing executed by the shape estimation device 102 will be described with reference to Figure 5. In the following description, each process is realized by the CPU 301 executing a computer program stored in the ROM 302 or RAM 303, but this is not limiting. For example, if dedicated hardware is used, the term "CPU 301" used in the following description may be replaced with "dedicated hardware."

[0032] In S501, the CPU 301 controls the communication interface 307 to communicate with the imaging device 101 and acquires camera parameters from the imaging device 101. In one example, the CPU 301 generates a request signal requesting the imaging device 101 to provide the camera parameters, transmits the request signal to the imaging device 101 via the communication interface 307, and acquires the camera parameters in response to the request signal. Note that the CPU 301 may acquire the camera parameters by determining them internally, for example, by communicating with the camera 111 via the communication interface 307 and acquiring the status of the camera 111. Once the CPU 301 has determined the camera parameters, the CPU 301 may provide the camera 111 with the camera parameters via the communication interface 307 to control the camera 111. Note that the camera parameters do not need to be acquired each time a captured image is acquired; they need only be acquired at least once before shape estimation. The acquired camera parameters are stored in the RAM 303 or the auxiliary storage device 304, for example, so that they can be used in subsequent processing.

[0033] In S502, the CPU 301 acquires a foreground image extracted from a captured image captured by the camera 111 of the imaging device 101. For example, the CPU 301 may acquire a captured image from the imaging device 101 via the communication interface 307, expand the captured image in the RAM 303, and perform a general foreground extraction process on the captured image to acquire the foreground image. The CPU 301 may also generate a silhouette image of the subject. For example, the silhouette image may be generated by setting pixel values ​​in an area where the subject does not exist to 0 and pixel values ​​in the subject area to a value other than 0. The silhouette image may be generated using a general method such as a background subtraction method, which calculates the difference between a background image captured in advance when the subject does not exist (e.g., an image of a stadium before a sports game starts) and a captured image of the subject. Note that the method for generating the silhouette image is not limited to this. For example, the silhouette image may be generated using a method for extracting the area of ​​the subject using a method for recognizing the subject (human body), etc. Furthermore, when foreground extraction processing is executed in the imaging device 101, the CPU 301 acquires a foreground image from the imaging device 101 via the communication interface 307. In this case, the imaging device 101 may execute the foreground extraction processing as described above. When acquiring a foreground image from the imaging device 101, the CPU 301 may generate a silhouette image by erasing texture information from the foreground image. When generating a silhouette image, the CPU 301 may further generate texture information for the foreground image. The silhouette image may also be generated in the imaging device 101. In this case, the CPU 301 may acquire the silhouette image via the communication interface 307. In this case, the CPU 301 may further acquire texture information for the foreground image via the communication interface 307. The silhouette image is used to estimate the general shape in step S503, which will be described later, and the texture information is used to improve the accuracy of the general shape in step S507. For this reason, the silhouette image and texture information may be temporarily stored in the RAM 303 or the auxiliary storage device 304.

[0034] In S503, the CPU 301 estimates the general shape of the subject using the silhouette image acquired in S502 and the camera parameters acquired in S501, and generates a general shape model. The general shape model is generated using an existing method, such as a volume intersection method. Through this process, the CPU 301 acquires point cloud information representing the shape of the subject. The coordinates [Pw]=(x, y, z) of each point are expressed, for example, in the same coordinate system (world coordinate system) as the coordinate system of the camera's external parameters. Note that throughout this embodiment, square brackets [·] indicate coordinates, vectors, or matrices containing multiple elements. Furthermore, the CPU 301 may divide the acquired point clouds into point cloud information for each subject by processing them using a general 3D labeling method based on the presence or absence of adjacent points. Each point is assigned a subject ID, which is the labeling result. Thereafter, the CPU 301 can acquire point cloud information for the corresponding subject by, for example, specifying the subject ID. Other general methods may also be used as shape estimation methods. The point cloud information for each subject is temporarily stored in, for example, the RAM 303 or the auxiliary storage device 304 for subsequent use.

[0035] In S504, CPU 301 generates a surface point cloud from the point cloud of the general shape model, excluding points located inside. The generated surface point cloud is stored in, for example, RAM 303 or auxiliary storage device 304 for subsequent use. Here, a method for determining whether each point included in the point cloud is an internal point or a surface point will be described. Whether a point is an internal point can be determined by referring to the connectivity of each point. Note that the point cloud is assumed to be regularly aligned and can be uniquely specified by coordinates (integers) on three orthogonal axes (x, y, z). Under this assumption, when a point included in the general shape model is set as a determination target, CPU 301 checks whether other points exist in six neighborhoods of the point (for example, in the x±1, y±1, and z±1 directions). If other points exist in all six neighborhood directions, CPU 301 determines that the target point is an internal point; otherwise, it determines that the target point is a surface point. The CPU 301 can generate a set of points that make up the surface of the subject by repeatedly executing this process for all points. The generated surface point group is temporarily stored in the RAM 303 or the auxiliary storage device 304.

[0036] In S505, the CPU 301 acquires the camera parameters of the camera 111 and information on the surface point cloud stored, for example, in the RAM 303, and generates distance images from each camera to the object. For each camera, a minimum distance image indicating the minimum distance from the camera to the object surface and a maximum distance image indicating the maximum distance from the camera to the object surface are generated. The minimum distance image and maximum distance image are temporarily stored, for example, in the RAM 303 or the auxiliary storage device 304. Here, a method for generating distance images indicating the distance from the camera to the object surface will be described. The distance image has the same width and height as the captured image, and a distance value is stored in each pixel. First, the world coordinates [Pw] of the surface point cloud points are converted to coordinates [Pc] in the camera coordinate system by multiplying them by an extrinsic matrix [Te]. [Te] is a transformation matrix composed of the camera's extrinsic parameters. If the camera position is the origin and the direction in which the camera lens faces corresponds to the positive range of the z-axis of the camera coordinate system, the z-coordinate of [Pc] represents the distance from the camera to that point. Next, the image coordinate [Pi] of [Pc] is calculated to determine the coordinate of the range image where the distance is stored. [Pi] is calculated by multiplying the normalized camera coordinate, obtained by normalizing [Pc] by the z coordinate, by the intrinsic matrix [Ti]. [Ti] is a matrix constructed based on the camera's internal parameters. If the pixel in [Pi] being calculated already stores a distance value for another point (i.e., a distance value calculated for another point in the surface point cloud projected onto the same image coordinate), the z coordinate value of the newly calculated coordinate is compared with the already stored value. When generating the shortest distance image, if the z coordinate value of the newly calculated coordinate is smaller than the already stored value, the z coordinate value of the newly calculated coordinate is stored as the pixel value in [Pi]. When generating the longest distance image, if the z coordinate value of the newly calculated coordinate is greater than the already stored value, the z coordinate value of the newly calculated coordinate is stored as the pixel value in [Pi]. By executing these processes for all points in the surface point cloud, the CPU 301 can generate the shortest distance image and the longest distance image for one camera.By executing this process for all cameras, the CPU 301 can generate the shortest distance image and the longest distance image for all cameras.

[0037] In S506, the CPU 301 acquires camera parameters stored in, for example, the RAM 303, a point cloud indicating the general shape of the subject, and a minimum distance image. The CPU 301 also acquires depth threshold information used to determine visibility, stored in, for example, the RAM 303 or the auxiliary storage device 304 in advance. The depth threshold information may be specified by a user operation received via, for example, the operation unit 306. The CPU 301 determines the visibility of the point cloud based on the acquired information and temporarily stores the resulting visibility information in, for example, the RAM 303 or the auxiliary storage device 304. Here, a method for determining the visibility information will be described. The CPU 301 first converts the world coordinate [Pw] of an arbitrary point in the point cloud indicating the general shape into the camera coordinate system and the image coordinate system, and calculates the camera coordinate [Pc] and the image coordinate [Pi]. The CPU 301 then compares the pixel value of the distance image at [Pi] with the z coordinate of [Pc] (the distance from the camera to the point) and compares the difference d with the depth threshold. If the difference d is less than the depth threshold, CPU 301 determines that the point is visible from the camera. On the other hand, if the difference d is equal to or greater than the depth threshold, CPU 301 determines that the point is not visible from the camera. CPU 301 performs the above process for all points included in the point cloud and all cameras. This determines whether each point in the point cloud representing the general shape is visible from each of multiple cameras, and generates visibility information that can identify whether any point included in the point cloud representing the general shape is visible from any camera.

[0038] In S507, CPU 301 acquires camera parameters, a point cloud indicating the general shape of the subject, a minimum distance image, a maximum distance image, and visibility information, which are stored in RAM 303, for example. CPU 301 then improves the accuracy of the general shape of the subject based on this information to generate a 3D shape model of the subject. The improvement in accuracy of the general shape of the subject will be described later with reference to FIG. 6. CPU 301 stores the generated 3D shape model of the subject in RAM 303 or auxiliary storage device 304, for example. CPU 301 can also provide the generated 3D shape model of the subject to an external device, for example via communication interface 307.

[0039] Next, the procedure for improving the accuracy of the general shape of the subject will be described with reference to FIG.

[0040] In S601, the CPU 301 executes initialization processing. The initialization processing is processing for executing the following processing for all points of the general shape and all of the multiple cameras 111. In this embodiment, the index l of each point included in the general shape point cloud is set to 0, and the camera index k is also set to 0. Here, it is assumed that an index of 0 to lmax-1 is assigned to each of the lmax points included in the general shape point cloud, and each point is uniquely specified by the index. Similarly, it is assumed that an index of 0 to kmax-1 is assigned to each of the kmax cameras 111, and each camera is uniquely specified by the index. Note that this is intended to enable processing to be performed for all points to be processed included in the general shape point cloud and all cameras to be processed included in the multiple cameras 111, and indexes do not necessarily have to be used. For example, any identification information capable of identifying each point or camera included in the general shape point cloud may be used. In addition, when the general shape and camera parameters are stored in consecutive areas of RAM 303, the information may be read out in order from the first address of the area in which each piece of information is stored, so that processing can be performed for all points and cameras.

[0041] In S602, the CPU 301 refers to the visibility information generated in S506 and stored, for example, in the RAM 303, and determines whether the point with index l is visible from the camera with index k. If the CPU 301 determines that the point with index l is visible from the camera with index k, in S603, the CPU 301 converts the world coordinate [Pw] of the point with index l into camera image coordinates [Pi] using the extrinsic and intrinsic parameters of the camera with index k. The CPU 301 then refers to and acquires pixel values ​​of the image coordinates [Pi] in the shortest distance image and the longest distance image related to the camera with index k.

[0042] In S604, CPU 301 calculates the distance difference by subtracting the pixel value of the shortest distance image from the pixel value of the longest distance image acquired in S603, and compares this distance difference with a previously acquired distance difference threshold. If CPU 301 determines that the distance difference is less than the threshold, it does not determine whether to delete that point. That is, if the subject has a long, thin portion, that portion as viewed from at least one camera will have one point referenced in forming the shortest distance image and another point referenced in forming the longest distance image, and the distance difference between them will be small. Therefore, a point with index l having such characteristics is considered to correspond to a thin or flat portion of the subject, and is therefore excluded from deletion. In this case, processing for the point with index l ends at this stage, and CPU 301 proceeds to processing in S609, described below. The case in which CPU 301 determines in S602 that the point with index l is not visible from the camera with index k will be described below.

[0043] If CPU 301 determines in S604 that the distance difference is equal to or greater than the threshold, in S605 CPU 301 acquires color information at the camera image coordinate [Pi] by referencing the pixel in the foreground image of the camera. CPU 301 may reduce the influence of noise by using color information not only of the pixel in the foreground image but also of the pixels surrounding that pixel. In this case, the projection size (number of pixels) of the point being processed when projected onto the camera image may be calculated based on the physical size of the point being processed (or the distance between adjacent points). Furthermore, CPU 301 may acquire pixel values ​​of a projection area having a radius half the physical size of the point being processed and a range centered on the camera image coordinate [Pi], and may use the average or median of these pixel values ​​as the pixel value of the point being processed when projected onto the camera.

[0044] Thereafter, in S606, the CPU 301 increments the camera index. That is, it adds +1 to the index k. Furthermore, even if the CPU 301 determines in S602 that the point with index l is not visible from the camera with index k, it also increments the camera index in S606. Then, in S607, the CPU 301 determines whether a camera with the incremented index exists. Note that FIG. 6 shows that if the index reaches kmax, which is not assigned to any camera, it is determined that a camera with the incremented index does not exist. If the CPU 301 determines in S607 that a camera with the incremented index exists, it repeats the processes of S602 to S607 for the camera with the incremented index and the point with index l.

[0045] By repeatedly executing the processes of S602 to S607, CPU 301 can collect color information corresponding to the point with index l in one or more images captured by one or more cameras that can capture the point without obstructions. Then, in S608, CPU 301 uses the collected color information to calculate the variance of the color (pixel value), and if the variance exceeds a predetermined threshold, removes the point with index l from the point cloud of the general shape. For example, if the color is expressed using RGB, CPU 301 may calculate the variance for each color element. Note that CPU 301 may also calculate the variance in another color space by changing the color representation method. For example, CPU 301 may remove the point with index l from the point cloud of the general shape if the color variance for one or more color elements exceeds a predetermined threshold. Note that CPU 301 may also remove the point with index l from the point cloud of the general shape if the average value of the variance values ​​for each of the color elements exceeds a predetermined threshold. For the determination in S608, the CPU 301 may acquire information about a color variance threshold value for deletion determination, which is stored in advance in, for example, the RAM 303 or the auxiliary storage device 304. This predetermined threshold value information may be specified by, for example, a user operation received via the operation unit 306.

[0046] Meanwhile, while repeatedly executing the processes of S602 to S607, if the point with index 1 corresponds to a thin or elongated portion of the subject in the range image of any of the cameras, CPU 301 will not perform the process of S608. That is, CPU 301 excludes such points from being deleted from the point cloud of the general shape. This prevents, for example, points corresponding to different faces of a thin or elongated portion of the subject from being treated as the same point and mistakenly deleted in the subsequent process of S608. This reduces errors in the 3D shape model of the subject that is finally output.

[0047] This completes the processing for the point with index l. Next, in S609, the CPU 301 increments the index l. That is, +1 is added to the index l. Then, in S610, the CPU 301 determines whether a point with the incremented index exists. If the CPU 301 determines in S610 that a point with the incremented index exists, the CPU 301 initializes the camera index k in S611. Then, the CPU 301 repeats the processing from S602 for the point with the incremented index. On the other hand, if the CPU 301 determines in S610 that such a point does not exist, this means that processing for all points included in the point cloud of the general shape has been completed, and the processing ends. Note that FIG. 6 shows that if the index reaches lmax, which is not assigned to any point, it is determined that a point with the incremented index does not exist.

[0048] (Second embodiment) In the first embodiment, an example was described in which a precision improvement process is performed only once on the general shape and a final 3D shape model of the subject is output. In the present embodiment, an example is described in which the 3D shape model resulting from the precision improvement process is used as a point cloud of the general shape and the above-mentioned precision improvement process is repeatedly applied. Note that, as shown in FIG. 7, the system configuration in this embodiment is generally similar to that of the first embodiment, except that the 3D shape model output by the shape precision improvement unit 136 is fed back to the surface shape generation unit 133 as a point cloud of the general shape.

[0049] Fig. 8 illustrates the processing executed by the shape estimation device 102 in this embodiment. This processing is the same as the processing shown in Fig. 5 for the first embodiment, except that processing of S801 is added between S505 and S506, and processing returns to S504 after processing of S507.

[0050] After performing the general shape high-precision processing in S507, CPU 301 repeats the processing from S504 onward using the high-precision point cloud as the general shape point cloud. If CPU 301 determines to continue the processing in S801, it performs further high-precision processing. On the other hand, if CPU 301 determines not to continue the processing in S801, it can output the result of the previous high-precision processing in S507 as the final 3D shape model of the subject.

[0051] Here, a method for determining whether to continue the process of S801 will be described. For example, the CPU 301 makes this determination using information on the number of repetitions or the amount of change in the distance image.

[0052] When information on the number of repetitions is used, information on the predetermined number of repetitions to be performed is input to the system in advance, for example, through a configuration file, and stored in RAM 303 or the auxiliary storage device 304. In this case, CPU 301 may acquire the information on the predetermined number of repetitions by referencing RAM 303 or the auxiliary storage device 304. The information on the predetermined number of repetitions may also be specified, for example, by a user operation received via the operation unit 306. CPU 301 may determine in S801 whether the number of times the high accuracy processing was performed in S507 has reached the predetermined number. In this case, if the number of times the high accuracy processing was performed in S507 has reached the predetermined number, CPU 301 may determine in S801 that no further processing will be performed and terminate the processing. On the other hand, if the number of times the high accuracy processing was performed in S507 has not reached the predetermined number, CPU 301 determines in S801 to continue the processing and subsequently executes the processing of S506. In this way, when the determination of whether to continue the processing is made based on the number of times the high accuracy processing has been performed, the CPU 301 may make the determination immediately after the end of the processing of S507. In other words, the CPU 301 does not need to repeat the processing of S504 and S505 in a situation where the high accuracy processing is not being performed any more.

[0053] On the other hand, when the amount of change in the distance image is used, CPU 301 calculates, for example, the distance difference between the shortest distance image from the second most recently executed high-precision processing and the shortest distance image from the most recently executed high-precision processing for each pixel value of each camera. Then, when the number of pixels whose distance difference is equal to or greater than a preset threshold is less than a predetermined number in all cameras, CPU 301 may determine that further high-precision processing is not necessary and may determine not to continue processing in S801. Furthermore, when the number of pixels whose distance difference is equal to or greater than a threshold is equal to or greater than a predetermined number in one or more cameras, CPU 301 may determine that further high-precision processing is necessary and may determine to continue processing in S801. Note that, for example, if the sum of the amounts of change in each camera and each pixel falls below a predetermined value, it may be determined that further repeated processing is not necessary. In this way, the necessity of repeated processing may be determined based on various criteria for the amount of change. Note that when the amount of change in the distance image is used in this way, the latest distance image needs to be calculated, so CPU 301 may execute processing of S801 after executing processing of S504 and S505. The predetermined number is input to the system in advance, for example, through a configuration file, and is stored in RAM 303 or auxiliary storage device 304. In this case, CPU 301 can acquire information about the predetermined number by referring to RAM 303 or auxiliary storage device 304. The information about the predetermined number may also be specified by, for example, a user operation received via operation unit 306. CPU 301 may determine to end the repetitive process based on the number of repetitions if the magnitude of the amount of change in the distance image does not converge.

[0054] Note that when executing the repetitive process, the CPU 301 may change the distance difference threshold used in S604 for each repetition. In this case, distance difference thresholds for the maximum number of times the high-precision process is expected to be executed are input to the system in advance, for example, via a configuration file, and stored in the RAM 303 or the auxiliary storage device 304. For each repetitive process, the CPU 301 references the RAM 303 or the auxiliary storage device 304 to set the distance difference threshold to be used in that repetitive process. The distance difference threshold may also be specified, for example, by a user operation received via the operation unit 306. The distance difference threshold may be set to a relatively large value in the first high-precision process and may be set to a smaller value with each repetition of the process. In this way, many points are deleted in the first high-precision process, in which large shape errors such as recesses remain, and each time the high-precision process is repeated, a precise determination of whether to delete points can be made.

[0055] By repeatedly executing the high-precision processing in this manner, it becomes possible to generate a three-dimensional shape model that more faithfully represents the subject.

[0056] While the above processes are described as temporarily storing information in RAM 303 or auxiliary storage device 304, some or all of the information may be retained in the cache of CPU 301 until the general shape model refinement process is completed. Furthermore, the above processes may be performed in parallel by multiple CPUs 301, for example, processing multiple points included in a general shape point cloud or processing multiple cameras. While the above embodiment describes an example in which the shape of a subject is represented by a point cloud, similar processing can be performed when the subject is represented by other elements. That is, for a first element group representing a subject using arbitrary elements, a distance image from a camera to the surface of an object represented by the first element group may be acquired, and unnecessary elements may be deleted based on the distance image to generate a second element group. Elements may include, for example, faces, vertices, and edges of a mesh structure.

[0057] Although the above-described process has been described as an example in which a minimum distance image and a maximum distance image are generated, the process is not limited to this. For example, in S603, CPU 301 may calculate the distance between the first point and a second point, which is different from the first point, among the intersections between a line connecting the camera position with index k to the point (first point) of index l and the surface of the general shape. In this case, since the first point is a point visible from the camera, the distance between the first point and the camera position corresponds to the minimum distance. On the other hand, for example, when a recess is formed by repeating the high-precision process, it is assumed that there are multiple second points, which are different from the first point, among the intersections between a line connecting the camera position and the first point and the surface of the general shape. In this case, for example, the point closest to the first point may be selected as the second point. In addition, if the element representing the shape of the subject is not a point, CPU 301 may extend a line connecting the edges of the range indicated by the first element from the camera position and identify the range of the second element corresponding to the position where the line intersects with the surface of the shape of the subject. The CPU 301 may then determine whether to delete the element based on the difference in distance between the range of the first element and the range of the second element. The determination of whether to repeat the above-described process may be based on information about the shortest distance. The determination of whether to perform the repeat process based on the shortest distance image may be based on another distance reference. For example, the determination of whether to perform the repeat process may be based on the distance from the camera to the second-closest surface. Alternatively, another convergence determination criterion may be used, such as terminating the repeat process when the magnitude of the change in the element group falls below a predetermined value.

[0058] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0059] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0060] 101: imaging device, 102: shape estimation device, 111: camera, 132: general shape estimation unit, 133: surface shape generation unit, 134: distance image generation unit, 135: visibility determination unit, 136: shape precision improvement unit, 301: CPU, 303: RAM

Claims

1. an estimation means for estimating a first group of elements representing a three-dimensional shape of a subject based on captured images obtained by capturing images of the subject from different directions using a plurality of image capturing devices; a determination means for determining, for each element included in the first element group, whether or not to delete the element from the first element group based on a color in the captured image corresponding to the element; a generating means for generating a second group of elements by deleting the element determined to be deleted from the first group of elements; and the determination means determines that, for element A among elements constituting the surface of the shape represented by the first element group, if a distance between element A and an element different from element A that corresponds to a position where a straight line connecting element A and an imaging device that has imaged element A intersects with the surface does not exceed a predetermined threshold, element A is not to be deleted from the first element group, regardless of the color; A shape estimation device characterized by:

2. 2. The shape estimation device according to claim 1, further comprising means for generating a third element group from the first element group by excluding elements that do not correspond to a surface of the shape represented by the first element group.

3. The shape estimation device according to claim 1 or 2, characterized in that, when a straight line connecting the imaging device that captured the element A and the element A intersects the surface at multiple positions, the determination means determines whether the distance between the element A and an element that corresponds to a position closest to the element A and is different from the element A among the multiple positions exceeds the predetermined threshold.

4. The image capturing device further includes an acquisition means for acquiring, from an image capturing device that captures an image of the element A, a minimum distance image that indicates the distance to the closest position and a maximum distance image that indicates the distance to the farthest position on the surface of the shape represented by the first element group, 3. The shape estimation device according to claim 1, wherein the determining means specifies the distance using the shortest distance image and the longest distance image.

5. 5. The shape estimation device according to claim 1, wherein the generating means provides feedback to the determining means so as to execute repeated processing using the second element group as the first element group.

6. 6. The shape estimation device according to claim 5, wherein the repeated processing is performed until a predetermined number of repetitions are reached.

7. 7. The shape estimation device according to claim 5, wherein the repeated processing is performed until a change in distance from the plurality of imaging devices to a surface of the shape represented by the second element group satisfies a predetermined criterion.

8. 8. The shape estimation device according to claim 5, wherein a different predetermined threshold value is set for each of the repeated processes.

9. 9. The shape estimation device according to claim 1, wherein the determination means uses a color in a range when each element included in the first element group is projected onto the captured image as a color corresponding to the element, and determines whether or not to delete the element from the first element group.

10. 10. The shape estimation device according to claim 1, wherein the first element group and the second element group are point groups.

11. A processing method executed by a shape estimation device, comprising: an estimation step of estimating a first group of elements representing a three-dimensional shape of the subject based on captured images obtained by capturing images of the subject from different directions using a plurality of image capturing devices; a determination step of determining, for each element included in the first element group, whether or not to delete the element from the first element group based on a color in the captured image corresponding to the element; a generating step of generating a second group of elements by deleting the element determined to be deleted from the first group of elements; Including, In the determination step, for an element A among the elements constituting the surface of the shape represented by the first element group, if a distance between the element A and an element different from the element A, which corresponds to a position where a straight line connecting the element A and an imaging device that has imaged the element A intersects with the surface, does not exceed a predetermined threshold, the element A is determined not to be a target for deletion from the first element group, regardless of the color. A processing method characterized by:

12. A program for causing a computer to function as the shape estimation device according to any one of claims 1 to 10.

Images