Three-dimensional model generation method and three-dimensional model generation apparatus

Abstract

The three-dimensional model generation method is a three-dimensional model generation method executed by a computer, acquires a first image obtained by capturing an object from a first viewpoint and a second image obtained by capturing the object from a second viewpoint; detects a first line constituted by continuous edges and corresponding to an outline of the object in at least the first image (S115); and generates a plurality of first three-dimensional points expressing the outline in a three-dimensional space in the computer based on the first image, the second image, and the first line (S116).

Description

Technical Field

[0001] This disclosure relates to a method and apparatus for generating three-dimensional models. Background Technology

[0002] Patent Document 1 discloses a technique for generating a three-dimensional model of a subject using multiple images obtained by photographing the subject from multiple viewpoints.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2017-130146 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] Regarding the generation and processing of 3D models, the goal is to improve the accuracy of 3D model generation.

[0008] This disclosure provides methods for generating 3D models that can improve the accuracy of 3D model generation.

[0009] Methods used to solve problems

[0010] A three-dimensional model generation method of the present disclosure is a computer-executed three-dimensional model generation method that obtains a first image obtained by photographing a subject from a first viewpoint and a second image obtained by photographing the subject from a second viewpoint; at least in the first image, a first line consisting of continuous edges and corresponding to the contour of the subject is detected; based on the first image, the second image, and the first line, a plurality of first three-dimensional points representing the contour in a three-dimensional space within the computer are generated.

[0011] Furthermore, a three-dimensional model generation apparatus according to a technical solution of this disclosure includes a computer; the computer acquires a first image obtained by photographing a subject from a first viewpoint and a second image obtained by photographing the subject from a second viewpoint; the computer detects, at least in the first image, a first line consisting of continuous edges and corresponding to the contour of the subject; and the computer generates a plurality of first three-dimensional points representing the contour in a three-dimensional space within the computer based on the first image, the second image, and the first line.

[0012] Furthermore, this disclosure can also be implemented as a program that causes a computer to perform the steps included in the above-described three-dimensional model generation method. Additionally, this disclosure can also be implemented as a non-transitory recording medium such as a CD-ROM that can be read by a computer and contains the program. Furthermore, this disclosure can also be implemented as information, data, or signals representing the program. Moreover, these programs, information, data, and signals can also be distributed via communication networks such as the Internet.

[0013] Invention Effects

[0014] The three-dimensional model generation method disclosed herein can improve the accuracy of three-dimensional model generation. Attached Figure Description

[0015] Figure 1 This is a diagram used to illustrate the outline of the three-dimensional model generation method of Implementation Method 1.

[0016] Figure 2 This is a block diagram showing the characteristic structure of the three-dimensional model generation apparatus of Embodiment 1.

[0017] Figure 3 This is a diagram used to illustrate the search process related to Implementation Method 1.

[0018] Figure 4 This is a block diagram showing the functional structure of the modification part related to Embodiment 1.

[0019] Figure 5 This is a diagram illustrating an example of the processing of the detection profile in Implementation 1.

[0020] Figure 6 This is a diagram illustrating an example of the processing of the detection profile in Implementation 1.

[0021] Figure 7 This is a diagram illustrating an example of the process for correcting the positions of multiple third-dimensional points in Implementation 1.

[0022] Figure 8 This is a flowchart illustrating an example of the operation of the three-dimensional model generation apparatus according to Embodiment 1.

[0023] Figure 9 This is a flowchart illustrating an example of the detailed process for generating step S104 of Implementation 1.

[0024] Figure 10 This is a diagram illustrating an example of the process for correcting the positions of multiple third-dimensional points in a variation of Embodiment 1.

[0025] Figure 11This is a diagram illustrating an example of the process for correcting the positions of multiple third-dimensional points in a variation of Embodiment 1.

[0026] Figure 12 This is a diagram illustrating an example of the process for correcting the positions of multiple third-dimensional points in a variation of Embodiment 1.

[0027] Figure 13 This is a diagram illustrating an example of the process for correcting the positions of multiple third-dimensional points in a variation of Embodiment 1.

[0028] Figure 14 This is a diagram illustrating an example of the processing performed by the correction unit in a variation 2 of embodiment 1.

[0029] Figure 15 This is a diagram illustrating an example of the processing performed by the correction unit in a variation 2 of embodiment 1.

[0030] Figure 16 This is a block diagram showing the characteristic structure of the three-dimensional model generation apparatus of Embodiment 2.

[0031] Figure 17 This is a diagram illustrating an example of the processing for correcting each frame in Implementation 2.

[0032] Figure 18 This is a diagram illustrating an example of the processing for correcting each frame in Implementation 2.

[0033] Figure 19 This is a flowchart illustrating an example of the detailed process for generating step S104 of embodiment 2. Detailed Implementation

[0034] (The process of reaching this agreement is made public)

[0035] In the technology disclosed in Patent Document 1, a three-dimensional model is generated by searching for similar points among multiple images. Typically, in the search for similar points, when searching for similar points for a single pixel in one image from other images, epipolar lines on the other images are calculated based on the camera's geometric constraints, and multiple pixels along these epipolar lines are searched. Furthermore, triangulation is performed using the pixels matched in the similar point search and the camera's position and pose when the image was captured, and the positions of the captured three-dimensional points are determined using the matched pixels. By performing this processing on multiple pixels separately, a three-dimensional model containing multiple three-dimensional points is generated.

[0036] However, in the aforementioned conventional techniques, since three-dimensional models are generated from two-dimensional images, the accuracy of the positions of multiple three-dimensional points in the three-dimensional model may decrease due to the aliasing that occurs in the two-dimensional image. In particular, the farther the subject is from the camera, the larger the spatial size corresponding to one pixel in the two-dimensional image, and therefore the greater the error in the position of the generated three-dimensional points. Consequently, the accuracy of the generated three-dimensional model decreases.

[0037] Therefore, this disclosure provides a method for generating three-dimensional models that can improve the accuracy of three-dimensional model generation.

[0038] A three-dimensional model generation method of the present disclosure is a computer-executed three-dimensional model generation method that obtains a first image obtained by photographing a subject from a first viewpoint and a second image obtained by photographing the subject from a second viewpoint; at least in the first image, a first line consisting of continuous edges and corresponding to the contour of the subject is detected; based on the first image, the second image, and the first line, a plurality of first three-dimensional points representing the contour in a three-dimensional space within the computer are generated.

[0039] Therefore, by detecting the first line corresponding to the contour of the subject from the image, and using the detected first line in addition to multiple images, multiple first 3D points are generated. In this way, by detecting the first line corresponding to the contour of the subject and considering the first line in the generation of the first 3D model, errors caused by aliasing in the image can be reduced. Therefore, the generation accuracy of the first 3D model can be improved.

[0040] Alternatively, the first line may be represented in units smaller than the pixel units in the first image.

[0041] Alternatively, by matching similar points in the first and second images to generate multiple second three-dimensional points representing the subject in the three-dimensional space, a plurality of second three-dimensional points are generated. In order to generate the plurality of first three-dimensional points, (i) based on the first line, a plurality of third three-dimensional points corresponding to the contour are determined among the plurality of second three-dimensional points; (ii) the positions of the plurality of third three-dimensional points are corrected.

[0042] Therefore, multiple third-dimensional points are identified as a group of points corresponding to the contour of the subject, and the positions of these points are corrected using this identified group. Thus, the positions of the point groups can be appropriately adjusted according to the shape of the subject's contour.

[0043] Alternatively, in the second image described above, a second line consisting of continuous edges and corresponding to a contour containing at least a portion of the contour corresponding to the first line described above can be detected. For the generation of the plurality of first three-dimensional points, further, (i) based on the second line, a plurality of fourth three-dimensional points are determined among the plurality of second three-dimensional points, the plurality of fourth three-dimensional points corresponding to the contour corresponding to the second line described above, and including the corrected plurality of third three-dimensional points and the uncorrected second three-dimensional points; (ii) among the plurality of fourth three-dimensional points, at least the position of the uncorrected second three-dimensional points is corrected.

[0044] Therefore, it is possible to generate the positions of multiple first three-dimensional points corresponding to the continuous contours of the subject with high precision.

[0045] Alternatively, in order to generate the aforementioned plurality of first three-dimensional points, a plane is determined by performing principal component analysis on the aforementioned plurality of third three-dimensional points, and the positions of the aforementioned plurality of third three-dimensional points are corrected in such a way that the aforementioned plurality of third three-dimensional points are close to the aforementioned plane.

[0046] Therefore, the positions of multiple third-dimensional points can be corrected with high precision based on the shape of the contour.

[0047] Alternatively, in order to generate the plurality of first three-dimensional points, a plane is generated that has the smaller sum of distances from the plurality of third three-dimensional points to the first plane and the second plane. The first plane passes through the centroid of the plurality of third three-dimensional points and takes the first principal component obtained through the principal component analysis as its normal. The second plane passes through the centroid and takes the second principal component obtained through the principal component analysis as its normal.

[0048] Therefore, even when the detected first line is not long enough, the positions of multiple third three-dimensional points can be determined with high precision based on the first line as a line segment.

[0049] Alternatively, in order to generate the first three-dimensional point, an approximate line can be generated by performing the least squares method on the plurality of third three-dimensional points, and the positions of the plurality of third three-dimensional points can be corrected in such a way that the plurality of third three-dimensional points are close to the approximate line.

[0050] Therefore, it is possible to accurately correct the positions of multiple third-dimensional points based on the approximate lines that approximate a group of points.

[0051] Alternatively, in order to generate the plurality of first three-dimensional points, three-dimensional points located at a distance closer than a specified distance from the first viewpoint may be removed from the plurality of first three-dimensional points.

[0052] In this way, the first 3D model is not generated using contours in parts where the accuracy is likely to be above a certain level, thus reducing the processing load without causing a decrease in accuracy.

[0053] Alternatively, for the generation of the aforementioned plurality of first three-dimensional points, edges detected from images with a specified resolution or higher in the aforementioned first image and the aforementioned second image may not be used.

[0054] In this way, the first 3D model is not generated using contours in parts where the accuracy is likely to be above a certain level, thus reducing the processing load without causing a decrease in accuracy.

[0055] Alternatively, in the above detection, a second line consisting of continuous edges and corresponding to the outline of the subject can be detected in the second image; in the generation of the plurality of first three-dimensional points, (i) the first image and the second image are corrected respectively using the first line and the second line, and (ii) a matching search for similar points is performed in the corrected first image and the corrected second image.

[0056] Therefore, by using the image corrected by contours to generate the first 3D model, it is possible to generate a first 3D model that reduces the error caused by aliasing in the image.

[0057] Furthermore, a three-dimensional model generation apparatus according to a technical solution of this disclosure includes a computer; the computer acquires a first image obtained by photographing a subject from a first viewpoint and a second image obtained by photographing the subject from a second viewpoint; the computer detects, at least in the first image, a first line consisting of continuous edges and corresponding to the contour of the subject; and the computer generates a plurality of first three-dimensional points representing the contour in a three-dimensional space within the computer based on the first image, the second image, and the first line.

[0058] Therefore, by detecting the first line corresponding to the contour of the subject from the image, and using the detected first line in addition to multiple images, multiple first 3D points are generated. In this way, by detecting the first line corresponding to the contour of the subject and considering the first line in the generation of the first 3D model, errors caused by aliasing in the image can be reduced. Therefore, the generation accuracy of the first 3D model can be improved.

[0059] Hereinafter, various embodiments of the three-dimensional model generation method of this disclosure will be described in detail with the aid of accompanying drawings. Furthermore, each embodiment described below represents a specific example of this disclosure. Therefore, the numerical values, shapes, constituent elements, arrangements and connection patterns of constituent elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure.

[0060] Furthermore, the figures are schematic diagrams and not necessarily rigorous illustrations. Additionally, the same labels are assigned to substantially identical structures across figures, and there are instances where repetitive explanations are omitted or simplified.

[0061] (Implementation Method 1)

[0062] [summary]

[0063] First, refer to Figure 1 An overview of the three-dimensional model generation method of Implementation Method 1 is provided.

[0064] Figure 1 This is a diagram used to illustrate the outline of the three-dimensional model generation method of Implementation Method 1. Figure 2 This is a block diagram showing the characteristic structure of the three-dimensional model generation apparatus 100 according to Embodiment 1.

[0065] In 3D model generation methods, such as Figure 1 As shown, a three-dimensional model of a defined region is generated based on multiple images captured from multiple viewpoints using multiple camera devices 301. Here, the defined region is a region containing stationary objects or moving objects, or both. In other words, the defined region is, for example, a region containing at least one of a stationary object and a moving object as the subject.

[0066] Examples of defined areas that include both stationary and moving objects include stadiums for sports games such as basketball, or spaces on roads where people or vehicles are present. Furthermore, a defined area may include not only a specific object as the subject of a photograph, but also scenery. Figure 1 The example shown is a building. Furthermore, the following will not only include specific objects as subjects, but also defined areas such as landscapes, which will also be simply referred to as subjects.

[0067] like Figure 2 As shown, the 3D model generation system 400 includes a camera device group 300 comprising multiple camera devices 301, an estimation device 200, and a 3D model generation device 100.

[0068] (Multiple camera devices)

[0069] The multiple camera devices 301 are multiple camera devices that capture images of a designated area. Each of the multiple camera devices 301 captures an image of the subject and outputs the captured frames to the estimation device 200. In this embodiment, the camera device group 300 includes two or more camera devices 301. Furthermore, the multiple camera devices 301 capture the same subject from different viewpoints. A frame, in other words, is an image.

[0070] Furthermore, assuming the 3D model generation system 400 includes a group of camera devices 300, but is not limited to this, it may also include a single camera device 301. For example, in the 3D model generation system 400, a single camera device 301 can be moved while simultaneously capturing images of a subject existing in actual space to generate a multi-viewpoint image composed of multiple frames with different viewpoints. These multiple frames are frames captured (generated) by camera devices 301 that differ in at least one of position and orientation. That is, the multiple frames are frames captured from different viewpoints.

[0071] Furthermore, each imaging device 301 can be either a camera that generates two-dimensional images or a camera equipped with a three-dimensional measurement sensor that generates three-dimensional models. In this embodiment, the multiple imaging devices 301 are cameras that generate two-dimensional images by taking pictures.

[0072] Multiple camera devices 301 can be directly connected to the estimation device 200 via wired or wireless communication, or indirectly connected to the estimation device 200 via a hub (not shown) such as a communication device or server, so that the frames captured separately can be output to the estimation device 200.

[0073] In addition, frames captured by multiple camera devices 301 can also be output to the estimation device 200 in real time. Alternatively, frames can be first recorded in an external storage device such as a memory or a cloud server, and then output from these external storage devices to the estimation device 200.

[0074] Furthermore, the multiple camera devices 301 can be fixed cameras such as surveillance cameras, mobile cameras such as stereo cameras, smartphones or wearable cameras, or mobile cameras such as drones with shooting functions.

[0075] (Speculation device)

[0076] The estimation device 200 performs camera correction by having one or more camera devices 301 capture images of the subject from multiple viewpoints. For example, the estimation device 200 performs camera correction by estimating the position and orientation of multiple camera devices 301 based on multiple frames captured by the multiple camera devices 301 respectively. Here, the orientation of the camera device 301 refers to at least one of the shooting direction of the camera device 301 and the tilt of the camera device 301. The shooting direction of the camera device 301 is the direction of the optical axis of the camera device 301. The tilt of the camera device 301 is the angle of rotation of the camera device 301 about the optical axis from the reference orientation.

[0077] Specifically, the estimation device 200 estimates the camera parameters of the multiple camera devices 301 based on multiple frames acquired from the multiple camera devices 301. Here, camera parameters are parameters representing the characteristics of the camera devices 301, including internal parameters such as the focal distance and image center of the camera devices 301, and external parameters representing the position (more specifically, the three-dimensional position) and pose of the camera devices 301. That is, the position and pose of each of the multiple camera devices 301 are obtained by estimating the camera parameters of each of the multiple camera devices 301.

[0078] Furthermore, the method by which the estimation device 200 estimates the position and orientation of the camera device 301 is not particularly limited. For example, the estimation device 200 may use Visual-SLAM (Simultaneous Localization and Mapping) technology to estimate the position and orientation of multiple camera devices 301. Alternatively, the estimation device 200 may use Structure-From-Motion technology to estimate the position and orientation of multiple camera devices 301.

[0079] Here, using Figure 3 The method for predicting the position and orientation of multiple camera devices 301 by the prediction device 200 is explained. Figure 3 This diagram is used to illustrate the search process performed by the restoration section (described later). However, since the same process is also performed in the inference method, it utilizes... Figure 3 Please provide an explanation.

[0080] The inference device 200 uses Visual-SLAM or Structure-From-Motion technology to search for feature points. This feature point search involves extracting characteristic points from each of the multiple frames 531-533 captured by the multiple camera devices 301, and then extracting groups of similar points across the multiple frames from these extracted feature points. By searching for feature points, the inference device 200 can determine the points on the subject 510 captured in the multiple frames 531-533. Therefore, it can use the extracted groups of similar points to calculate the three-dimensional coordinates (i.e., three-dimensional position) of the points on the subject 510 using the principle of triangulation.

[0081] In this way, the estimation device 200 can infer the position and pose of each camera device 301 by extracting and using groups of similar points. During the estimation of the position and pose of each camera device 301, the estimation device 200 can also calculate the three-dimensional coordinates for each group of similar points, generating a three-dimensional model 520 containing multiple three-dimensional points represented by the calculated three-dimensional coordinates. Each of the multiple three-dimensional points represents a position on the subject in three-dimensional space. The estimation device 200 obtains the position and pose of each camera device 301 and map information as the estimation result. Because the obtained three-dimensional model 520 is optimized along with camera parameters, it contains information with higher accuracy than the specified accuracy. Furthermore, the three-dimensional model 520 includes the three-dimensional positions of each of the multiple three-dimensional points. In addition, the three-dimensional model 520 may also include not only multiple three-dimensional positions, but also the color of each three-dimensional point, the surface shape surrounding each three-dimensional point, and information indicating which frame each three-dimensional point was generated from.

[0082] Furthermore, to accelerate the estimation process, the estimation device 200 can generate a 3D model 520 including sparse 3D point groups by limiting the number of similar point groups to a predetermined number. This is because, even with a predetermined number of similar point groups, the estimation device 200 can estimate the position and orientation of each camera device 301 with sufficient accuracy. Additionally, the predetermined number can be determined to be the number of points that can estimate the position and orientation of each camera device 301 with sufficient accuracy. Furthermore, the estimation device 200 can also use groups of similar points with a predetermined similarity or higher to estimate the position and orientation of each camera device 301. As a result, the estimation device 200 can limit the number of similar point groups used in the estimation process to a predetermined number of similar groups with a predetermined similarity or higher.

[0083] Furthermore, the estimation device 200 can, for example, calculate the distance between the camera device 301 and the subject 510 based on the position and posture of the camera device 301 estimated using the above-described technology, as a camera parameter. Additionally, the 3D model generation system 400 can also be equipped with a distance sensor, which is used to determine the distance between the camera device 301 and the subject 510.

[0084] The inference device 200 can be directly connected to the 3D model generation device 100 via wired or wireless communication, or indirectly connected via a hub (not shown) such as a communication device or server. Thus, the inference device 200 outputs multiple frames received from multiple camera devices 301 and multiple inferred camera parameters of the multiple camera devices 301 to the 3D model generation device 100.

[0085] In addition, multiple camera parameters inferred by the inference device 200 can also be output to the 3D model generation device 100 in real time. Furthermore, multiple camera parameters can also be first recorded in an external storage device such as a memory or a cloud server, and then output from these external storage devices to the 3D model generation device 100.

[0086] It is speculated that the device 200 includes at least a computer system, which includes, for example, a control program, a processor or logic circuit for executing the control program, and a recording device for storing the control program in internal memory or in external memory that can be accessed.

[0087] (3D model generation device)

[0088] The 3D model generation apparatus 100 generates a 3D model of a specified area based on multiple frames captured by multiple camera devices 301 and camera parameters inferred by the inference device 200. Specifically, the 3D model generation apparatus 100 is an apparatus that performs 3D model generation processing to generate a 3D model of a subject in a virtual 3D space based on multiple camera parameters and multiple frames corresponding to the multiple camera devices 301.

[0089] Furthermore, the 3D model of the subject is a data representation of the subject's 3D shape and color, reconstructed in a virtual 3D space from frames of the actual subject. The 3D model of the subject is a set of points representing the 3D positions of multiple points on the subject captured from multiple 2D images taken by multiple camera devices 301 from multiple viewpoints.

[0090] Three-dimensional position can be represented, for example, by three-valued information consisting of X-components, Y-components, and Z-components representing the positions of the mutually orthogonal X-axis, Y-axis, and Z-axis. That is, three-dimensional position is represented by three-dimensional coordinates in a three-dimensional orthogonal coordinate system. However, three-dimensional position is not limited to three-dimensional coordinates in a three-dimensional orthogonal coordinate system; it can also be represented by three-dimensional coordinates in a three-dimensional polar coordinate system. Furthermore, the information contained in multiple points representing three-dimensional position may include not only positional information (i.e., coordinate information) but also information such as color and surface shape attributes representing each point and its surroundings.

[0091] The 3D model generation apparatus 100 includes at least a computer system, which includes, for example, a control program, a processor or logic circuit for executing the control program, and a recording device such as internal memory or accessible external memory for storing the control program. The 3D model generation apparatus 100 is an information processing device. The functions of each processing unit of the 3D model generation apparatus 100 can be implemented either by software or by hardware.

[0092] Furthermore, the 3D model generation apparatus 100 can also pre-store camera parameters of multiple camera devices 301. In this case, the 3D model generation system 400 may not need to include the estimation device 200. Additionally, the multiple camera devices 301 can be wirelessly or wiredly connected to the 3D model generation apparatus 100.

[0093] Furthermore, multiple frames captured by the camera device 301 can be directly output to the 3D model generation device 100. In this case, the camera device 301 can be directly connected to the 3D model generation device 100 via wired or wireless communication, or it can be indirectly connected to the estimation device 200 via a hub (not shown) such as a communication device or server.

[0094] Next, refer to Figure 2 The structure of the three-dimensional model generation device 100 is described in detail.

[0095] The 3D model generation apparatus 100 is an apparatus that generates 3D models based on multiple frames. The 3D model generation apparatus 100 includes a receiving unit 110, a storage unit 120, an acquisition unit 130, a generation unit 140, and an output unit 150.

[0096] The receiving unit 110 receives multiple frames captured by multiple imaging devices 301 and multiple camera parameters inferred by the inference device 200 from the inference device 200. Thus, the receiving unit 110 obtains a first frame (first image) obtained by photographing the subject from a first viewpoint and a second frame (second image) obtained by photographing the subject from a second viewpoint. That is, the multiple frames received by the receiving unit 110 include the first frame and the second frame. For example, if the subject is a stationary object, these multiple frames are obtained by photographing the subject from multiple different viewpoints. For example, if the subject is a moving object, these multiple frames are obtained by photographing the subject from multiple viewpoints at predetermined timings (i.e., simultaneously). The 3D model generation apparatus 100 performs processing to generate a first 3D model using a frame set as the processing unit, where the multiple frames are treated as a single set. The frame set is sometimes referred to as a multi-viewpoint image. The receiving unit 110 may also obtain a 3D model 520 from the inference device 200. The receiving unit 110 outputs the received multiple frames and camera parameters to the storage unit 120.

[0097] The receiving unit 110 is, for example, a communication interface for communicating with the estimation device 200. When the 3D model generation apparatus 100 and the estimation device 200 communicate wirelessly, the receiving unit 110 includes, for example, an antenna and a wireless communication circuit. Alternatively, when the 3D model generation apparatus 100 and the estimation device 200 communicate via wired connection, the receiving unit 110 includes, for example, a connector for connecting to a communication line and a wired communication circuit. Furthermore, the receiving unit 110 can also receive multiple frames from multiple imaging devices 301 without going through the estimation device 200.

[0098] The storage unit 120 stores multiple frames and multiple camera parameters received by the receiving unit 110. The storage unit 120 may also store the 3D model 520 received by the receiving unit 110. Furthermore, the storage unit 120 may also store the processing results of each processing unit included in the 3D model generation apparatus 100. For example, the storage unit 120 stores control programs executed by each processing unit included in the 3D model generation apparatus 100. The storage unit 120 may be implemented, for example, by an HDD (Hard Disk Drive), flash memory, or the like.

[0099] The acquisition unit 130 acquires multiple frames and camera parameters of each imaging device 301 stored in the storage unit 120, and outputs them to the generation unit 140.

[0100] Alternatively, the 3D model generation apparatus 100 may not include the storage unit 120 and the acquisition unit 130. In this case, the receiving unit 110 may output multiple frames received from multiple imaging devices 301 and camera parameters of each imaging device 301 received from the estimation device 200 to the generation unit 140.

[0101] The generation unit 140 generates a first 3D model using multiple frames and multiple camera parameters. The first 3D model contains multiple first 3D points. The generation unit 140 has a restoration unit 141 and a correction unit 142. The restoration unit 141 generates multiple second 3D points representing the subject in 3D space by matching similar points in multiple frames, such as the first frame and the second frame. The correction unit 142 corrects the positions of the multiple second 3D points. Thus, the first 3D model is generated. The multiple first 3D points contained in the first 3D model are generated by the correction unit 142 correcting at least a portion of the multiple second 3D points. Hereinafter, the restoration unit 141 and the correction unit 142 will be described in detail.

[0102] The restoration unit 141 searches for similar points in the second frame from among multiple second points in a search region based on the first point, which are similar to the first point in the first frame. The first frame is the frame (image) that serves as the reference source for the first point among the multiple frames. The second frame is one of the remaining frames after excluding the first frame, and is the frame (image) that serves as the reference destination for the search for similar points. Specifically, the first point is one of multiple first pixels. Multiple first pixels are pixels contained in the first frame, for example, all pixels contained in the first frame. The search region is the area in the multiple second frames defined by the epipolar line corresponding to the first point in the first frame, for example, the area formed by multiple second points on the epipolar line. Specifically, the multiple second points are multiple second pixels contained within the search region.

[0103] Alternatively, the restoration unit 141 can also search for similar points (similar pixels) similar to the first pixel from among the multiple second pixels included in the second frame for each of the multiple first pixels. The multiple second pixels, for example, are all the pixels included in the second frame. Furthermore, the restoration unit 141 can search for similar points of the first pixel either by using each of the multiple second frames as the search object, or by using at least one of the multiple second frames as the search object.

[0104] Figure 3 This diagram illustrates the search process performed by the restoration unit 141. Figure 3 The image shows an example of a first frame 531 containing a subject 510 captured by a camera device 301 from a first viewpoint V1, a second frame 532 containing a subject 510 captured by a camera device 301 from a second viewpoint V2, and a second frame 533 containing a subject 510 captured by a camera device 301 from a third viewpoint V3.

[0105] The restoration unit 141 calculates epipolar lines for each first pixel. These epipolar lines are obtained by projecting a straight line connecting the position of the camera device 301 that captured the first frame and the two-dimensional coordinates of that first pixel on the first frame onto the second frame of the object being processed. For example, as Figure 3 As shown, the restoration unit 141 calculates that the straight line L1 connecting the first viewpoint V1 and the first pixel 541 is projected onto the epipolar line 552 on the second frame 532. Furthermore, the restoration unit 141 calculates that the straight line L1 is projected onto the epipolar line 553 on the second frame 533. Additionally, the restoration unit 141 searches for similar points on the epipolar lines 552 and 553 that are similar to the first pixel 541 of the processed object in the first frame 531.

[0106] The restoration unit 141 calculates the similarity between the first pixel and each of the multiple second pixels in the search region. Specifically, the restoration unit 141 uses Equation 1 to calculate N(I, J), which represents the Normalized Cross Correlation (NCC) between small regions, as the similarity between the first frame and the second frame of the processing object. N(I, J) is represented by a value between -1 and 1, with the closer to 1, the higher the similarity.

[0107] [Formula 1]

[0108]

[0109] I xy and J xy This represents the pixel value within a small region of each frame.

[0110] and This represents the average pixel value within a small region of each frame.

[0111] Furthermore, the restoration unit 141 can also calculate SSD(I,J) as the similarity instead of NCC. SSD (Sum of Squared Difference) is the sum of squared differences in pixel values ​​between small regions between the first frame and the second frame of the processed object. The smaller the SSD(I,J) value, the higher the similarity.

[0112] [Formula 2]

[0113]

[0114] I xy and J st This represents the pixel value within a small region of each frame.

[0115] The restoration unit 141 determines the second pixels 542 and 543, which have the highest similarity to the first pixel 541, as similar points to the first pixel. Using the first pixel 541 and the second pixels 542 and 543 selected as similar points to the first pixel 541, the restoration unit 141 calculates a three-dimensional point represented by the three-dimensional coordinates of the point on the subject 510 based on the principle of triangulation.

[0116] In addition, when multiple three-dimensional points are calculated for a first pixel, the restoration unit 141 can also generate a three-dimensional point corresponding to a first pixel by calculating the average of the multiple three-dimensional points.

[0117] Furthermore, if no camera determination information for the camera device 301 that generated each frame is attached to each of the multiple frames, the restoration unit 141 may attach camera determination information for the camera device 301 that generated each frame to the header of each frame, or it may attach multiple camera determination information for multiple camera devices 301 to the header of a frame set containing multiple frames. The camera determination information may include identification information that identifies the camera device 301 corresponding to the camera determination information, or it may include camera parameters of the camera device 301 corresponding to the camera determination information.

[0118] Figure 4 This is a block diagram showing the functional configuration of the correction unit 142.

[0119] like Figure 4 As shown, the correction unit 142 includes a detection unit 142a and a three-dimensional point correction unit 142b.

[0120] The detection unit 142a detects lines consisting of continuous edges that correspond to the outline of the subject 510, at least in the first frame. The line detected in the first frame is an example of the first line. Specifically, the detection unit 142a may also detect edges using a gradient method employing first derivative on the pixel values ​​of each pixel in the first frame. Alternatively, the detection unit 142a may also detect edges using a Laplacian method employing second derivative on the pixel values ​​of each pixel in the first frame. Furthermore, the edges detected by the detection unit 142a are points (edge ​​points) in the first frame.

[0121] Furthermore, the detection unit 142a uses multiple edges detected in the first frame to detect continuous edges. The detection unit 142a can utilize well-known algorithms to detect continuous edges. For example, the detection unit 142a can also use LSD (Line Segment Detector) to detect linear continuous edges. By using LSD, the detection unit 142a detects the start and end points of the continuous edge, and identifies the straight line connecting the start and end points as the continuous edge. However, the algorithm for detecting linear continuous edges is not limited to LSD.

[0122] The contour corresponding to the line detected by the detection unit 142a can be, for example, a line segment, a curve, a plane, or a curved surface. The contour can also be a line or surface that forms part of a specific shape. The specific shape can be a prism, pyramid, cylinder, cone, sphere, etc., or a solid object of revolution formed by rotating a line segment or curve about a certain axis of rotation.

[0123] The profile can also be distinguished according to each type of line shape. The line shape can be a straight line (line segment) connecting two points, or it can be a curve connecting two points. The curve connecting two points can also be a smooth curve without any curved parts. In this way, the line detected by the detection unit 142a can also be a line of finite length divided by two points. Furthermore, in the case of a surface, the profile can also be a surface containing the detected line. If the detected line is a line segment, the profile can also be a plane; if the detected line is a curve, the profile can also be a curved surface.

[0124] Figure 5 and Figure 6 This diagram illustrates an example of the processing of the detection line performed by the detection unit 142a. The following explanation will focus on the case of the detection line segment detected by the detection unit 142a.

[0125] exist Figure 5 In the first frame 531, the region 561 containing the linear shape of the subject 510 is represented by a dashed line. Since the first frame 531 represents the subject 510 in pixels, the outline of the subject 510 is also represented in pixels. Therefore, in the region 561 where a linear shape extending diagonally in the first frame 531 is typically captured, such as... Figure 6 As shown in (a), significant aliasing occurs where the edges appear stepped. However, the detected continuous edges are not represented in pixel units, so the outline of the subject 510 is represented with high reproducibility by the continuous edges. In other words, the position of the detected continuous edges is determined by the first or second derivative of the brightness gradient obtained based on the relationship between the position and pixel value of each pixel, so it is not limited to the position of pixels, but determined with a precision of a region unit smaller than a pixel unit (decimal pixel precision). That is, the detection unit 142a detects the line formed by the continuous edges at a position with decimal pixel precision. The line formed by the continuous edges is represented in units smaller than the pixel units in the first frame 531. Therefore, the continuous edges represent the outline 562 with high reproducibility in a region unit smaller than a pixel unit. Furthermore, since the detection unit 142a uses the first or second derivative of the brightness gradient to detect the continuous edges, it is possible to obtain a result in which the aliasing of multiple pixels is canceled out.

[0126] Therefore, by generating three-dimensional points not only based on the pixels of the first frame 531, but using continuous edges that can represent the outline of the subject 510 with high reproducibility, more accurate three-dimensional points can be generated.

[0127] The 3D point correction unit 142b determines, among a plurality of second 3D points, a group of a plurality of third 3D points that includes the contour corresponding to line 562. The group may also include a plurality of 3D points generated by a plurality of pixels through which line 562 passes in the first frame. Furthermore, the group may also include a plurality of 3D points generated by a plurality of pixels through which line 562 passes in the first frame and the pixels surrounding those pixels. The pixels surrounding those pixels may also include all of the pixels located closer to a predetermined number of pixels separated from those pixels. Thus, the 3D point correction unit 142b determines, for each of the more than one line detected by the detection unit 142a, a group of a plurality of third 3D points defined by the portion corresponding to that line.

[0128] Furthermore, the 3D point correction unit 142b corrects the positions of multiple third 3D points belonging to groups defined by each line. Specifically, the 3D point correction unit 142b determines a plane by performing principal component analysis on multiple third 3D points belonging to each group for each group, and corrects the positions of the multiple third 3D points in a manner that makes the multiple third 3D points approximately oriented towards the plane. In this way, the 3D point correction unit 142b corrects the positions of multiple third 3D points corresponding to the contour among multiple second 3D points. As a result, multiple first 3D points are generated.

[0129] Figure 7 This diagram illustrates an example of the process performed by the three-dimensional point correction unit 142b to correct the positions of multiple third three-dimensional points in accordance with Embodiment 1. Figure 7 (a) represents multiple third-dimensional points 570 belonging to one group.

[0130] The three-dimensional point correction unit 142b performs principal component analysis on multiple third-dimensional points 570 to calculate, for example... Figure 7 The first principal component 571 and the second principal component 572 are shown in (b). Specifically, the three-dimensional point correction unit 142b calculates the variance-covariance matrix of the three-dimensional coordinates of a plurality of third three-dimensional points 570, and calculates two principal components selected sequentially from the larger of the obtained principal components (intrinsic vectors) as the first principal component 571 and the second principal component 572.

[0131] Next, the 3D point correction unit 142b generates a first plane based on the first principal component 571 and a second plane based on the second principal component 572 as candidates (candidate planes) for use in the correction. The first plane is a plane that passes through the centroids of the plurality of third 3D points 570 and has the first principal component 571 as its normal. The second plane is a plane that passes through the centroids of the plurality of third 3D points 570 and has the second principal component 572 as its normal. The 3D point correction unit 142b determines the plane among the first and second planes whose sum of distances from the plurality of third 3D points 570 is smaller as the plane used in the correction. Here, the formula representing the candidate plane is expressed by Equation 3.

[0132] ax + by + cz + d = 0 (Equation 3)

[0133] In this case, the distance D between the plane and the point (x0, y0, z0) is represented by Equation 4.

[0134] [Formula 3]

[0135]

[0136] If we assume that the number of third 3D points belonging to the group is n, and the 3D coordinates of the i-th 3D point are (xi, yi, zi), then the total distance Dall between the n third 3D points and the candidate surface is represented by Equation 5.

[0137] [Formula 4]

[0138]

[0139] The three-dimensional point correction unit 142b calculates Dall for both the first and second planes, and determines the plane with the smallest Dall as the correction plane 573. As a result, the three-dimensional point correction unit 142b... Figure 7 In the example shown in (c), the first plane based on the first principal component 571 is determined as the modified plane 573.

[0140] After determining the correction surface, the three-dimensional point correction unit 142b is as follows: Figure 7 As shown in (d), for multiple third three-dimensional points 570, the intersection of the projection line of the pixel on the first frame corresponding to the third three-dimensional point to the third three-dimensional point and the correction surface 573 is calculated as the correction point 574.

[0141] Here, if we let the three-dimensional unit vector corresponding to each pixel of the third three-dimensional point be (Vx, Vy, Vz), then it can be expressed as the third three-dimensional point corresponding to the pixel (Xn, Yn, Zn) = t(Vx, Vy, Vz).

[0142] If the plane is represented as in Equation 3, then t is represented by Equation 6 below.

[0143] [Formula 5]

[0144]

[0145] Furthermore, the correction point (X'n, Y'n, Z'n) is represented by the following Equation 7.

[0146] [Formula 6]

[0147]

[0148] Furthermore, if we assume it is a pinhole camera, the three-dimensional unit vector (Vx, Vy, Vz) corresponding to the pixel is represented by the following Equation 8.

[0149] [Formula 7]

[0150]

[0151] In Equation 8, (x, y) are the two-dimensional coordinates of the pixel in the first frame. (cx, cy) is the center position of the first frame in the internal parameters of the camera device 301 that captured the first frame. (fx, fy) is the focal distance in the internal parameters of the camera device 301 that captured the first frame.

[0152] Thus, the 3D point correction unit 142b generates a first 3D model containing multiple correction points 574 by correcting the positions of multiple third 3D points 570. Consequently, the multiple correction points 574 are arranged along the line detected in the first frame. Therefore, in the generated first 3D model, when viewed from the viewpoint of the first frame, the portion corresponding to the line detected in the first frame has a shape along that line. Furthermore, since the multiple first 3D points are generated as a result of correcting the multiple third 3D points 570, they include multiple correction points 574. That is, the multiple first 3D points include multiple uncorrected second 3D points and multiple correction points 574.

[0153] [The operation of the 3D model generation device]

[0154] Next, use Figure 8 The operation of the three-dimensional model generation device 100 is explained. Figure 8 This is a flowchart illustrating an example of the operation of the 3D model generation device 100.

[0155] First, in the 3D model generation apparatus 100, the receiving unit 110 receives multiple frames captured by multiple imaging devices 301 and camera parameters of each imaging device 301 from the estimation device 200 (S101). Step S101 is an example of the step of acquiring multiple images. Alternatively, the receiving unit 110 may receive multiple frames and multiple camera parameters at different times, rather than at a single timing. That is, the acquisition of multiple frames and multiple camera parameters can be performed at the same timing or at different timings.

[0156] Next, the storage unit 120 stores the multiple frames captured by the multiple camera devices 301 and the camera parameters of each camera device 301 received by the receiving unit 110 (S102).

[0157] Next, the acquisition unit 130 acquires multiple frames and multiple camera parameters stored in the storage unit 120, and outputs the acquired multiple frames and multiple camera parameters to the generation unit 140 (S103).

[0158] Next, the generation unit 140 generates a first 3D model using multiple frames and multiple camera parameters (S104). Details of step S104, which generates the first 3D model, are provided below. Figure 9 To be discussed later.

[0159] Next, the output unit 150 outputs the first three-dimensional model generated in the generation unit 140 (S105).

[0160] Figure 9 This is a flowchart illustrating a detailed example of the generation process of step S104 performed by the generation unit 140.

[0161] The generation unit 140 performs loop 1 (S111) for each frame set of multi-view images captured at corresponding time intervals. In loop 1, loop 2 is performed for each frame set.

[0162] The generation unit 140 performs loop 2 (S112) on each of the first pixels of the first frame in the frame set of the processing object. In loop 2, each of the multiple first pixels is processed by steps S113 to S116.

[0163] For the first pixel of the processing object, the restoration unit 141 searches for similar points similar to the first pixel from among a plurality of second pixels on the epipolar line corresponding to the first pixel in a plurality of second frames in the frame set of the processing object (S113).

[0164] The restoration unit 141 generates a second three-dimensional point by performing triangulation using the search results obtained in step S113. Thus, the restoration unit 141 generates a second three-dimensional model containing multiple second three-dimensional points (S114). Details of steps S113 and S114 are already described in the description of the restoration unit 141, and are therefore omitted.

[0165] The correction unit 142 detects lines consisting of continuous edges that correspond to the outline of the subject from the first frame (S115).

[0166] The correction unit 142 corrects the positions of multiple third-dimensional points corresponding to the detected line among the multiple second-dimensional points according to the shape of the line (S116). Thus, the correction unit 142 generates a first-dimensional model containing multiple first-dimensional points. Details of steps S115 and S116 have been described in the description of the correction unit 142, and are therefore omitted.

[0167] If the processing of steps S113 to S116 is completed for all the first pixels contained in the first frame of the frame set of the processed object, then loop 2 ends.

[0168] If loop 2 ends for the entire set of frames, then loop 1 ends.

[0169] In addition, Figure 9 In the example, it is assumed that loop 2 includes both the process of generating the second 3D model and the process of correcting the positions of multiple third 3D points, but it is not limited to this; these two processes may not be included in the same loop. That is, the loop for generating the second 3D model and the loop for correcting the positions of multiple third 3D points can also be independent.

[0170] [Effects, etc.]

[0171] The three-dimensional model generation method of this embodiment obtains a first frame obtained by photographing the subject from a first viewpoint and a second frame obtained by photographing the subject from a second viewpoint (S101). At least in the first frame, a line consisting of continuous edges and corresponding to the outline of the subject is detected (S115). Based on the first frame, the second frame, and the line, a plurality of first three-dimensional points representing the outline in the three-dimensional space within the computer are generated (S116).

[0172] According to this 3D model generation method, lines corresponding to the contour of the subject are detected from the frames. In addition to multiple frames, multiple first-dimensional points are generated using the detected lines. Because lines corresponding to the subject's contour are detected in this way, and multiple first-dimensional points are generated considering these lines, errors caused by aliasing in the image can be reduced. Therefore, the generation accuracy of the first-dimensional model can be improved.

[0173] Furthermore, in the three-dimensional model generation method of this embodiment, the contours represented by multiple first three-dimensional points in the three-dimensional space within the computer when viewed from a viewpoint corresponding to the first viewpoint have shapes along lines. That is, in the three-dimensional model generation method, multiple first three-dimensional points are generated in a manner that follows the shape of the contour of the subject when viewed from the first viewpoint, thus improving the generation accuracy of the first three-dimensional model.

[0174] Furthermore, in the three-dimensional model generation method of this embodiment, multiple second three-dimensional points representing the subject in the three-dimensional space of the computer are generated by searching and matching similar points in the first and second frames (S113, S114). In order to generate multiple first three-dimensional points (S116), multiple third three-dimensional points corresponding to the contour are determined based on the lines among the multiple second three-dimensional points, and the positions of the multiple third three-dimensional points are corrected.

[0175] Therefore, multiple third-dimensional points are identified as point groups corresponding to the contour, and the positions of these point groups are corrected using these defined point groups. Thus, the positions of the point groups can be appropriately adjusted according to the shape of the contour.

[0176] Furthermore, in the three-dimensional model generation method of this embodiment, in order to generate multiple first three-dimensional points (S116), a plane is generated by performing principal component analysis on multiple third three-dimensional points, and the positions of the multiple third three-dimensional points are corrected in such a way that the multiple third three-dimensional points are close to the plane. Therefore, the positions of the multiple third three-dimensional points can be corrected with high precision according to the shape of the contour.

[0177] Furthermore, in the three-dimensional model generation method of this embodiment, in order to generate a plurality of first three-dimensional points (S116), a plane with a smaller sum of distances from a first plane and a second plane to a plurality of third three-dimensional points is generated as a plane used in correction. The first plane passes through the centroid of the plurality of third three-dimensional points and has a normal to a first principal component obtained through principal component analysis. The second plane passes through the centroid and has a normal to a second principal component obtained through principal component analysis. Therefore, even if the length of the detected line is insufficient, the positions of the plurality of third three-dimensional points can be corrected with high precision based on the line as a line segment.

[0178] [Variation Example]

[0179] (Variation Example 1)

[0180] In the three-dimensional model generation method of Embodiment 1 described above, the three-dimensional model generation apparatus 100 generates a correction surface by performing principal component analysis on a plurality of third three-dimensional points belonging to a group defined by lines, and uses the correction surface to correct the positions of the plurality of third three-dimensional points, but is not limited thereto. In the three-dimensional model generation method of Modification 1, an approximate line can also be generated by performing least squares on the plurality of third three-dimensional points, and the position of the plurality of third three-dimensional points can be corrected using the approximate line.

[0181] In this case, the processing of the three-dimensional point correction unit 142b in the correction unit 142 is different, so the processing of the three-dimensional point correction unit 142b will be explained.

[0182] Similar to Embodiment 1, the three-dimensional point correction unit 142b determines a group of multiple third-dimensional points that include the portion corresponding to line 562 among the multiple second-dimensional points. Furthermore, the three-dimensional point correction unit 142b generates an approximate line by performing a least-squares method on the multiple third-dimensional points. The three-dimensional point correction unit 142b corrects the positions of the multiple third-dimensional points in a manner that makes them approximate the line.

[0183] Figures 10-13 This is a diagram illustrating an example of the process performed by the three-dimensional point correction unit 142b of a modified embodiment 1 to correct the positions of a plurality of third three-dimensional points. Figure 10 This represents multiple third-dimensional points (580) belonging to one group.

[0184] like Figure 10 As shown, the three-dimensional point correction unit 142b projects multiple third three-dimensional points 580 onto the XZ plane. Let the number of the multiple third three-dimensional points 580 belonging to the group be n, and the coordinates of the i-th three-dimensional point in the XZ plane be (Xi, Zi). Furthermore, assume that Equation 9 represents the formula for the approximate line obtained.

[0185] Z = aX + b (Equation 9)

[0186] At this point, the three-dimensional point correction unit 142b can calculate a and b using the least squares method and the following equation 10.

[0187] [Formula 8]

[0188]

[0189] Therefore, as Figure 11 As shown, when the two-dimensional plane containing multiple third three-dimensional points 580 is designated as the XZ plane, the three-dimensional point correction unit 142b can use the least squares method to generate approximate lines 581 for multiple third three-dimensional points 580 in the XZ plane.

[0190] Next, as Figure 12 As shown, the 3D point correction unit 142b calculates the intersection of the straight line connecting the viewpoint at the time of capturing the first frame and the approximate line 581 with the i-th third 3D point as the correction point. Specifically, the 3D point correction unit 142b calculates the intersection of the straight line connecting the viewpoint at the time of capturing the first frame and the approximate line 581 with the approximate line 581 ... with the approximate line 581. i Y' i Z' i When the straight line extending from the center of the camera of the imaging device 301 at the time of shooting the first frame is set to Z = cX in the XZ coordinate system, the intersection point (X') of the straight line and the approximate line is... i Z' i It is represented by Equation 11.

[0191] [Formula 9]

[0192]

[0193] Thus, the three-dimensional point correction unit 142b calculates the intersection point of each third three-dimensional point with the approximate line 581. Using the calculated coordinates of the intersection points and Equation 12, the three-dimensional point correction unit 142b calculates the Y coordinate of the correction point in the XYZ coordinate system.

[0194] Y' i =n x X' i +n z Z' i (Equation 12)

[0195] Here, the straight line extending from the camera center of the imaging device 301 after the correction point is represented by Equation 13.

[0196] [Formula 10]

[0197]

[0198] Without the width of a straight line, multiple third-dimensional points will exist on the same plane. Here, the three-dimensional unit vector corresponding to the pixel of the i-th third-dimensional point is represented by Equation 14.

[0199] [Formula 11]

[0200]

[0201] It is the i-th three-dimensional unit vector.

[0202] In this case, such as Figure 13 As shown, the three-dimensional point correction unit 142b samples two points from a plurality of third-dimensional points to calculate plane 582. The normal to plane 582 is represented by Equation 15, and the plane is represented by Equation 16.

[0203] [Formula 12]

[0204]

[0205] It is the normal to the plane.

[0206] n x x+n y y+n z z = 0 (Equation 16)

[0207] In addition, as shown in Equation 17, the normal can be obtained by the cross product of vectors.

[0208] [Formula 13]

[0209]

[0210] It is the normal to the plane.

[0211] Thus, the 3D point correction unit 142b generates a first 3D model containing multiple correction points 583 by correcting the positions of multiple third 3D points 580. Consequently, the multiple correction points 583 are arranged along the lines detected in the first frame. Therefore, in the generated first 3D model, the contours of the multiple first 3D points in the computer's 3D space when viewed from the starting point corresponding to the first viewpoint have a shape along a line. Furthermore, since the multiple first 3D points are generated as a result of correcting the multiple third 3D points 580, they include multiple correction points 583. That is, the multiple first 3D points include multiple uncorrected second 3D points and multiple correction points 583.

[0212] Furthermore, in the method for generating the three-dimensional model of the modified example, in order to generate the first three-dimensional model (S116), an approximate line is generated by performing the least squares method on multiple third three-dimensional points, and the positions of the multiple third three-dimensional points are corrected in such a way that the multiple third three-dimensional points are close to the approximate line. Therefore, the positions of the multiple third three-dimensional points can be corrected with high precision based on the approximate line that approximates the group of points.

[0213] (Variation Example 2)

[0214] The modified example 2 will be explained.

[0215] In the three-dimensional model generation method of Modification Example 2, the three-dimensional model generation apparatus 100 can also generate a surface for correction using multiple corrected third three-dimensional points. In this case, the processing of the correction unit 142 is different, so the processing of the correction unit 142 will be described.

[0216] Figure 14 This is a diagram illustrating an example of the processing performed by the correction unit 142 in a variation 2 of embodiment 1. Figure 15 This is a diagram illustrating an example of the processing performed by the correction unit 142 in a variation 2 of embodiment 1.

[0217] In the first frame 591, the detection unit 142a of the correction unit 142 detects a line 601, which is composed of continuous edges and corresponds to the outline of the subject. Then, in the second frame 592, the detection unit 142a detects a line 602, which is composed of continuous edges and corresponds to the outline of the subject. The line 602 corresponds to an outline that includes at least a portion of the outline of the subject corresponding to the line 601. That is, the outline corresponding to the line 602 and the outline corresponding to the line 601 contain overlapping portions. In addition, the line 601 corresponds to a first line, and the line 602 corresponds to a second line.

[0218] Specifically, in the second frame 592, the detection unit 142a detects lines corresponding to the contour of the subject, compares the detected lines with line 601, and detects the lines whose consistency with line 601 is higher than a predetermined consistency as line 602. The consistency between the detected line and line 601 can be, for example, the similarity between the normal vector of the detected line and the normal vector of line 601. Furthermore, in addition to being based on the similarity between normal vectors, the consistency can also be calculated based on the distance between the position of the detected line in three-dimensional space and the position of line 601 in three-dimensional space. In this case, the consistency is calculated such that the shorter the distance, the greater the consistency. This similarity can be calculated using well-known methods.

[0219] The three-dimensional point correction unit 142b determines a first group among a plurality of second three-dimensional points that includes a plurality of third three-dimensional points 610 contained in the contour corresponding to the line 601. Thus, for each line 601 detected by the detection unit 142a in the first frame 591, the three-dimensional point correction unit 142b determines a first group that includes a plurality of third three-dimensional points 610 determined corresponding to a portion of that line 601.

[0220] Next, the 3D point correction unit 142b corrects the positions of multiple third 3D points 610 belonging to the first group determined by each line 601. Specifically, for each first group, the 3D point correction unit 142b determines a plane 621 by performing principal component analysis on the multiple third 3D points 601 belonging to each first group, and corrects the positions of the multiple third 3D points 610 in such a way that the multiple third 3D points 601 are close to the plane 621. In this way, the 3D point correction unit 142b corrects the positions of the multiple third 3D points 610 corresponding to the contour among the multiple second 3D points. As a result, multiple first 3D points 611 are generated.

[0221] Next, the 3D point correction unit 142b determines, among the plurality of second 3D points, a plurality of fourth 3D points 613 that correspond to the contour of line 602. The plurality of fourth 3D points 613 include the corrected plurality of third 3D points 611 and the uncorrected second 3D points 612. Thus, for one or more lines 602 detected by the detection unit 142a in the second frame 592, the 3D point correction unit 142b determines a second group that includes the plurality of fourth 3D points 613 determined from the portion corresponding to the line 602.

[0222] Next, the three-dimensional point correction unit 142b corrects the positions of the plurality of fourth three-dimensional points 613 belonging to the second group determined by each line 602. Specifically, the three-dimensional point correction unit 142b corrects the positions of at least the uncorrected second three-dimensional points 612 among the plurality of fourth three-dimensional points 613. For each second group, the three-dimensional point correction unit 142b determines a plane 622 by performing principal component analysis on the plurality of fourth three-dimensional points 613 belonging to each second group, and corrects the positions of the plurality of fourth three-dimensional points 613 in a manner that the plurality of fourth three-dimensional points 613 are close to the determined plane 622. At this time, the three-dimensional point correction unit 142b can either correct the positions of all the plurality of fourth three-dimensional points 613, or only correct the positions of the uncorrected second three-dimensional points 612 among the plurality of fourth three-dimensional points 613.

[0223] If the distance D1 from the first viewpoint V11 to the outline of the subject corresponding to the line 601 detected in the first frame 591 is shorter than the distance D2 from the second viewpoint V12 to the outline of the subject corresponding to the line 602 detected in the second frame 592, the three-dimensional point correction unit 142b may use a plane located on the extension line of the plane 621 determined for correcting the plurality of third three-dimensional points 610 to correct at least the uncorrected second three-dimensional points 612 among the plurality of fourth three-dimensional points 613.

[0224] When a continuous shape of a subject is captured not within a single frame but across multiple frames, generating multiple first 3D points based on the results of each frame's capture may result in multiple first 3D points that fail to represent a continuous shape, even if it is. According to the 3D model generation method of Modification 2, multiple fourth 3D points 613 containing corrected third 3D points 610 are used to correct at least the positions of uncorrected second 3D points 612 among the multiple fourth 3D points 613. That is, considering a continuous shape, the corrected third 3D points 610 are used to correct at least the uncorrected second 3D points 612, thus enabling the generation of multiple first 3D points corresponding to the continuous contour of the subject with high accuracy.

[0225] (Implementation Method 2)

[0226] The three-dimensional model generation apparatus 100A of Embodiment 2 will be described.

[0227] Figure 16 This is a block diagram showing the characteristic structure of the three-dimensional model generation apparatus 100A according to Embodiment 2.

[0228] The structure of the generation unit 140A differs from that of the three-dimensional model generation apparatus 100A in Embodiment 2 compared to that in Embodiment 1. Therefore, the generation unit 140A will be described. Furthermore, the constituent elements of the three-dimensional model generation apparatus 100A other than the generation unit 140A are the same as those of the three-dimensional model generation apparatus 100 in Embodiment 1, so they are given the same reference numerals and their descriptions are omitted.

[0229] Specifically, generation unit 140A differs from generation unit 140 in that it generates the first 3D model using the corrected multiple frames after correcting them. Generation unit 140A has a correction unit 141A and a restoration unit 142A. Correction unit 141A detects lines consisting of continuous edges that correspond to the outline of the subject from each of the multiple frames. That is, correction unit 141A detects a first line consisting of continuous edges in the first frame. Furthermore, correction unit 141A detects a second line consisting of continuous edges in the second frame. This process is the same as the line detection process from frames by correction unit 142 in embodiment 1. Next, correction unit 141A corrects each of the multiple frames using the lines detected in each of the multiple frames. That is, correction unit 141A corrects the first frame and the second frame respectively using the first line and the second line.

[0230] Furthermore, the line detected by the correction unit 141A can be either a straight line (line segment) or a curve. The detected line is not represented by multiple pixels. The detected line is determined by the position with a fraction of a pixel precision. If the detected line is a line segment, it is a line segment connecting the start and end points; if it is a curve, it is a smooth line connecting the start and end points. The correction unit 141A can use a known algorithm to detect lines composed of continuous edges.

[0231] Figure 17 This diagram illustrates an example of the process performed by the correction unit 141A to correct each frame in relation to Embodiment 2. Figure 17 Figure (a) is a diagram showing another example of region 561 in Embodiment 1. In region 561 of Embodiment 2, pixels are missing due to significant aliasing.

[0232] The correction unit 141A performs the same processing as the detection unit 142a in embodiment 1 in region 561, such as... Figure 17The detection line 562 is shown in (b). The correction unit 141A calculates the average value of the pixel group of multiple pixels traversed by the line 562 as the corrected pixel value. If the value of the i-th pixel in the pixel group is I... i Let the corrected pixel value be I' i If the number of pixels in a pixel group is n, then the correction unit 141A can calculate the corrected pixel value I' as shown in Equation 18. i .

[0233] [Formula 14]

[0234]

[0235] Furthermore, the correction unit 141A modifies all the pixel values ​​of the pixel group of the multiple pixels through which the line 562 passes by using the corrected pixel value I'. i Replacement, such as by Figure 18 Region 561 is corrected as shown in region 590 (a). Additionally, Figure 18 This is a diagram illustrating an example of the processing for correcting each frame in Implementation 2.

[0236] Furthermore, the correction unit 141A may not use the pixel values ​​of all pixels in the pixel group for correction; instead, it may sample more than one pixel and use the pixel values ​​of the sampled more than one pixel for correction. For example, the frame may be corrected by replacing all pixel values ​​of the pixel group of multiple pixels traversed by line 562 with the average of the pixel values ​​of the sampled more than one pixel. Additionally, the correction unit 141A may use line 562 to connect pixel groups as follows... Figure 18 (b) shows the process of resolving the resolution.

[0237] The restoration unit 142A searches for matching points of similarity among the frames corrected by the correction unit 141A, i.e., the multiple corrected frames, i.e., the first corrected frame and the second corrected frame. The restoration unit 142A performs the same processing as that performed by the restoration unit 141 in Embodiment 1.

[0238] Next, the operation of the 3D model generation apparatus 100A will be explained. Embodiment 1 of the 3D model generation apparatus 100A Figure 8 The details of step S104 in the flowchart are different, so the details of step S104 will be explained.

[0239] Figure 19 This is a flowchart illustrating a detailed example of the generation process of step S104 performed by the generation unit 140A in Embodiment 2.

[0240] The generation unit 140A performs loop 11 (S121) for each frame of the frame set of multi-viewpoint images captured at corresponding time intervals. In loop 11, steps S122 and S123 are performed for each frame.

[0241] The correction unit 141A detects the lines of the subject in the frame by detecting lines composed of continuous edges from the frame of the processing object (S122).

[0242] The correction unit 141A uses the lines detected in the frame to correct the frame of the processing object (S123).

[0243] If steps S122 and S123 are completed for all frames, then loop 11 ends.

[0244] Next, the restoration unit 142A takes the corrected multiple frames as the processing object and performs loop 12 (S124) on each of the multiple first pixels of the first frame in the corrected multiple frames. In loop 12, the processing of steps S125 and S126 is performed on each first pixel.

[0245] For the first pixel of the processing object, the restoration unit 142A searches for similar points similar to the first pixel from among a plurality of second pixels on the epipolar line corresponding to the first pixel in a plurality of second frames in the frame set of the processing object (S125).

[0246] The restoration unit 142A performs triangulation using the search results obtained in step S125 to generate a first three-dimensional point. Thus, the restoration unit 142A generates a first three-dimensional model containing multiple first three-dimensional points (S126).

[0247] If the processing of steps S125 and S126 is completed for all the first pixels contained in the first frame of the frame set of the processing object, then loop 12 ends.

[0248] According to the three-dimensional model generation apparatus 100A of Embodiment 2, a first three-dimensional model is generated using an image corrected by lines, thus enabling the generation of a first three-dimensional model with reduced errors caused by aliasing in the image.

[0249] In the three-dimensional model generation apparatus 100A of embodiment 2, multiple frames obtained by photographing the subject from multiple viewpoints are also acquired (S101), a line consisting of continuous edges and corresponding to the outline of the subject is detected from one of the multiple frames (first frame) (S122), and multiple first three-dimensional points are generated based on the multiple frames and the line (S126).

[0250] (Other implementation methods)

[0251] In embodiments 1 and 2 described above, an example of generating the first three-dimensional model using lines corresponding to the outline of the subject was given. However, it is also possible not to use all lines to generate the first three-dimensional model. For example, the three-dimensional model generation apparatus 100 and 100A removes three-dimensional points located at a distance closer than a predetermined distance from the viewpoint where one frame was captured from a plurality of first three-dimensional points.

[0252] For example, regarding the 3D model generation apparatus 100 of Embodiment 1, if, based on detected lines, multiple third 3D points corresponding to the contours of the lines are located closer to the viewpoint than a predetermined distance, the correction unit 142 may not use these lines to correct the positions of the multiple third 3D points. That is, the correction unit 142 may also not correct the positions of the second 3D points among the generated multiple second 3D points that are located closer to the viewpoint than a predetermined distance. In other words, the 3D model generation apparatus 100 may switch between correcting or not correcting each line based on the distance from the viewpoint where one frame was captured to the contour of the subject corresponding to that line.

[0253] In this way, lines are not used to generate the first three-dimensional model in parts where the accuracy is likely to be above a certain level, thus reducing the processing load without causing a decrease in accuracy.

[0254] Furthermore, for example, in the case of the 3D model generation apparatus 100A of Embodiment 2, if the distance of the subject captured in each frame from the viewpoint can be obtained, it is not necessary to use a line corresponding to a contour located at a position closer to the viewpoint than a predetermined distance to correct the pixel value of the pixel group containing that contour. For example, the distance of the subject from the viewpoint can also be estimated using the 3D model 520 obtained through the estimation processing of the estimation device 200. That is, the 3D model generation apparatus 100 can also switch whether to correct the portion of the image corresponding to the contour based on the distance from the viewpoint where one frame was captured to the contour corresponding to that line.

[0255] Furthermore, in embodiments 1 and 2 described above, an example was given of detecting lines for multiple frames and generating a first three-dimensional model using the detected lines. However, it is also possible to generate the first three-dimensional model without using lines obtained from all frames. For example, the three-dimensional model generation apparatus 100 and 100A may also generate multiple first three-dimensional points without using edges detected from frames with a specified resolution or higher among the multiple frames.

[0256] For example, the 3D model generation devices 100 and 100A may not perform line detection processing on frames with a resolution higher than a specified resolution. This is because the accuracy of the first 3D model generated using frames with a resolution higher than the specified resolution is a certain level of high accuracy. That is, the 3D model generation devices 100 and 100A can also switch whether to perform line detection on a frame based on the frame's resolution.

[0257] In this way, in images where the accuracy is likely to be above a certain level, lines are not used to generate multiple first-dimensional points, thus reducing the processing load without causing a decrease in accuracy.

[0258] The above description describes the three-dimensional model generation method of this disclosure based on the above embodiments, but this disclosure is not limited to the above embodiments.

[0259] For example, in the above embodiments, it is assumed that each processing unit of the 3D model generation apparatus, etc., is implemented by a CPU and a control program. For example, the components of this processing unit may each consist of one or more electronic circuits. The one or more electronic circuits may be general-purpose circuits or special-purpose circuits. The one or more electronic circuits may also include, for example, semiconductor devices, ICs (Integrated Circuits), or LSIs (Large Scale Integration). ICs or LSIs may be integrated into one chip or multiple chips. Here they are referred to as ICs or LSIs, but the terminology varies depending on the degree of integration; they may also be called system LSIs, VLSIs (Very Large Scale Integration), or ULSIs (Ultra Large Scale Integration). Furthermore, FPGAs (Field Programmable Gate Arrays) programmed after the LSI is manufactured can also be used for the same purpose.

[0260] Furthermore, the general or specific form of this disclosure can also be implemented by a system, apparatus, method, integrated circuit, or computer program. Alternatively, it can be implemented by a computer-readable, non-transitory recording medium such as an optical disc, HDD (Hard Disk Drive), or semiconductor memory storing the computer program. Furthermore, it can be implemented by any combination of system, apparatus, method, integrated circuit, computer program, and recording medium.

[0261] In addition, this disclosure also includes forms obtained by applying various modifications to the embodiments as conceived by those skilled in the art, or forms achieved by arbitrarily combining the constituent elements and functions of the embodiments without departing from the spirit of this disclosure.

[0262] Industrial applicability

[0263] This disclosure can be applied to 3D model generation devices or 3D model generation systems, such as graphic creation, terrain or building structure recognition, human behavior recognition, or free viewpoint image generation.

[0264] Label Explanation

[0265] 100, 100A 3D model generation device

[0266] 110 Receiving Department

[0267] 120 Storage Department

[0268] 130 obtained department

[0269] 140, 140A Production Section

[0270] 141, 142A Restoration Section

[0271] 142, 141A Correction Section

[0272] 142a Testing Department

[0273] 142b 3D Point Correction Unit

[0274] 150 Output Unit

[0275] 200 Speculation Device

[0276] 300 camera devices

[0277] 301 camera device

[0278] 400 3D Model Generation System

[0279] 500, 510 subjects

[0280] 520 3D Model

[0281] 531 Frame 1

[0282] 532, 533, Frame 2

[0283] 541 First pixel

[0284] Pixels 542 and 543 (2nd pixel)

[0285] 552, 553 polar lines

[0286] Areas 561 and 590

[0287] 562 line

[0288] 570, 580, third three-dimensional point

[0289] 571 1st principal component

[0290] 572 2nd principal component

[0291] 573 Correction face

[0292] Correction points 574 and 583

[0293] 581 approximation line

[0294] 582 plane

[0295] L1 straight line

[0296] Distance between D1 and D2

[0297] V1, V11 First Viewpoint

[0298] V2, V12 Second Viewpoint

[0299] V3 Third Viewpoint

Claims

1. A method for generating a three-dimensional model, executed by a computer, wherein, A first image is obtained by photographing the subject from a first viewpoint, and a second image is obtained by photographing the subject from a second viewpoint. At least in the first image above, a first line consisting of continuous edges and corresponding to the outline of the subject is detected; Based on the first image, the second image, and the first line, a plurality of first three-dimensional points representing the contour in the three-dimensional space within the computer are generated. further, By matching similar points in the first and second images, multiple second three-dimensional points representing the subject in the three-dimensional space are generated. In order to generate the above multiple first-dimensional points, (i) Based on the first line mentioned above, determine a plurality of third three-dimensional points corresponding to the above contour among the plurality of second three-dimensional points; (ii) Correct the positions of the aforementioned multiple third-dimensional points. In order to generate the aforementioned plurality of first three-dimensional points, a plane is determined by performing principal component analysis on the aforementioned plurality of third three-dimensional points, and the positions of the aforementioned plurality of third three-dimensional points are corrected in such a way that the aforementioned plurality of third three-dimensional points are close to the aforementioned plane.

2. The three-dimensional model generation method as described in claim 1, wherein, The first line mentioned above is represented in units smaller than the pixel units in the first image mentioned above.

3. The three-dimensional model generation method as described in claim 1, wherein, further, In the second image described above, a second line is detected that is composed of continuous edges and corresponds to a contour that includes at least a portion of the contour corresponding to the first line described above. To generate the aforementioned multiple first-dimensional points, further, (i) Based on the second line mentioned above, a plurality of fourth three-dimensional points are determined among the plurality of second three-dimensional points. The plurality of fourth three-dimensional points correspond to the contour corresponding to the second line mentioned above, and include the modified plurality of third three-dimensional points and the unmodified second three-dimensional points. (ii) Among the above-mentioned multiple fourth three-dimensional points, at least the positions of the above-mentioned uncorrected second three-dimensional points shall be corrected.

4. The three-dimensional model generation method as described in claim 1, wherein, In order to generate the aforementioned plurality of first three-dimensional points, a first plane and a second plane are generated, wherein the plane with the smaller sum of distances from the aforementioned plurality of third three-dimensional points is generated as the aforementioned plane. The first plane passes through the centroid of the aforementioned plurality of third three-dimensional points and takes the first principal component obtained through the aforementioned principal component analysis as its normal. The second plane passes through the aforementioned centroid and takes the second principal component obtained through the aforementioned principal component analysis as its normal.

5. The three-dimensional model generation method according to any one of claims 1 to 4, wherein, In order to generate the plurality of first three-dimensional points, three-dimensional points located at a distance closer than a specified distance from the first viewpoint are removed from the plurality of first three-dimensional points.

6. The three-dimensional model generation method according to any one of claims 1 to 4, wherein, In order to generate the aforementioned plurality of first three-dimensional points, edges detected from images with a specified resolution or higher from the aforementioned first image and the aforementioned second image are not used.

7. The three-dimensional model generation method as described in claim 1 or 2, wherein, In the above detection, a second line consisting of continuous edges and corresponding to the outline of the subject is detected in the second image. In the generation of the above-mentioned multiple first three-dimensional points, (i) the above-mentioned first line and the above-mentioned second line are used to correct the above-mentioned first image and the above-mentioned second image respectively, and (ii) the matching of similar points between them is performed in the corrected above-mentioned first image and the corrected above-mentioned second image.

8. A method for generating a three-dimensional model, executed by a computer, wherein, A first image is obtained by photographing the subject from a first viewpoint, and a second image is obtained by photographing the subject from a second viewpoint. At least in the first image above, a first line consisting of continuous edges and corresponding to the outline of the subject is detected; Based on the first image, the second image, and the first line, a plurality of first three-dimensional points representing the contour in the three-dimensional space within the computer are generated. further, By matching similar points in the first and second images, multiple second three-dimensional points representing the subject in the three-dimensional space are generated. In order to generate the above multiple first-dimensional points, (i) Based on the first line mentioned above, determine a plurality of third three-dimensional points corresponding to the above contour among the plurality of second three-dimensional points; (ii) Correct the positions of the aforementioned multiple third-dimensional points. In order to generate the first three-dimensional point, an approximate line is generated by performing the least squares method on the multiple third three-dimensional points, and the positions of the multiple third three-dimensional points are corrected in such a way that the multiple third three-dimensional points are close to the approximate line.

9. A three-dimensional model generation device, wherein, Possess a computer; The computer obtains a first image by photographing the subject from a first viewpoint and a second image by photographing the subject from a second viewpoint; The computer detects, at least in the first image, a first line consisting of continuous edges and corresponding to the outline of the subject; Based on the first image, the second image, and the first line, the computer generates a plurality of first three-dimensional points representing the contour in a three-dimensional space within the computer. further, The computer generates multiple second three-dimensional points representing the subject in the three-dimensional space by matching similar points in the first and second images. In order to generate the aforementioned multiple first-dimensional points, the computer described above... (i) Based on the first line mentioned above, determine a plurality of third three-dimensional points corresponding to the above contour among the plurality of second three-dimensional points; (ii) Correct the positions of the aforementioned multiple third-dimensional points. In order to generate the plurality of first three-dimensional points, the computer determines a plane by performing principal component analysis on the plurality of third three-dimensional points, and corrects the positions of the plurality of third three-dimensional points in such a way that the plurality of third three-dimensional points are close to the plane.

10. A three-dimensional model generation device, wherein, Possess a computer; The computer obtains a first image by photographing the subject from a first viewpoint and a second image by photographing the subject from a second viewpoint; The computer detects, at least in the first image, a first line consisting of continuous edges and corresponding to the outline of the subject; Based on the first image, the second image, and the first line, the computer generates a plurality of first three-dimensional points representing the contour in a three-dimensional space within the computer. further, The computer generates multiple second three-dimensional points representing the subject in the three-dimensional space by matching similar points in the first and second images. In order to generate the aforementioned multiple first-dimensional points, the computer described above... (i) Based on the first line mentioned above, determine a plurality of third three-dimensional points corresponding to the above contour among the plurality of second three-dimensional points; (ii) Correct the positions of the aforementioned multiple third-dimensional points. In order to generate the first three-dimensional point, the computer generates an approximate line by performing the least squares method on the multiple third three-dimensional points, and corrects the position of the multiple third three-dimensional points in such a way that the multiple third three-dimensional points are close to the approximate line.

Images