Three-dimensional model generation method and three-dimensional model generation apparatus

Abstract

A three-dimensional model generation method comprising: detecting first lines respectively from a plurality of images of a subject region captured from a plurality of viewpoints (S121), excluding a second line that reaches the end of the image from the plurality of detected first lines (S122), and using a plurality of line segments, which are the plurality of first lines after the second line has been excluded, to generate a line segment model, which is a three-dimensional model of the subject region expressed by lines (S123). For example, in this three-dimensional model generation method a third line that is among the plurality of first lines and that reaches the end of the image may be divided into a second line reaching the end of the image and a line segment that does not reach the end of the image, and the second line may be excluded from the first lines.

Description

Technical Field

[0001] The present disclosure relates to a three-dimensional model generation method and a three-dimensional model generation apparatus.

Background Art

[0002] Patent Document 1 discloses a technique for generating a three-dimensional model of a subject using a plurality of images obtained by photographing the subject from a plurality of viewpoints.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the generation process of a three-dimensional model, it is desired to improve the accuracy of the three-dimensional model. An object of the present disclosure is to provide a three-dimensional model generation method or a three-dimensional model generation apparatus capable of improving the accuracy of a three-dimensional model.

Means for Solving the Problems

[0005] A three-dimensional model generation method according to an aspect of the present disclosure includes: By the processor, detecting a first line from each of a plurality of images obtained by photographing a target region from a plurality of viewpoints; By the processor, excluding a second line that reaches the image edge from the plurality of detected first lines; By the processor, generating a line segment model, which is a three-dimensional model of the target region represented by lines, using the plurality of line segments that are the plurality of first lines after excluding the second line.

Effects of the Invention

[0006] The present disclosure can provide a three-dimensional model generation method or a three-dimensional model generation apparatus capable of improving the accuracy of a three-dimensional model.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0008] The three-dimensional model generation method according to one aspect of the present disclosure detects a first line from each of a plurality of images obtained by photographing a target region from a plurality of viewpoints, excludes a second line that reaches the image edge from the plurality of detected first lines, and uses a plurality of line segments that are the plurality of first lines after excluding the second line to generate a line segment model that is a three-dimensional model of the target region represented by lines.

[0009] According to this, the accuracy of the three-dimensional model can be improved.

[0010] For example, among the plurality of first lines, a third line that reaches the image edge may be divided into the second line that reaches the image edge and a line segment that does not reach the image edge, and the second line may be excluded from the plurality of first lines.

[0011] According to this, since a part of the line segment that reaches the image edge can be used for generating the three-dimensional model, the accuracy of the three-dimensional model can be improved.

[0012] For example, the third line may be divided at a position where the third line overlaps with another line to identify the second line that reaches the image edge and the line segment that does not reach the image edge.

[0013] According to this, the line segment can be appropriately divided.

[0014] For example, in the detection of the first line, the first line is detected by a search process using a search window on the image, and the search window may be a rectangle whose long side is along the direction of the first line.

[0015] According to this, the detection accuracy of the line segment can be improved, so the accuracy of the three-dimensional model generated using the line segment can be improved.

[0016] For example, the rectangle may be arranged such that the long side is along the direction of the first line and along a direction other than the vertical and horizontal directions of the image.

[0017] According to this, the detection accuracy of the line segment can be improved, and thus the accuracy of the three-dimensional model generated using the line segment can be improved.

[0018] Further, a three-dimensional model generation device according to an aspect of the present disclosure includes a processor and a memory. The processor uses the memory to detect a first line from each of a plurality of images obtained by photographing a target area from a plurality of viewpoints, excludes a second line that reaches the image edge from the plurality of detected first lines, and generates a line segment model, which is a three-dimensional model of the target area represented by lines, using the plurality of line segments that are the first lines after excluding the second line.

[0019] According to this, the accuracy of the three-dimensional model can be improved. Note that it is not essential to exclude the second line, and it is sufficient to identify the line segments.

[0020] These general or specific aspects may be implemented by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0021] Hereinafter, embodiments will be specifically described with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Among the components in the following embodiments, the components not described in the independent claims are described as optional components.

[0022] (Embodiment) First, the configuration of the three-dimensional model generation device according to the present embodiment will be described. FIG. 1 is a block diagram of a three-dimensional model generation device 100 according to the present embodiment. The three-dimensional model generation device 100 generates a line segment model, which is a three-dimensional model represented by line segments.

[0023] Note that a three-dimensional model is a representation on a computer of an imaged target area (object or target space) or the like. The three-dimensional model has, for example, position information of each three-dimensional location on the measurement target. Also, the line segment model is not limited to line segments and may be a three-dimensional model represented by lines. Here, a line includes a line segment with two ends, a ray with only one end, and a straight line without ends. Also, having no ends means, for example, being cut off in the image (reaching both ends of the image).

[0024] The three-dimensional model generation device 100 includes an image acquisition unit 101, a preprocessing unit 102, a line segment detection unit 103, and a line segment reconstruction unit 104.

[0025] FIG. 2 is a flowchart of the three-dimensional model generation process by the three-dimensional model generation device 100. First, the image acquisition unit 101 acquires a plurality of images taken by an imaging device such as a camera (S101). Here, the plurality of images are two or more images taken of the same target area (object or target space) from different viewpoints. Each of the plurality of images may be a still image or may be composed of a moving image. Note that the image acquisition unit 101 may acquire a plurality of images from an external device via a network or the like. Also, when the three-dimensional model generation device 100 is included in a terminal device or the like having a built-in camera, the image acquisition unit 101 may acquire a plurality of images taken by the camera.

[0026] Also, the plurality of images may be a plurality of images taken by a plurality of fixed cameras. Also, the plurality of images may be two-viewpoint images taken by a stereo camera from one position. Also, the plurality of images may be a plurality of frames included in a moving image taken while a single camera is moving. Also, the plurality of images may be a combination of these.

[0027] Next, the preprocessing unit 102 executes preprocessing for image preprocessing, or line segment reconstruction (line segment matching), or a combination thereof (S102). Image preprocessing includes, for example, brightness adjustment, noise removal, resolution conversion, color space conversion, lens distortion correction, projective conversion, affine conversion, edge enhancement processing, trimming processing, or a combination thereof.

[0028] Note that the timing at which the preprocessing (S102) is executed may be any timing as long as it is before the line segment detection processing (S103). For example, the preprocessing (S102) may be performed in conjunction with the timing at which the line segment detection processing (S103) is performed, or the preprocessing (S102) may be performed in advance, and a plurality of images after preprocessing may be stored in the storage unit provided in the three-dimensional model generation device 100. Note that the preprocessing does not necessarily have to be performed. That is, the three-dimensional model generation device 100 does not necessarily have to include the preprocessing unit 102.

[0029] Next, the line segment detection unit 103 detects line segments from each of the plurality of images after preprocessing (S103). The details of this processing will be described later.

[0030] Next, the line segment reconstruction unit 104 performs line segment reconstruction to generate a line segment model, which is a three-dimensional model, using the detected plurality of line segments (S104). Specifically, the line segment reconstruction unit 104 calculates the feature amounts of the detected line segments. Next, the line segment reconstruction unit 104 performs line segment matching between images using the calculated feature amounts. That is, the line segment reconstruction unit 104 detects corresponding line segments that are corresponding (identical) line segments between images. Next, the line segment reconstruction unit 104 estimates the camera parameters (three-dimensional position and orientation) of each image and the three-dimensional position of the line segments by geometric calculation using the relationship of the corresponding line segments. Thereby, a line segment model is generated. Note that the above-described line segment reconstruction method is an example, and any known method may be used.

[0031] Hereinafter, the line segment detection processing (S103) will be described. FIG. 3 is a flowchart of the line segment detection processing (S103).

[0032] First, the line segment detection unit 103 receives a plurality of pre-processed images and performs region division on each image (S111). FIGS. 4 to 9 are diagrams for explaining this region division process. The figure is an enlarged view of a part of the image, and each rectangle represents a pixel.

[0033] First, as shown in FIG. 4, the line segment detection unit 103 detects the luminance gradient of each pixel. In the figure, for pixels having a luminance gradient (pixels whose luminance gradient magnitude is equal to or greater than a threshold value), the direction of the luminance gradient is indicated in the direction of the arrow. Next, the line segment detection unit 103 assigns pixels with similar luminance gradient directions to the same region.

[0034] Specifically, as shown in FIG. 5, the line segment detection unit 103 sets a target pixel 111 from the pixels having a luminance gradient. Next, the line segment detection unit 103 sets, for example, a search window 112 centered on the target pixel. Next, the line segment detection unit 103 detects, from a plurality of pixels within the search window 112, pixels having a luminance gradient direction close to the luminance gradient direction of the target pixel 111. For example, the line segment detection unit 103 detects pixels whose luminance gradient angle difference from the luminance gradient of the target pixel 111 is equal to or less than a predetermined threshold value. As a result, for example, the pixels indicated by the hatching in FIG. 6 are detected as pixels in the same region as the target pixel 111.

[0035] Next, as shown in FIGS. 7 and 8, the line segment detection unit 103 moves the target pixel 111 and the search window 112 and repeats the same process. For example, the line segment detection unit 103 sets, as a new target pixel 111, a pixel that has been detected as being included in the same region as the previous target pixel 111 and has not been set as the target pixel 111 in the past. Also, the line segment detection unit 103 moves the center of the search window 112 to the position of the new target pixel 111. The line segment detection unit 103 repeats this process until no new pixels can be detected, thereby detecting a region composed of a plurality of pixels having the same luminance gradient as the first target pixel 111. Also, by repeatedly performing this series of processes with pixels not assigned to a region as new first target pixels, a plurality of regions are detected. Note that the first target pixel 111 may be set randomly or may be a pixel at the edge of the image.

[0036] When detecting a pixel having a luminance gradient direction close to the luminance gradient direction of the target pixel 111, the reference luminance gradient used as a criterion may be the luminance gradient of the first target pixel 111, or the luminance gradient of the current target pixel 111, or a value calculated from the luminance gradients of a plurality of pixels already assigned to that area (for example, an average value or a weighted average value), etc. Further, the search method described here is an example, and methods other than the above may be used.

[0037] Here, in the present embodiment, the shape of the search window 112 is rectangular, and the length in the horizontal direction and the length in the vertical direction in the image are different. Here, an example in the case where the size of the search window 112 is 3×7 pixels is shown, but the size of the search window 112 may be any size. Further, the line segment detection unit 103 arranges the search window 112 so that the long side of the search window 112 is in a direction perpendicular to the luminance gradient. For example, in the example of FIG. 5, since the direction of the luminance gradient is along the vertical direction of the image, the search window 112 is arranged so that the long side of the search window 112 is along the horizontal direction of the image. Here, the direction perpendicular to the luminance gradient direction corresponds to the line segment direction (the direction in which the line segment extends). Also, the direction of the luminance gradient may be the direction of the luminance gradient of the first target pixel 111, or the direction of the luminance gradient of the current target pixel 111, or a direction calculated from the luminance gradient directions of a plurality of pixels already assigned to that area (for example, an average value or a weighted average value), etc.

[0038] Note that the direction of the search window 112 does not have to exactly match the direction of the luminance gradient. For example, a plurality of candidates for the direction of the search window 112 may be preset, and the candidate in the direction closest to the direction perpendicular to the luminance gradient may be selected. The plurality of candidates includes at least the vertical direction and the horizontal direction (the horizontal direction (row direction) and the vertical direction (column direction) of the image). Also, as shown in FIG. 9, the plurality of candidates may include an oblique direction (a direction other than the horizontal and vertical directions of the image). Further, when the search window 112 is arranged obliquely, the pixel to be searched may be a pixel that at least partially includes the search window 112, a pixel that includes all regions of the search window 112, or a pixel that includes an area equal to or greater than a predetermined area of the search window 112. For example, the search window 112 is arranged such that the angle difference between the long side of the search window 112 and the direction of the line segment (the direction perpendicular to the direction of the luminance gradient) is smaller than the angle difference between the short side of the search window 112 and the direction of the line segment.

[0039] Here, when the intensity of the luminance gradient in the image is insufficient due to the influence of blur or noise, a portion that should be detected as one line segment (region) may be detected as two line segments (regions). For example, in the examples shown in FIGS. 4 to 8 and the like, the region located between the right region and the left region is a region where the intensity of the luminance gradient is insufficient due to the influence of blur or noise. In this case, for example, when a square search window (e.g., 3×3) is used, the left and right regions are detected as separate regions. Similarly, even when the 3×7 search window 112 is used, when the long side is arranged in the vertical direction, the left and right regions are detected as separate regions. On the other hand, as described above, by arranging the search window 112 such that the long side of the search window 112 is perpendicular to the direction of the luminance gradient (the horizontal direction in FIG. 5 and the like), the left and right regions can be detected as a single region. Therefore, the line segment detection unit 103 can improve the accuracy of region division. As a result, the accuracy of line segment detection is improved, and the accuracy of the line segment model is improved.

[0040] Although an example using luminance gradients for detecting line segment regions has been shown, generally any method used for edge detection may be used. For example, instead of luminance gradients, gradients of other pixel values may be used. Here, other pixel values may be, for example, gradients of one of the R, G, and B components. For example, when the color of the object is known, the component corresponding to the color may be used. Also, not limited to gradients (first derivative values), second derivative values may be used.

[0041] Once again, the description of FIG. 3 will be given. After the region division process (S111), the line segment detection unit 103 assigns a rectangle that covers the obtained divided region to each divided region (S112). Next, the line segment detection unit 103 executes a threshold process, which is a filtering process based on the shape and size of the rectangle (S113). Specifically, the line segment detection unit 103 compares the size (area) of the rectangle with the size (area) of the divided region, and if the ratio of the divided region to the rectangle is equal to or greater than a predetermined threshold, the center line of the rectangle is detected as a line segment. Thereby, the line segments included in each image are detected.

[0042] Next, the line segment detection unit 103 detects pairs of line segments that intersect each other, and generates a plurality of line segments by dividing the line segments at the intersection points (S114). FIGS. 10 and 11 are diagrams for explaining the line segment division process. In the example shown in FIG. 10, there are four line segments A, B, C, and D in the image 121. Here, the line segment A and the line segment C intersect, and the line segment B and the line segment C intersect. Therefore, the line segment detection unit 103 divides the line segment A at the intersection of the line segment A and the line segment C to generate the line segments A1 and A2. The line segment detection unit 103 divides the line segment B at the intersection of the line segment B and the line segment C to generate the line segments B1 and B2. The line segment detection unit 103 divides the line segment C at the intersection of the line segment A and the line segment C to generate the line segments C1 and C2.

[0043] Here, an example of dividing each line segment at the intersection of two line segments has been shown. However, a certain line segment may be divided at a position where it overlaps (connects) with another line segment. For example, when one end of the first line segment is located on the second line segment (for example, in the case of a T shape), the second line segment may be divided at the position where one end of the first line segment overlaps (connects) with the second line segment.

[0044] Also, the position for dividing the line segment is not limited to the intersection position of two line segments. For example, a line segment reaching the image edge may be divided at a position a predetermined distance away from the image edge.

[0045] Next, the line segment detection unit 103 performs a filtering process on the divided line segments based on the position and length of the line segments. Specifically, the line segment detection unit 103 excludes line segments reaching the image edge from a plurality of line segments (S115). FIG. 12 is a diagram for explaining this line segment exclusion process. In the example shown in FIG. 12, since the line segments A2, B2, and C2 reach the screen edge of the image 121, the line segments A2, B2, and C2 are excluded. Here, for a line segment to reach the image edge means that at least one end of the line segment coincides with the screen edge. Also, line segment reconstruction (S104) is performed using the line segments generated in this way.

[0046] Here, the length of a line segment reaching the image edge is different from that of an actual line segment. Therefore, if line segment reconstruction is performed using such a line segment, the accuracy of line segment reconstruction may decrease. On the other hand, in the present embodiment, the accuracy of line segment reconstruction can be improved by excluding such line segments.

[0047] Also, in the present embodiment, as a pre - processing for excluding line segments, the detected line segments are separated at the intersections with other line segments. Thereby, the range of the line segments to be excluded in the exclusion of line segments can be reduced, so that a decrease in the accuracy of line segment reconstruction can be suppressed.

[0048] In the above description, after the line segment division (S114) is performed, the exclusion process (S115) is performed on the divided line segments. However, instead of performing the line segment division (S114), the exclusion process (S115) may be performed on the plurality of line segments generated in step S113.

[0049] As described above, the three-dimensional model generation device 100 according to the present embodiment performs the processing shown in FIG. 13. The three-dimensional model generation device 100 detects a first line from each of a plurality of images obtained by photographing a target region from a plurality of viewpoints (S121), excludes a second line reaching the image edge from the detected first line (S122), and uses the plurality of line segments which are the plurality of first lines after excluding the second line segments to generate a line segment model which is a three-dimensional model of the target region represented by lines (S123). According to this, the three-dimensional model generation device can improve the accuracy of the three-dimensional model.

[0050] For example, the three-dimensional model generation device 100 divides a third line reaching the image edge among the plurality of first lines into a second line reaching the image edge and a line segment not reaching the image edge, and excludes the second line from the plurality of first lines. In other words, the three-dimensional model generation device 100 leaves the line segment of the third line that does not reach the image edge as a line segment. According to this, since a part of the line segment reaching the image edge can be used for generating the three-dimensional model, the accuracy of the three-dimensional model can be improved.

[0051] For example, the three-dimensional model generation device 100 divides the third line at a position where the third line overlaps with another line, thereby specifying a second line reaching the image edge and a line segment not reaching the image edge. The overlapping position corresponds to, for example, a corner of an object. The corner of the object is a location where feature point mapping is easy during three-dimensional model generation. Therefore, if the third line is divided at a position where the third line overlaps with another line, the accuracy of the three-dimensional model can be improved.

[0052] For example, in the detection of the first line, the three-dimensional model generation device 100 detects the first line by a search process using a search window on the image, and the search window is a rectangle with the long side along the direction of the first line. According to this, the detection accuracy of the line segment can be improved, so the accuracy of the three-dimensional model generated using the line segment can be improved.

[0053] For example, the rectangle is arranged such that the long side is along the direction of the first line and along a direction other than the vertical and horizontal directions of the image. According to this, the detection accuracy of the line segment can be improved, so the accuracy of the three-dimensional model generated using the line segment can be improved.

[0054] For example, the three-dimensional model generation device 100 includes a processor and a memory, and the processor performs the above processing using the memory.

[0055] As described above, the three-dimensional model generation device and the like according to the embodiments of the present disclosure have been described, but the present disclosure is not limited to these embodiments.

[0056] Also, each processing unit included in the three-dimensional model generation device and the like according to the above embodiment is typically realized as an LSI which is an integrated circuit. These may be individually formed into one chip, or may be formed into one chip so as to include some or all of them.

[0057] Also, the integration into an integrated circuit is not limited to an LSI, and it may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

[0058] In addition, in each of the above embodiments, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0059] Further, the present disclosure may be realized as a three-dimensional model generation method or the like executed by a three-dimensional model generation device or the like. Further, the present disclosure may be realized as a device or method having some operations included in the three-dimensional model generation device. For example, the present disclosure may be realized as a line segment detection device including at least the line segment detection unit 103.

[0060] Also, the division of blocks in the block diagram is an example, and a plurality of blocks may be realized as one block, one block may be divided into a plurality, or a part may be transferred to other blocks. Also, the operations of a plurality of blocks having similar operations may be executed by a single piece of hardware or software in parallel or time-divisionally.

[0061] Also, the order in which each step in the flowchart is executed is for illustration in order to specifically explain the present disclosure, and may be an order other than the above. Also, some of the above steps may be executed simultaneously (in parallel) with other steps.

[0062] As described above, the three-dimensional model generation device or the like according to one or more aspects has been described based on the embodiments, but the present disclosure is not limited to these embodiments. As long as the gist of the present disclosure is not deviated from, various modifications conceived by those skilled in the art applied to these embodiments or forms constructed by combining components in different embodiments may also be included within the scope of one or more aspects.

Industrial Applicability

[0063] The present disclosure can be applied to a three-dimensional model generation device.

Description of Symbols

[0064] 100 Three-dimensional model generation device 101 Image acquisition unit 102 Preprocessing unit 103 Line segment detection unit 104 Line segment reconstruction unit 111 Pixel of interest 112 Search window 121 Image

Claims

1. The processor detects a first line from each of a plurality of images obtained by photographing a target area from a plurality of viewpoints, the processor excludes a second line reaching the image edge from the plurality of detected first lines, the processor generates a line segment model, which is a three-dimensional model of the target area represented by lines, using a plurality of line segments that are the first lines after excluding the second line. A three-dimensional model generation method.

2. The processor divides a third line reaching the image edge among the plurality of first lines into the second line reaching the image edge and a line segment not reaching the image edge, and excludes the second line from the plurality of first lines. The three-dimensional model generation method according to Claim 1.

3. The processor divides the third line at a position where the third line overlaps with another line, thereby identifying the second line reaching the image edge and the line segment not reaching the image edge. The three-dimensional model generation method according to Claim 2.

4. In the detection of the first line, the processor detects the first line by a search process using a search window in the image, the search window is a rectangle whose long side is along the direction of the first line. The three-dimensional model generation method according to any one of Claims 1 to 3.

5. The rectangle is arranged such that the long side is along the direction of the first line and along a direction other than the vertical and horizontal directions of the image. The three-dimensional model generation method according to Claim 4.

6. A processor and a memory, the processor uses the memory to detect a first line from each of a plurality of images obtained by photographing a target area from a plurality of viewpoints, In each of the plurality of images, a line segment model, which is a three-dimensional model of the target region represented by a line, is generated using a line segment that is the first line from which a second line reaching the edge of the image is excluded. Three-dimensional model generation device.

Images