Image processing program, image processing method, and information processing device

Abstract

To improve extraction accuracy of a feature point from an image obtained by imaging an object.SOLUTION: An information processing device 101 performs clustering on a first image 110 while varying a cluster number to calculate a first cluster number and a first base color of each cluster obtained by classifying a pixel group in the first image 110 by the first cluster number. The information processing device 101, for each of a plurality of images 120 obtained by imaging the object from different angles, generates a first replaced image in which a portion other than a color corresponding to the first base color of each cluster is replaced with a specific color. The information processing device 101, for each of the plurality of first replaced images 130, performs clustering using the first cluster number to calculate a second base color of each cluster of the first cluster number, and generates a second replaced image in which a pixel classified into each cluster is replaced with the second base color of each cluster.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing program, an image processing method, and an information processing device. [Background technology]

[0002] Conventionally, there is a technology called photogrammetry that automatically generates a 3D (Dimensions) model based on images of a subject taken from various angles. SFM (Structure From Motion) is performed as a pre-processing step for photogrammetry. SFM is a process that extracts feature points that indicate the same location in multiple images from an image and estimates the position and orientation of the camera that captured the image. In SFM, feature points are extracted based on color gradients, for example.

[0003] Prior art techniques include deriving low-resolution images with gradually different resolutions from an original image, and determining, from the lowest-resolution image with the lowest resolution, the initial values ​​of each component of the representative vector of each cluster, as well as the initial value of the distortion threshold that serves as an indicator of the degree of optimal classification in the image with the next lowest resolution.

[0004] There is also a technique in which feature groups consisting of features of each pixel that constitutes an image are clustered in a feature space based on the features, and feature groups that are determined to be reclustered based on conditions set for each clustered feature group are re-clustered in the feature space based on the features.

[0005] There is also a technology that calculates feature points and feature amounts for each of multiple still images captured of a target for which 3D information is to be generated, calculates the distance between still images for all combinations of two still images from the multiple still images from the feature amounts, and generates a tree diagram that hierarchically combines clusters into which the multiple still images are classified based on the distance between the still images. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2004-258749 A [Patent Document 2] JP 2002-222419 A [Patent Document 3] JP 2020-095498 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in conventional techniques, when generating a 3D model based on an image of an object, it is sometimes impossible to extract feature points from the image with high accuracy.

[0008] According to one aspect, the present invention aims to improve the accuracy of extracting feature points from an image of a target object. [Means for solving the problem]

[0009] In one embodiment, a first image in which parts of an image of an object other than the object are converted to a specific color, and a plurality of images of the object are captured from different angles are acquired, and clustering is performed to classify a group of pixels in the first image into clusters of the number of clusters while changing the number of clusters based on the pixel values ​​of each pixel in the acquired first image. In this way, a first cluster number for classifying the group of pixels in the first image and each cluster obtained by classifying the group of pixels in the first image by the first cluster number are determined so that the number of clusters is as large as possible and the minimum value of the color distance of the base colors between different clusters is not less than a threshold value. the first base color of a raster; generating a first replaced image for each of the acquired images by replacing portions of the color other than the color corresponding to the calculated first base color of each cluster with a specific color; performing clustering for each of the generated first replaced images to classify pixels in the first replaced image into clusters of the first cluster number based on pixel values ​​of each pixel in the first replaced image; calculating a second base color for each cluster of the first cluster number; and generating a second replaced image in which the pixels classified into each cluster are replaced with the second base color of the cluster. Effect of the Invention

[0010] Advantageous Effects of Invention According to one aspect of the present invention, it is possible to improve the accuracy of extracting feature points from an image of a captured object. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of an image processing method according to an embodiment. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of a system configuration of an information processing system 200. As shown in FIG. [Diagram 3] FIG. 3 is a block diagram showing an example of the hardware configuration of the image processing device 201. As shown in FIG. [Figure 4] FIG. 4 is an explanatory diagram showing an example of the contents stored in the representative image management table 220. As shown in FIG. [Diagram 5] FIG. 5 is an explanatory diagram showing an example of the contents of the captured image management table 230. As shown in FIG. [Figure 6] FIG. 6 is a block diagram showing an example of the functional configuration of the image processing device 201. As shown in FIG. [Figure 7] FIG. 7 is an explanatory diagram showing a specific example of the representative image. [Figure 8] FIG. 8 is an explanatory diagram showing an example of calculation of the color gradient cluster number and the representative image base color. [Figure 9] FIG. 9 is an explanatory diagram showing an example of generation of the first replaced captured image. [Figure 10] FIG. 10 is an explanatory diagram showing an example of calculation of the replaced captured image base color. [Figure 11] FIG. 11 is an explanatory diagram showing an example of generation of the second replaced captured image. [Figure 12] FIG. 12 is an explanatory diagram showing an example of generation of the third replaced captured image. [Figure 13A] FIG. 13A is an explanatory diagram (part 1) showing an image comparison example before and after image processing. [Figure 13B] FIG. 13B is an explanatory diagram (part 2) showing a comparative example of images before and after image processing. [Figure 14] FIG. 14 is a flowchart showing an example of an image processing procedure of the image processing device 201. [Figure 15] FIG. 15 is a flowchart showing an example of a specific processing procedure of the representative image base color calculation process. [Figure 16] FIG. 16 is a flowchart illustrating an example of a specific processing procedure of the replaced image generating process. [Figure 17] FIG. 17 is a flowchart showing an example of a specific processing procedure of the color reduction process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of an image processing program, an image processing method, and an information processing device according to the present invention will be described in detail with reference to the accompanying drawings.

[0013] (Embodiment) Fig. 1 is an explanatory diagram showing an example of an image processing method according to an embodiment. In Fig. 1, an information processing device 101 is a computer that processes an image of an object. Here, the object is an object for which a 3D model is to be generated. The object is, for example, a structure such as a steel tower, a bridge, or a dam.

[0014] Photogrammetry is a technology that generates 3D models based on images of a subject taken from various angles (viewpoints, directions), and because it can generate 3D models that are lifelike, it is used for building inspections, digital twins, etc. There is also an increasing demand for 3D modeling of large structures such as steel towers, bridges, and dams.

[0015] As a pre-processing step for photogrammetry, SFM is performed on captured images, extracting feature points from the images, matching feature points that indicate the same locations between images, and estimating the position and orientation of the cameras that captured each image by triangulation, etc. Feature amounts are extracted based on the color gradient (color change) in the image, for example, by a machine learning-based method such as deep learning.

[0016] However, with conventional technology, when the subject is an object with a monotonous color, it is difficult to accurately extract feature points that indicate the same parts between images. For example, steel towers and bridge piers often have monotonous colors and no patterns. If feature points that indicate the same parts between images cannot be accurately extracted, there is a problem that the error of SFM (camera position estimation) becomes large, and ultimately the accuracy of generating 3D models decreases.

[0017] Furthermore, even if feature points are extracted from the background parts other than the subject and matching between images is successful, if valid feature points cannot be extracted from the subject, the error in SFM will be large and it will be difficult to improve the accuracy of generating a 3D model.

[0018] In this embodiment, an image processing method is described that improves the accuracy of extracting feature points from an image of an object, even if the object has a monotonous color. For example, even an object with a monotonous color may have uneven paint, scratches, dirt, etc. If such areas can be made to stand out in the image, it becomes easier to find feature points that indicate the same places between images.

[0019] The information processing device 101 performs, for example, a color reduction process using clustering on an image, making it possible to extract slight color changes such as uneven paint application, scratches, and stains as feature points. Here, examples of the processing performed by the information processing device 101 (corresponding to the following processes (1) to (4)) will be described.

[0020] (1) The information processing device 101 performs clustering to classify the pixels in the first image 110 into clusters of the number of clusters while changing the number of clusters based on the pixel values ​​of each pixel in the first image 110. As a result, the information processing device 101 calculates a first cluster number for classifying the pixels in the first image 110 and a first base color of each cluster obtained by classifying the pixels in the first image 110 by the first cluster number.

[0021] Here, the first image 110 is an image in which the parts of the captured image of the object 102 other than the object 102 are converted into a specific color. The first image 110 is, for example, an image (representative image) in which the entire object 102 is captured. The specific color can be set arbitrarily, and is set to, for example, "black." The first image 110 is created, for example, by manually painting the parts of the captured image other than the object 102 black.

[0022] A pixel (or picture element) is an element that constitutes an image, and is the smallest unit that has color information. A pixel value is the color information that a pixel has, for example, an RGB value. An RGB value is a numerical value that expresses a color as a combination of red (R), green (G), and blue (B). For example, an RGB value specifies each color of red (R), green (G), and blue (B) with a value between 0 and 255.

[0023] Clustering is a process of classifying pixels in an image into two or more groups. Clustering is performed, for example, using the k-means method. The k-means method is a clustering technique that classifies data into k groups. For example, the k-means method classifies pixels so that pixels in the same cluster have similar color information and pixels in different clusters have different color information.

[0024] The base color of a cluster is a color that characterizes the cluster and is determined based on the pixel values ​​of the pixels in the cluster. The base color of each cluster is, for example, a color that is labeled to each cluster in clustering using machine learning. The base color of each cluster is, for example, an average color calculated from the RGB values ​​of the pixels in each cluster.

[0025] Specifically, for example, the information processing device 101 calculates the first cluster number and the first base color of each cluster so that the number of clusters is as large as possible and the minimum color distance of the base colors between different clusters is not less than a threshold value.

[0026] Here, the color distance between base colors in different clusters corresponds to an index value that indicates the similarity of the base colors between clusters. The closer the color distance, the higher the similarity between the base colors, and the farther the color distance, the lower the similarity between the base colors. The color distance between base colors is expressed by the distance in the RGB three-dimensional space, calculated based on the RGB values ​​of each base color, for example.

[0027] The threshold value can be set arbitrarily. For example, if the threshold value is too small, the number of clusters will be too large, and the color reduction effect will tend to be small. On the other hand, if the threshold value is too large, the number of clusters will be too small, and the color that characterizes the object will tend to be unable to be sufficiently extracted. For this reason, the threshold value is set manually based on, for example, past experience.

[0028] In calculating the first cluster number, the information processing device 101 may, for example, sequentially perform clustering while increasing the number of clusters to search for the maximum cluster number for which the minimum value of the color distance between the base colors is not less than a threshold value. Also, the information processing device 101 may sequentially perform clustering while decreasing the number of clusters to search for the maximum cluster number for which the minimum value of the color distance between the base colors is not less than a threshold value.

[0029] Here, it is assumed that "4" is calculated as the first cluster number, and "first base colors c1 to c4" are calculated as the first base colors of the clusters classified by the first cluster number "4". In this way, the information processing device 101 extracts representative colors used in the object 102 from the first image 110.

[0030] (2) The information processing device 101 generates a first replaced image by replacing with a specific color portions of the images 120 other than the color corresponding to the calculated first base color of each cluster. Here, the images 120 are images of an object photographed from different angles.

[0031] The colors corresponding to the first base color include, for example, the first base color and colors adjacent to the first base color. The colors adjacent to the first base color are a set of colors close to the first base color. The colors adjacent to the first base color are, for example, a set of colors whose color distance from the first base color is within a predetermined range in the RGB three-dimensional space.

[0032] The replacement with a specific color means replacing the color information of the pixel with color information indicating the specific color. The specific color in the first replaced image is the same color (e.g., black) as the specific color in the first image 110. However, the specific color in the first replaced image may be a color different from the specific color in the first image 110.

[0033] For example, the information processing device 101 generates a first replaced image 130-1 by replacing portions of an image 120-1 included in the plurality of images 120 with specific colors other than the colors corresponding to the first base colors c1 to c4. However, the base colors c1 to c4 include a base color corresponding to the specific color (the specific color or a color close to the specific color).

[0034] For this reason, the information processing device 101 may exclude the specific color or a base color close to the specific color from the base colors c1 to c4. For example, assume that the base color c4 is the specific color. In this case, the information processing device 101 excludes the base color c4 from the base colors c1 to c4. Then, the information processing device 101 may generate a first replaced image in which the portions other than the colors corresponding to the base colors c1 to c3 are replaced with the specific color.

[0035] Here, it is assumed that a first replaced image (for example, first replaced image 130-1) is generated for each of the multiple images 120, resulting in the generation of "multiple first replaced images 130." As a result, the information processing device 101 generates an image (first replaced image) for each of the multiple images 120 in which portions other than the object 102 are replaced with a specific color.

[0036] (3) For each of the generated multiple first replaced images 130, the information processing device 101 performs clustering to classify pixels in the first replaced image into clusters of the first cluster number based on the pixel values ​​of each pixel in the first replaced image, and calculates a second base color for each cluster of the first cluster number.

[0037] Here, the first cluster number is "4". In this case, for example, the information processing device 101 performs clustering for the first replaced image 130-1 based on the pixel values ​​of each pixel in the first replaced image 130-1 to classify the pixels in the first replaced image 130-1 into clusters with the first cluster number of "4". Then, the information processing device 101 calculates the second base color of each cluster with the first cluster number of "4".

[0038] Here, it is assumed that the pixels in first replaced image 130-1 are classified into clusters based on the first cluster number "4" and are defined as "clusters C1 to C4," and second base colors c11 to c14 of each of clusters C1 to C4 are calculated.

[0039] (4) The information processing device 101 generates a second replaced image by replacing the pixels classified into each cluster with the second base color of the cluster for each of the generated first replaced images 130. Here, replacing with the second base color means replacing color information of the pixel with color information indicating the second base color.

[0040] The information processing device 101 replaces pixels of various colors belonging to each cluster with the second base color, thereby reducing the number of colors expressing the object 102 in the first replaced image to the number of first clusters. For example, the information processing device 101 generates a second replaced image 140-1 by replacing pixels classified into each of the clusters C1 to C4 among the pixel groups in the first replaced image 130-1 with the second base colors c11 to c14 of each of the clusters C1 to C4.

[0041] Here, it is assumed that a second-replaced image (eg, second-replaced image 140-1) is generated for each of the plurality of first-replaced images 130, resulting in the generation of "plurality of second-replaced images 140."

[0042] In this way, the information processing device 101 can perform color reduction processing by clustering on images (plurality of images 120) captured of the object 102. This makes it possible for the information processing device 101 to accentuate color changes even in an object with a monotonous color, thereby improving the accuracy of extracting feature points from an image.

[0043] Color image 150 shows a color image expressing object 102 in each of the multiple images 120. Color image 160 shows a color image expressing object 102 in each of the multiple second replaced images 140. In color image 150, object 102 is expressed in, for example, 100,000 colors.

[0044] On the other hand, in the color image 160, the object 102 is expressed in three colors (three colors other than the specific color). The information processing device 101 performs a color reduction process that deliberately reduces the expressiveness of colors, for example, from 100,000 colors to three colors, thereby making it possible to extract slight color changes such as uneven paint application or scratches as feature points.

[0045] Furthermore, the information processing device 101 may perform a camera position estimation process for the multiple images 120 based on the generated multiple second replaced images 140. Here, the camera position estimation process is a process (SFM process) of estimating the position and orientation of the camera that captured each of the multiple images 120.

[0046] This enables the information processing device 101 to capture color changes such as uneven paint application or scratches as feature points, and accurately perform feature point matching (matching feature points that indicate the same location) between multiple images, thereby improving the estimation accuracy of SFM.

[0047] However, the camera position estimation process may be executed in a computer other than the information processing device 101. In this case, the information processing device 101 may output the generated second-replaced images 140 to the other computer. The other computer can perform the camera position estimation process for the multiple images 120 based on the output second-replaced images 140.

[0048] (System configuration example of information processing system 200) Next, a system configuration example of an information processing system 200 including the information processing device 101 shown in Fig. 1 will be described. Here, a case where the information processing device 101 shown in Fig. 1 is applied to an image processing device 201 in the information processing system 200 will be described as an example.

[0049] Fig. 2 is an explanatory diagram showing an example of a system configuration of an information processing system 200. In Fig. 2, the information processing system 200 includes an image processing device 201 and a client device 202. In the information processing system 200, the image processing device 201 and the client device 202 are connected via a wired or wireless network 210. The network 210 is, for example, the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0050] Here, the image processing device 201 is a computer that has a representative image management table 220 and a captured image management table 230 and generates a 3D model of an object. The image processing device 201 is, for example, a server. The contents stored in the representative image management table 220 and the captured image management table 230 will be described later with reference to Figs. 4 and 5.

[0051] The client device 202 is a computer used by a user of the information processing system 200. The client device 202 is, for example, a personal computer (PC) or a tablet PC.

[0052] In this embodiment, the image processing device 201 and the client device 202 are provided separately, but the present invention is not limited to this. For example, the image processing device 201 may be realized by the client device 202. Furthermore, the information processing system 200 may include a plurality of client devices 202.

[0053] (Example of hardware configuration of image processing device 201) Next, an example of the hardware configuration of the image processing device 201 will be described.

[0054] Fig. 3 is a block diagram showing an example of the hardware configuration of the image processing device 201. In Fig. 3, the image processing device 201 has a CPU (Central Processing Unit) 301, a memory 302, a disk drive 303, a disk 304, a communication I / F (Interface) 305, a portable recording medium I / F 306, and a portable recording medium 307. In addition, each component is connected to each other by a bus 300.

[0055] Here, the CPU 301 is responsible for the overall control of the image processing device 201. The CPU 301 may have multiple cores. The memory 302 has, for example, a read only memory (ROM), a random access memory (RAM), and a flash ROM. Specifically, for example, the flash ROM stores an OS program, the ROM stores application programs, and the RAM is used as a work area for the CPU 301. The programs stored in the memory 302 are loaded into the CPU 301 to cause the CPU 301 to execute the coded processes.

[0056] The disk drive 303 controls reading / writing data from / to the disk 304 under the control of the CPU 301. The disk 304 stores data written under the control of the disk drive 303. The disk 304 is, for example, a magnetic disk, an optical disk, or the like.

[0057] The communication I / F 305 is connected to the network 210 through a communication line, and is connected to an external computer (e.g., the client device 202 shown in FIG. 2) via the network 210. The communication I / F 305 manages the interface between the network 210 and the inside of the device, and controls input and output of data from the external computer. The communication I / F 305 is, for example, a modem or a LAN adapter.

[0058] The portable recording medium I / F 306 controls reading / writing of data from / to the portable recording medium 307 under the control of the CPU 301. The portable recording medium 307 stores data written under the control of the portable recording medium I / F 306. The portable recording medium 307 is, for example, a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disk), or a USB (Universal Serial Bus) memory.

[0059] In addition to the above-mentioned components, the image processing device 201 may have, for example, a GPU (Graphics Processing Unit), an input device, a display, etc. Furthermore, the image processing device 201 may not have, for example, the portable recording medium I / F 306 and the portable recording medium 307 among the above-mentioned components. Furthermore, the client device 202 shown in Fig. 2 can also be realized by the same hardware configuration as the image processing device 201. However, in addition to the above-mentioned components, the client device 202 has, for example, an input device, a display, etc.

[0060] (Storage contents of representative image management table 220 and captured image management table 230) 4 and 5, the contents stored in the representative image management table 220 and the captured image management table 230 of the image processing device 201 will be described. The representative image management table 220 and the captured image management table 230 are realized by a storage device such as the memory 302 and the disk 304 shown in FIG.

[0061] Fig. 4 is an explanatory diagram showing an example of the contents stored in the representative image management table 220. In Fig. 4, the representative image management table 220 has fields for the number of color gradient clusters, a cluster ID, and a representative image base color, and stores representative image base color information (for example, representative image base color information 400-1 to 400-6) as records by setting information in each field.

[0062] Here, the color gradient cluster number is the number of clusters into which the pixel groups in the representative image are classified. The cluster ID is an identifier that identifies the cluster into which the pixel groups in the representative image are classified. The representative image base color is the base color of each cluster into which the pixel groups in the representative image are classified by the color gradient cluster number.

[0063] Fig. 5 is an explanatory diagram showing an example of the contents stored in the captured image management table 230. In Fig. 5, the captured image management table 230 has fields for image ID, pixel ID, original RGB value, replacement flag, cluster ID, and replaced captured image base color, and stores captured image management information (for example, captured image management information 500-1 to 500-n) as records by setting information in each field (n is a natural number equal to or greater than 2).

[0064] Here, the image ID is an identifier that uniquely identifies an image (captured image) of an object. The pixel ID is an identifier that identifies a pixel in the image. The original RGB value is the original RGB value (pixel value) that the pixel has. The number of pixels in each image is, for example, the same.

[0065] The replacement flag indicates whether or not the pixel has been replaced with a specific color. Here, a replacement flag of "1" indicates that the pixel has been replaced with a specific color. A replacement flag of "0" indicates that the pixel has not been replaced with a specific color. In the initial state, the replacement flag is "0". The cluster ID is an identifier that identifies a cluster that classifies a group of pixels in an image. The replaced captured image base color is the base color of the cluster.

[0066] (Example of functional configuration of image processing device 201) Next, an example of the functional configuration of the image processing device 201 will be described.

[0067] Fig. 6 is a block diagram showing an example of a functional configuration of the image processing device 201. In Fig. 6, the image processing device 201 includes an acquisition unit 601, a first calculation unit 602, a first generation unit 603, a second calculation unit 604, a second generation unit 605, a third generation unit 606, an execution control unit 607, and an output unit 608. The acquisition unit 601 to the output unit 608 are functions that constitute the control unit 600, and specifically, the functions are realized by causing the CPU 301 to execute a program stored in a storage device such as the memory 302, the disk 304, or the portable recording medium 307 shown in Fig. 3, or by the communication I / F 305. The processing results of each functional unit are stored in a storage device such as the memory 302 or the disk 304.

[0068] The acquisition unit 601 acquires a representative image of an object. Here, the representative image is an image (first image) in which a portion other than the object in an image (captured image) of the object is converted into a specific color. The representative image is, for example, an image in which the entire object is captured. The object is, for example, a structure such as a steel tower, a bridge, or a dam. The object is, for example, photographed by a drone (unmanned aerial vehicle).

[0069] The specific color can be set arbitrarily, for example, to black ([RGB]=

[0000] ). However, if the object is a blackish object, the specific color may be set to a color other than black, for example, white ([RGB]=[255 255 255]). The representative image is created, for example, by filling in the areas other than the object in the image of the object with a specific color through user input in the client device 202 shown in FIG. 2.

[0070] Furthermore, the representative image may be created in the image processing device 201. For example, the image processing device 201 receives an input of an image of an object captured from the client device 202. Then, the image processing device 201 may create the representative image by filling in the parts of the received image other than the object with a specific color in response to an operational input by an administrator of the information processing system 200.

[0071] The acquisition unit 601 also acquires a plurality of images of the object taken from different angles. Here, the plurality of images are images used to generate a 3D model of the object, and are, for example, several tens to several hundreds of captured images of the object taken from various angles.

[0072] In the following description, a plurality of images captured from different angles of an object may be referred to as a "group of captured images #1 to #n" (n is a natural number equal to or greater than 2). Any captured image from the group of captured images #1 to #n may be referred to as a "captured image #i" (i=1, 2, ..., n).

[0073] Specifically, for example, the acquiring unit 601 acquires the representative image of the object and the captured image group #1 to #n by receiving them from the client device 202 (corresponding to an input source of the captured image group #1 to #n). The acquiring unit 601 may also acquire the representative image of the object and the captured image group #1 to #n by a user's operation input using an input device (not shown).

[0074] A specific example of the representative image will be described later with reference to Fig. 7. Also, a specific example of the captured image groups #1 to #n will be described later with reference to Fig. 9.

[0075] The first calculation unit 602 calculates a color gradient cluster number and a representative image base color for the acquired representative image. Here, the color gradient cluster number is the number of clusters (first cluster number) into which the pixel groups in the representative image are classified. The representative image base color corresponds to the base color (first base color) of each cluster into which the pixel groups in the representative image are classified by the color gradient cluster number. The base color of a cluster is a color that characterizes the cluster, and is determined based on, for example, the RGB values ​​of the pixels in the cluster.

[0076] Specifically, for example, the first calculation unit 602 performs RGB clustering processing on the representative image while changing the number of clusters k (k is a natural number equal to or greater than 2), thereby calculating the number of color gradient clusters and the representative image base color so that the number of clusters k is as large as possible and the minimum value of the color distance of the base colors between different clusters is not less than the threshold value X.

[0077] The RGB clustering process is a process of classifying pixels in an image (e.g., a representative image) into clusters with a cluster number k based on the RGB values ​​of each pixel in the image. The RGB clustering process is performed, for example, by the k-means method. The threshold value X can be set arbitrarily, and is set, for example, manually based on past experience.

[0078] More specifically, for example, the first calculation unit 602 performs the RGB clustering process while gradually increasing the number of clusters k from 2 until the minimum value of the color distance of the base colors between different clusters becomes less than the threshold value X. The minimum value of the color distance of the base colors between different clusters tends to become smaller as the number of clusters k increases.

[0079] Then, the first calculation unit 602 determines the number of color gradient clusters to be the number obtained by subtracting 1 from the number of clusters k when the minimum value of the color distance of the base colors between different clusters becomes less than the threshold value X. Furthermore, the first calculation unit 602 may specify, as the representative image base color, the base color of each cluster into which the pixel group in the representative image is classified using the determined number of color gradient clusters.

[0080] The calculated color gradient cluster number and representative image base color are stored, for example, in the representative image management table 220 shown in Fig. 4. A calculation example of the color gradient cluster number and representative image base color will be described later with reference to Fig. 8.

[0081] The base color of each cluster obtained by classifying the pixels in the representative image by the color gradient cluster number includes a base color corresponding to a specific color (e.g., black). Therefore, the first calculation unit 602 may specify, as the representative image base color, a base color excluding the specific color and a color adjacent to the specific color among the base colors of each cluster obtained by classifying the pixels in the representative image by the color gradient cluster number.

[0082] The first generating unit 603 generates a first replaced captured image #i by replacing a portion of each captured image #i of the acquired captured image group #1 to #n, other than the color corresponding to the calculated representative image base color, with a specific color. Here, the color corresponding to the representative image base color includes, for example, the representative image base color and a color adjacent to the representative image base color.

[0083] The neighboring colors of the representative image base color are, for example, a set of colors existing in a spherical space of radius r centered on the reference point, which is the representative image base color, in an RGB three-dimensional space. The radius r can be set arbitrarily. For example, in the case of 8-bit color, the radius r is set to a value of about 15. The specific color in the first replaced captured image #i is, for example, the same color (such as black) as the specific color in the representative image.

[0084] Information for identifying pixels in the captured image #i that have been replaced with a specific color is managed, for example, using the captured image management table 230 shown in Fig. 5. The captured image management table 230 stores the RGB values ​​of each pixel (pixel ID) in the captured image #i in association with the image ID of the captured image #i. The first generating unit 603 sets, for example, "1" to a replacement flag in the captured image management table 230 that corresponds to a pixel (pixel ID) in the captured image #i that has been replaced with a specific color.

[0085] An example of how the first replaced captured image #i is generated will be described later with reference to FIG.

[0086] The second calculation unit 604 performs RGB clustering processing on each of the first replaced captured images #i in the generated first replaced captured images #1 to #n using the calculated color gradient cluster number to calculate a replaced captured image base color. Here, the replaced captured image base color corresponds to the base color (second base color) of each cluster obtained by classifying the pixel group in the first replaced captured image #i by the color gradient cluster number.

[0087] Information (classification result) specifying the cluster into which each pixel in the first replaced captured image #i is classified is stored, for example, in the captured image management table 230. For example, the second calculation unit 604 sets the cluster ID of the cluster into which each pixel is classified to the cluster ID in the captured image management table 230 corresponding to each pixel in the first replaced captured image #i (corresponding to each pixel in the captured image #i). Also, the second calculation unit 604 sets the replaced captured image base color of the cluster into which each pixel is classified to the replaced captured image base color in the captured image management table 230 corresponding to each pixel in the first replaced captured image #i.

[0088] An example of calculating the replaced captured image base color will be described later with reference to FIG.

[0089] The second generating unit 605 generates a second replaced captured image #i by replacing pixels classified into each cluster of the color gradient cluster number with the replaced captured image base color of each cluster for each first replaced captured image #i of the first replaced captured image group #1 to #n. Specifically, for example, the second generating unit 605 refers to the captured image management table 230 and replaces each pixel in the first replaced captured image #i with the replaced captured image base color of the cluster to which the pixel is classified.

[0090] An example of how the second replaced captured image #i is generated will be described later with reference to FIG.

[0091] The third generating unit 606 generates a third replaced captured image #i by returning a portion of the second replaced captured image #i replaced with a specific color in the second replaced captured image #i to the color before replacement for each of the generated second replaced captured images #1 to #n. The third replaced captured image #i corresponds to, for example, a portion of the second replaced captured image #i (such as the background other than the object) replaced with black (specific color) returned to its original color.

[0092] Specifically, for example, the third generation unit 606 refers to the captured image management table 230 and generates the third replaced captured image #i by replacing the RGB values ​​(replaced captured image base colors) of pixels in the second replaced captured image #i whose replacement flags are “1” with the original RGB values.

[0093] The output unit 608 outputs the generated third replaced captured image groups #1 to #n in association with the captured image groups #1 to #n. The output format of the output unit 608 may be, for example, storage in a storage device such as the memory 302 or the disk 304, or transmission to another computer via the communication I / F 305. The other computer may be, for example, the client device 202 (the input source of the captured image groups #1 to #n) shown in FIG. 2 or a server (not shown).

[0094] Specifically, for example, the output unit 608 outputs to another computer the captured image #i of each of the captured image groups #1 to #n and the generated third replaced captured image #i of each of the third replaced captured image groups #1 to #n in association with each other. This allows the other computer to execute a camera position estimation process for the captured image groups #1 to #n based on the third replaced captured image groups #1 to #n. The camera position estimation process is a process (SFM process: Structure from Motion process) that estimates the position and orientation of the camera (imaging device) that captured each of the captured image groups #1 to #n.

[0095] Furthermore, the output unit 608 may output the generated second replaced captured image groups #1 to #n in association with the captured image groups #1 to #n. Specifically, for example, the output unit 608 may output each captured image #i of the captured image groups #1 to #n and each second replaced captured image #i of the generated second replaced captured image groups #1 to #n in association with each other to another computer.

[0096] This allows another computer to execute a camera position estimation process for the captured image group #1 to #n based on the second replaced captured image group #1 to #n. However, in the second replaced captured image group #1 to #n, the portions other than the color corresponding to the representative image base color (for example, background portions other than the object) remain in a state where they are replaced with a specific color (for example, black). For this reason, the third replaced captured image group #1 to #n is more preferable than the second replaced captured image group #1 to #n as an input image for the camera position estimation process.

[0097] The execution control unit 607 performs a camera position estimation process for the captured image group #1 to #n. Specifically, for example, the execution control unit 607 performs a camera position estimation process for the captured image group #1 to #n based on the generated third-replaced captured image group #1 to #n. To explain in more detail, for example, the execution control unit 607 inputs the third-replaced captured image group #1 to #n into existing photogrammetry software, thereby estimating the position and orientation of the camera that captured each captured image #i corresponding to each third-replaced captured image #i.

[0098] Furthermore, the execution control unit 607 may perform a camera position estimation process for the captured image group #1 to #n based on the generated second replaced captured image group #1 to #n. To explain in more detail, for example, the execution control unit 607 may input the second replaced captured image group #1 to #n to existing photogrammetry software to estimate the position and orientation of the camera that captured each captured image #i corresponding to each second replaced captured image #i.

[0099] Furthermore, the execution control unit 607 may generate a 3D model of the object by executing MVS (Multi-View Stereo) processing on the captured image group #1 to #n based on the estimated positions and orientations of the cameras that captured each of the captured image groups #1 to #n. Here, the MVS processing is processing that generates a 3D shape of the object from the shooting positions (position, orientation) of the cameras that captured the object.

[0100] Specifically, for example, the execution control unit 607 replaces the input image to the photogrammetry software with the captured image group #1 to #n, and executes MVS processing from the estimated camera position and orientation to generate a 3D model of the object. The 3D model of the object is, for example, point cloud data representing the object.

[0101] Furthermore, the output unit 608 outputs the generated 3D model of the object in association with the captured image group #1 to #n. Specifically, for example, the output unit 608 may output the generated 3D model (point cloud data) of the object to the client apparatus 202 that is the input source of the captured image group #1 to #n.

[0102] Furthermore, the output unit 608 may output the processing result of the camera position estimation processing for the captured image group #1 to #n. Specifically, for example, the output unit 608 may output the processing result of the camera position estimation processing to another computer. This allows the other computer to execute MVS processing for the captured image group #1 to #n.

[0103] It should be noted that the image processing device 201 does not need to include the execution control unit 607 when the camera position estimation process and the MVS process for the captured image groups #1 to #n are executed in another computer.

[0104] (Image processing example of the image processing device 201) Next, a description will be given of an example of image processing by the image processing device 201. First, a specific example of a representative image of an object will be described with reference to FIG.

[0105] Fig. 7 is an explanatory diagram showing a specific example of a representative image. In Fig. 7, a captured image 701 is an image obtained by capturing an object Ob, and is an original image of a representative image 702. The object Ob is a "steel tower" and is an object with a monotonous color. Note that Fig. 7 uses an example of an image in which a part of the object Ob is captured as the captured image 701, but an image in which the entire object Ob is captured is preferable.

[0106] The representative image 702 is an image (first image) in which the parts of the captured image 701 other than the object Ob are converted into a specific color. Here, the specific color is "black." The representative image 702 is created, for example, by painting the parts of the captured image 701 other than the object Ob with black, in response to an operational input by the user.

[0107] Next, a calculation example of the color gradient cluster number and the representative image base color will be described with reference to Fig. 8. The image processing device 201 calculates the color gradient cluster number and the representative image base color by performing RGB clustering processing on the representative image 702 shown in Fig. 7 while changing the cluster number k.

[0108] 8 is an explanatory diagram showing an example of calculation of the color gradient cluster number and the representative image base color. In Fig. 8, first, the image processing device 201 performs RGB clustering processing on the representative image 702 by setting the cluster number k to "k=2".

[0109] Clusters 811 and 812 show the classification results obtained by performing RGB clustering processing on the representative image 702, with the number of clusters k set to "k=2." In FIG. 8, the horizontal axis shows the number of pixels classified into each cluster. The vertical axis shows the base color [R, G, B] of each cluster. Here, the value of each color is expressed as a "value from 0 to 255" (8-bit color).

[0110] [136 138 140] is the base color of cluster 811.

[0112] is the base color of cluster 812. The base color of each cluster 811, 812 is the color labeled to each cluster 811, 812 in the RGB clustering process, and is based on the RGB values ​​of the pixels in each cluster 811, 812.

[0111]

[0112] is a color close to black. For this reason, it can be said that cluster 811 contains many pixels that form parts other than the object Ob (parts converted to black). On the other hand, it can be said that cluster 812 contains many pixels that form the object Ob.

[0112] Here, the color distance between the base color [136 138 140] of cluster 811 and the base color

[0112] of cluster 812 is 236.7. In addition, because the number of clusters k is 2, the color distance of 236.7 between the base colors of clusters 811 and 812 is the minimum value.

[0113] The image processing device 201 performs RGB clustering processing on the representative image 702 while increasing the number of clusters k by 1, until the minimum value of the color distance of the base colors between different clusters becomes less than the threshold value X. Here, the threshold value X is set to "X=30".

[0114] In this case, because the minimum value of the color distance of the base colors, "236.7", is equal to or greater than the threshold value X, the image processing device 201 increments the number of clusters k and performs RGB clustering processing on the representative image 702. Here, it is assumed that the RGB clustering processing on the representative image 702 is performed while incrementing the number of clusters k by 1 until the number of clusters k becomes "k=6".

[0115] Clusters 821 to 826 show the classification results obtained by performing RGB clustering processing on the representative image 702, with the number of clusters k set to "k=6." Here, [184 192 193] is the base color of cluster 821. [123 123 124] is the base color of cluster 822. [46 53 69] is the base color of cluster 823.

[0116] [152 157 160] is the base color of cluster 824.

[0000] is the base color of cluster 825. [92 94 97] is the base color of cluster 826. The base colors of each cluster 821-826 are colors labeled to each cluster 821-826 in the RGB clustering process, and are based on the RGB values ​​of the pixels in each cluster 821-826.

[0117]

[0000] is black. For this reason, it can be said that the cluster 825 contains many pixels that form parts other than the object Ob (parts converted to black). On the other hand, it can be said that the clusters 821 to 824 and 826 contain pixels that form the object Ob dispersedly.

[0118] Here, the minimum color distance between the base colors of different clusters among clusters 821 to 826 is the color distance "57.4" between the base color of cluster 822 [123 123 124] and the base color of cluster 824 [152 157 160].

[0119] In this case, since the minimum value of the color distance of the base colors, “57.4”, is equal to or greater than the threshold value X, the image processing device 201 increments the number of clusters k and performs RGB clustering processing on the representative image 702 .

[0120] Clusters 831 to 837 show the classification results obtained by performing RGB clustering processing on the representative image 702, with the number of clusters k set to "k=7." Here, [49 56 70] is the base color of cluster 831. [95 96 99] is the base color of cluster 832. [151 155 158] is the base color of cluster 833. [123 123 124] is the base color of cluster 834.

[0121] [194 201 202] is the base color of cluster 835. [177 185 187] is the base color of cluster 836.

[0000] is the base color of cluster 837. The base color of each cluster 831-837 is the color labeled to each cluster 831-837 in the RGB clustering process, and is based on the RGB values ​​of the pixels in each cluster 831-837.

[0122]

[0000] is black. Therefore, it can be said that the cluster 837 contains many pixels that form parts other than the object Ob (parts converted to black). On the other hand, it can be said that the clusters 831 to 836 contain pixels that form the object Ob in a dispersed manner.

[0123] Here, the minimum color distance between the base colors of different clusters among clusters 831 to 837 is the color distance "27.7" between the base color of cluster 835 [194 201 202] and the base color of cluster 836 [177 185 187].

[0124] In this case, since the minimum value of the color distance of the base colors, "27.7", is less than the threshold value X, the image processing device 201 ends the RGB clustering process for the representative image 702. Then, the image processing device 201 determines the number of color gradient clusters to be "6", which is obtained by subtracting 1 from the number of clusters "k=7".

[0125] Furthermore, the image processing device 201 specifies the base colors of the clusters 821 to 826 obtained by classifying the pixel groups in the representative image 702 with the color gradient cluster number "6" as the representative image base colors of the clusters 821 to 826. However, the image processing device 201 may exclude the base color of the cluster 825 corresponding to a specific color (here, black) from the representative image base colors among the base colors of the clusters 821 to 826.

[0126] Next, a generation example of a first replaced captured image #i generated for each captured image #i in the captured image groups #1 to #n will be described with reference to FIG.

[0127] Fig. 9 is an explanatory diagram showing an example of generating a first replaced captured image. In Fig. 9, a captured image group 900 is a plurality of images of an object Ob captured from different angles. The image processing device 201 generates a first replaced captured image by replacing the portions of the captured image group 900 other than the color corresponding to the calculated representative image base color with black (specific color).

[0128] Here, the representative image base color is defined as the base color of each of the clusters 821 to 824 and 826 shown in Fig. 8, excluding the base color of the cluster 825 corresponding to black (specific color). For example, the base color of the cluster 821 is [184 192 193]. Also, the colors corresponding to the representative image base color are defined as the representative image base color and the colors adjacent to the representative image base color.

[0129] The neighboring colors of the representative image base color are set as a reference point in the RGB three-dimensional space, and are a set of colors that exist in a spherical space with a radius of 15 centered on the representative image base color. For example, if the representative image base color is the base color [184 192 193] of cluster 821, the neighboring colors of the representative image base color are set as a reference point, and are a set of colors that exist in a spherical space with a radius of 15 centered on the base color [184 192 193].

[0130] The first replaced captured image group 901 is a collection of first replaced captured images generated for each of the captured image group 900. Each of the first replaced captured image group 901 is an image in which colors other than the representative image base color and colors adjacent to the representative image base color are replaced with black, and the image is such that parts other than the object Ob are replaced with black.

[0131] Next, an example of calculating the replaced captured image base color will be described with reference to FIG.

[0132] Fig. 10 is an explanatory diagram showing an example of calculation of a replaced captured image base color. In Fig. 10, the image processing device 201 performs RGB clustering processing on each of the generated first replaced captured images 901 with the calculated color gradient cluster number "6 (see Fig. 8)" and calculates a replaced captured image base color of each cluster of the classified color gradient cluster number.

[0133] Taking the first-replaced captured image 1001 included in the first-replaced captured image group 901 as an example, the image processing device 201 sets the number of clusters k to the color gradient cluster number "6" and performs RGB clustering processing on the first-replaced captured image 1001. Clusters 1011 to 1016 indicate classification results obtained by performing RGB clustering processing on the first-replaced captured image 1001 with the number of clusters k set to "k=6".

[0134] [99 96 93] is the base color of cluster 1011. [185 190 190] is the base color of cluster 1012. [59 56 53] is the base color of cluster 1013.

[0000] is the base color of cluster 1014. [156 155 152] is the base color of cluster 1015. [123 121 120] is the base color of cluster 1016.

[0135]

[0000] is black. Therefore, it can be said that the cluster 1014 contains many pixels that form parts other than the object Ob (parts converted to black). On the other hand, it can be said that the clusters 1011 to 1013, 1015, and 1016 contain pixels that form the object Ob dispersedly.

[0136] In this manner, the image processing device 201 calculates the base colors of the clusters 1011 to 1016 obtained by classifying the pixel groups in the first replaced captured image 1001 by the color gradient cluster number "6" as the replaced captured image base colors for the first replaced captured image 1001. Similarly, the image processing device 201 calculates the replaced captured image base colors of each cluster of the color gradient cluster number for the other first replaced captured images.

[0137] Next, a generation example of the second replaced captured image #i will be described with reference to FIG.

[0138] Fig. 11 is an explanatory diagram showing an example of generating a second replaced captured image. In Fig. 11, the image processing device 201 generates a second replaced captured image by replacing pixels classified into each cluster of the color gradient cluster number "6" for each of the first replaced captured images group 901 with the replaced captured image base color of the corresponding cluster.

[0139] Taking the first replaced captured image 1001 included in the first replaced captured image group 901 as an example, the image processing device 201 replaces pixels classified into cluster 1011 among the pixel group in the first replaced captured image 1001 with the replaced captured image base color [99 96 93] of the cluster 1011. Also, the image processing device 201 replaces pixels classified into cluster 1012 with the replaced captured image base color [185 190 190] of the cluster 1012.

[0140] Moreover, the image processing device 201 replaces the pixels classified into the cluster 1013 with the replaced captured image base color [59 56 53] of the cluster 1013. Moreover, the image processing device 201 replaces the pixels classified into the cluster 1014 with the replaced captured image base color

[0000] of the cluster 1014. Moreover, the image processing device 201 replaces the pixels classified into the cluster 1015 with the replaced captured image base color [156 155 152] of the cluster 1015. Moreover, the image processing device 201 replaces the pixels classified into the cluster 1016 with the replaced captured image base color [123 121 120] of the cluster 1016.

[0141] In this way, the image processing device 201 generates a second replaced captured image 1101 from the first replaced captured image 1001. The second replaced captured image 1101 is expressed using six replaced captured image base colors (the base colors of the clusters 1011 to 1016 shown in FIG. 10 ) and has fewer colors than the first replaced captured image 1001.

[0142] Similarly, the image processing device 201 generates second replaced captured images for the other first replaced captured images. As a result, a second replaced captured image group 1100 is generated. The second replaced captured image group 1100 includes the second replaced captured images generated for each of the first replaced captured images group 901.

[0143] Next, a generation example of the third replaced captured image #i will be described with reference to FIG.

[0144] Fig. 12 is an explanatory diagram showing an example of generating a third-replaced captured image. In Fig. 12, the image processing device 201 generates a third-replaced captured image by returning the part replaced with black (specific color) in the second-replaced captured image to the color before replacement for each of the second-replaced captured image group 1100. The color before replacement is specified from the captured image management table 230, for example.

[0145] Taking the second replaced captured image 1101 included in the second replaced captured image group 1100 as an example, the image processing device 201 generates a third replaced captured image 1201 by returning the part replaced with black in the second replaced captured image 1101 to the color before replacement (original color). In this way, the image processing device 201 can return the background part replaced with black (specific color) to the original color, making it easier to find feature points from the background part.

[0146] Similarly, the image processing device 201 generates third replaced captured images for the other second replaced captured images. As a result, a third replaced captured image group 1200 is generated. The third replaced captured image group 1200 includes the third replaced captured images generated for each of the second replaced captured images group 1100.

[0147] The image processing device 201 can accurately estimate the position and orientation of the camera that captured each of the captured images 900 (see FIG. 9) by inputting the third replaced captured image group 1200 into existing photogrammetry software, for example. Also, the image processing device 201 can accurately generate a three-dimensional model of the object Ob by replacing the input image to the photogrammetry software with the captured image group 900 and executing MVS processing from the estimated camera position and orientation, for example.

[0148] (Comparison of images before and after image processing) Next, FIG. 13A and FIG. 13B are explanatory diagrams showing comparative images before and after image processing.

[0149] 13A and 13B are explanatory diagrams showing comparative images before and after image processing. In Fig. 13A, a captured image 1301 is an image (corresponding to captured image #i) before image processing by the image processing device 201. When a part 1310 of a steel tower (object Ob) shown in the captured image 1301 is enlarged, it is possible to see uneven paint or dirt on the surface of a steel pipe 1311 of the steel tower (object Ob), but this is difficult to see due to the monotonous color.

[0150] In Fig. 13B, a captured image 1302 is an image (corresponding to the third-replaced captured image #i) after image processing by the image processing device 201. In the captured image 1302, a color reduction process that intentionally reduces the expressiveness of colors is performed by RGB clustering processing. When a portion 1320 (corresponding to the portion 1310 shown in Fig. 13A) of a steel tower (object Ob) shown in the captured image 1302 is enlarged, uneven paint or dirt on the surface of a steel pipe 1321 (corresponding to the steel pipe 1311 shown in Fig. 13A) of the steel tower (object Ob) becomes visible.

[0151] In this way, the image processing device 201 performs color reduction processing on the captured image 1301, thereby making it possible to extract slight color changes, such as uneven paint application or dirt, from the captured image 1302 as feature points.

[0152] (Image processing procedure of image processing device 201) Next, the image processing procedure of the image processing device 201 will be described with reference to FIG.

[0153] Fig. 14 is a flowchart showing an example of an image processing procedure of the image processing device 201. In the flowchart of Fig. 14, first, the image processing device 201 determines whether or not it has acquired a representative image of an object and a group of captured images #1 to #n in which the object is captured from different angles (step S1401).

[0154] Here, the image processing device 201 waits for the representative image and the captured image group #1 to #n to be acquired (step S1401: No). Then, when the image processing device 201 acquires the representative image and the captured image group #1 to #n (step S1401: Yes), it executes the representative image base color calculation process (step S1402). The specific processing procedure of the representative image base color calculation process will be described later with reference to FIG. 15.

[0155] Next, the image processing device 201 selects an unselected captured image group #i from among the acquired captured image groups #1 to #n (step S1403). Then, the image processing device 201 executes a replaced image generation process for the selected captured image group #i (step S1404). A specific processing procedure of the replaced image generation process will be described later with reference to FIG. 16.

[0156] Next, the image processing device 201 executes color reduction processing on the selected captured image group #i (step S1405). A specific processing procedure of the color reduction processing will be described later with reference to Fig. 17. Then, the image processing device 201 determines whether or not there is an unselected captured image group #i among the captured image groups #1 to #n (step S1406).

[0157] If there is an unselected captured image group #i (step S1406: Yes), the image processing device 201 returns to step S1403. On the other hand, if there is no captured image group #i (step S1406: No), the image processing device 201 executes SFM processing to estimate the position and orientation of the camera that captured each of the captured image groups #1 to #n (step S1407).

[0158] Next, the image processing device 201 executes MVS processing on the captured image group #1 to #n based on the estimated camera position and orientation to generate a 3D model of the object (step S1408). Then, the image processing device 201 outputs the generated 3D model of the object (step S1409) and ends a series of processes according to this flowchart.

[0159] As a result, the image processing device 201 can improve the accuracy of estimating the camera position and generate a three-dimensional model of the object with high accuracy, even if the object has a monotonous color.

[0160] Next, a specific processing procedure of the representative image base color calculation process in step S1402 shown in FIG. 14 will be described with reference to FIG.

[0161] Fig. 15 is a flowchart showing an example of a specific processing procedure of the representative image base color calculation process. In the flowchart of Fig. 15, first, the image processing device 201 sets the number of clusters k to "k=2" (step S1501), and performs RGB clustering processing on the representative image (step S1502).

[0162] Next, the image processing device 201 calculates the color distance of the base colors between different clusters (step S1503). The color distance is calculated for all combinations of the clusters of the cluster number k. Then, the image processing device 201 determines whether the minimum value of the calculated color distance of the base colors between different clusters is less than a threshold value X (step S1504).

[0163] If the minimum color distance is equal to or greater than the threshold value X (step S1504: No), the image processing device 201 increments the number of clusters k (step S1505) and returns to step S1502. On the other hand, if the minimum color distance is less than the threshold value X (step S1504: Yes), the image processing device 201 decrements the number of clusters k and identifies it as the number of color gradient clusters (step S1506).

[0164] Next, the image processing device 201 specifies the base color of each cluster obtained by classifying the pixels in the representative image by the color gradient cluster number as the representative image base color (step S1507). At this time, the image processing device 201 may exclude the base color of a cluster corresponding to a specific color from the base colors of each cluster obtained by classifying the pixels in the representative image by the color gradient cluster number from the representative image base color. Then, the image processing device 201 returns to the step that called the representative image base color calculation process.

[0165] This enables the image processing device 201 to extract a representative color that characterizes the object.

[0166] Next, a specific procedure for the replaced image generation process in step S1404 shown in FIG. 14 will be described with reference to FIG.

[0167] Fig. 16 is a flowchart showing an example of a specific processing procedure of the replaced image generation processing. In the flowchart of Fig. 16, first, the image processing device 201 specifies pixels of the specified representative image base color from among pixels in the selected captured image #i (step S1601). The pixels of the representative image base color are pixels whose color information (RGB values) matches the base color of each cluster specified as the representative image base color.

[0168] Next, the image processing device 201 identifies pixels of a color adjacent to the identified representative image base color from among the pixels in the selected captured image #i (step S1602). The pixels of a color adjacent to the representative image base color are pixels whose color information (RGB values) matches the color adjacent to the base color of each cluster identified as the representative image base color.

[0169] Then, the image processing device 201 generates a first replaced captured image #i in which pixels in the selected captured image #i other than those identified in steps S1601 and S1602 are replaced with a specific color (step S1603), and returns to the step that called the replaced image generation process.

[0170] In this way, the image processing device 201 can generate an image (first replaced captured image #i) in which the parts other than the object are replaced with a specific color (for example, black). However, even if it is a part other than the object, a part having a similar color to the object may remain without being replaced with the specific color.

[0171] Next, a specific procedure for the color reduction process in step S1405 shown in FIG. 14 will be described with reference to FIG.

[0172] Fig. 17 is a flowchart showing an example of a specific processing procedure of the color reduction processing. In the flowchart of Fig. 17, first, the image processing device 201 performs an RGB clustering process on the first replaced captured image #i generated in step S1603 shown in Fig. 16 by the number of color gradient clusters specified in step S1506 shown in Fig. 15 (step S1701).

[0173] Then, the image processing device 201 selects an unselected cluster from among the clusters of the number of classified color gradient clusters (step S1702). Next, the image processing device 201 identifies pixels classified into the selected cluster from among the pixel groups in the first replaced captured image #i (step S1703).

[0174] Then, the image processing device 201 replaces the identified pixel with the replaced captured image base color of the selected cluster (step S1704). Next, the image processing device 201 determines whether or not there is an unselected cluster among the number of color gradient clusters (step S1705).

[0175] If there is an unselected cluster (step S1705: Yes), the image processing device 201 returns to step S1702. On the other hand, if there is no unselected cluster (step S1705: No), the image processing device 201 proceeds to step S1706 since the second replaced captured image #i corresponding to the first replaced captured image #i has been generated.

[0176] Then, the image processing device 201 generates a third replaced captured image #i by returning the part replaced with the specific color in the generated second replaced captured image #i to the color before replacement (step S1706), and returns to the step that called the color reduction process.

[0177] In this way, the image processing device 201 can emphasize slight color changes by performing color reduction processing that deliberately reduces the color expression of the object captured in the captured image #i. Also, the image processing device 201 can make it easier to find feature points from the background by returning the background parts other than the object to their original colors.

[0178] As described above, according to the image processing device 201 of the embodiment, a representative image of an object is acquired, and RGB clustering processing is performed on the representative image while changing the cluster number k, so that the color gradient cluster number and the representative image base color can be calculated so that the cluster number k is as large as possible and the minimum value of the color distance of the base colors between different clusters is not less than the threshold value X. The representative image is an image (first image) in which parts other than the object in an image of the object are converted to a specific color. The specific color is, for example, black. The color gradient cluster number is the cluster number (first cluster number) for classifying the pixel group in the representative image. Moreover, the representative image base color corresponds to the base color (first base color) of each cluster obtained by classifying the pixel group in the representative image by the color gradient cluster number.

[0179] In this way, the image processing device 201 can extract colors that characterize the object while reducing the number of colors by clustering in order to emphasize color changes (color gradients) in the representative image. The representative image is preferably an image that shows the entire object in order to cover the colors used in the object.

[0180] Moreover, the image processing device 201 can obtain a group of captured images #1 to #n obtained by capturing an object from different angles, and generate a first replaced captured image #i by replacing (the color information of) a portion other than the color corresponding to the calculated representative image base color with a specific color for each captured image #i of the group of captured images #1 to #n. The color corresponding to the representative image base color is, for example, the representative image base color and a color adjacent to the representative image base color. The specific color is, for example, black.

[0181] As a result, the image processing device 201 can generate an image in which most of the parts other than the object are replaced with a specific color (for example, black) by assuming that the color corresponding to the representative image base color is the color that represents the object. Furthermore, the image processing device 201 can comprehensively cover the part that represents the object in the captured image #i by including colors adjacent to the representative image base color as the color that corresponds to the representative image base color, thereby preventing a part of the object from being replaced with a specific color. Furthermore, by setting the specific color to "black", for example, the image processing device 201 can make all RGB values ​​of the replaced parts (pixels) "0", thereby improving image processing efficiency.

[0182] Moreover, the image processing device 201 can calculate a replaced captured image base color by performing RGB clustering processing on the first replaced captured image #i of each of the generated first replaced captured images #1 to #n using the color gradient cluster number. The replaced captured image base color corresponds to the base color (second base color) of each cluster obtained by classifying the pixel group in the first replaced captured image #i using the color gradient cluster number. Then, the image processing device 201 can generate a second replaced captured image #i by replacing the pixels classified into each cluster of the color gradient cluster number with the replaced captured image base color of each cluster for each of the first replaced captured images #1 to #n.

[0183] As a result, the image processing device 201 can enhance the accuracy of extracting feature points from an image by performing a color reduction process on the first-replaced captured image #i that intentionally reduces the expressiveness of colors, thereby highlighting color changes (color gradients) in the image. For example, the image processing device 201 can generate an image (second-replaced captured image #i) from which slight color changes such as uneven paint application, dirt, and scratches on the surface of the object can be extracted as feature points, even if the object is an object of a monotonous color. Furthermore, since most of the objects other than the object are replaced with a specific color (e.g., black) in the first-replaced captured image #i, the image processing device 201 can narrow down the target of the color reduction process to the object.

[0184] Moreover, according to the image processing device 201, it is possible to perform a camera position estimation process for the captured image group #1 to #n based on the generated second replaced captured image group #1 to #n. The camera position estimation process is a process (SFM process) for estimating the position and orientation of the camera (imaging device) that captured each of the captured image groups #1 to #n.

[0185] This enables the image processing device 201 to capture slight color changes, such as uneven paint application or scratches, as feature points, and accurately perform feature point matching (matching feature points that indicate the same location) between multiple images, thereby improving the accuracy of estimating the camera position and orientation.

[0186] Moreover, according to the image processing device 201, for each of the generated second replaced captured images #i in the group of second replaced captured images #1 to #n, it is possible to generate a third replaced captured image #i by returning a portion in the second replaced captured image #i that has been replaced with a specific color to the color before replacement.

[0187] As a result, the image processing device 201 can return the parts (e.g., background parts) that have been replaced with a specific color (e.g., black) to their original colors, making it easier to find feature points from the background parts, etc., and further improving the estimation accuracy of the camera position and orientation.

[0188] Moreover, according to the image processing device 201, it is possible to perform the camera position estimation process for the group of taken images #1 to #n based on the generated third replaced group of taken images #1 to #n.

[0189] This enables the image processing device 201 to extract a greater number of effective feature points that match between a plurality of images, thereby improving the accuracy of estimating the position and orientation of the camera.

[0190] Moreover, the image processing device 201 can generate a 3D model of the object by executing MVS processing on the captured image groups #1 to #n based on the estimated positions and orientations of the cameras that captured each of the captured image groups #1 to #n. Then, the image processing device 201 can output the generated 3D model of the object in association with the captured image groups #1 to #n.

[0191] As a result, the image processing device 201 can generate a 3D model with high accuracy even if the object has a monotonous color. Furthermore, the image processing device 201 can output the generated 3D model of the object to the client device 202 or the like.

[0192] Moreover, according to the image processing device 201, it is possible to output the generated second replaced groups of taken pictures #1 to #n in association with the groups of taken pictures #1 to #n.

[0193] This enables the image processing device 201 to execute the camera position estimation process (SFM process) and the MVS process based on the second replaced captured image groups #1 to #n in a computer different from the information processing device 101.

[0194] Moreover, the image processing device 201 can output the generated third replaced groups of taken pictures #1 to #n in association with the groups of taken pictures #1 to #n.

[0195] This enables the image processing device 201 to execute the camera position estimation process (SFM process) and the MVS process based on the third replaced captured image groups #1 to #n in a computer different from the information processing device 101.

[0196] Moreover, according to the image processing device 201, by performing RGB clustering processing while gradually increasing the number of clusters k from 2 until the minimum value of the color distance of the base colors between different clusters becomes less than the threshold value X, it is possible to determine the number of color gradient clusters by subtracting 1 from the number of clusters k when the number of clusters becomes less than the threshold value X. Then, according to the image processing device 201, it is possible to specify the base color of each cluster obtained by classifying the pixel group in the representative image by the determined number of color gradient clusters as the representative image base color.

[0197] This allows the image processing device 201 to efficiently calculate the color gradient cluster number and the representative image base color.

[0198] In addition, according to the image processing device 201, the base colors of each cluster obtained by classifying the pixel groups in the representative image by the color gradient cluster number, excluding a specific color and colors adjacent to the specific color, may be identified as the representative image base color.

[0199] In this way, the image processing device 201 can prevent the color of the portion that has been replaced with a specific color in the representative image from being identified as a color that characterizes the object.

[0200] From these facts, the image processing device 201 can increase the number of effective feature points by highlighting slight color changes even in an image with little color change. For example, the image processing device 201 can capture even slight color changes such as stains or uneven paint application as feature points, even in structures (objects) with monotonous colors such as steel towers, bridges, and dams. As a result, the image processing device 201 can accurately match feature points between multiple images of an object captured from different angles, reduce errors in SFM processing, and improve the accuracy of generating a 3D model of the object.

[0201] The information processing system 200 is applied to a service that generates a 3D model of an object by uploading a group of images taken from various angles by a drone, etc. For example, by using the service when constructing a digital twin, even a structure with a monotonous color can be made into a 3D model with high accuracy.

[0202] The image processing method described in the present embodiment can be realized by executing a prepared program on a computer such as a personal computer or a workstation. The image processing program is recorded on a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, a DVD, or a USB memory, and is executed by being read from the recording medium by the computer. The image processing program may also be distributed via a network such as the Internet.

[0203] In addition, the information processing device 101 (image processing device 201) described in this embodiment can also be realized by an application-specific IC such as a standard cell or a structured ASIC (Application Specific Integrated Circuit) or a PLD (Programmable Logic Device) such as an FPGA.

[0204] The following supplementary notes are further disclosed regarding the above-described embodiment.

[0205] (Note 1) A first image in which parts of an image of an object other than the object are converted to a specific color, and a plurality of images of the object taken from different angles are obtained, performing clustering to classify pixels in the first image into clusters according to the number of clusters while changing the number of clusters based on pixel values ​​of each pixel in the acquired first image, thereby calculating a first cluster number for classifying pixels in the first image and a first base color of each cluster obtained by classifying pixels in the first image by the first cluster number, so that the number of clusters is as large as possible and the minimum value of the color distance of the base colors between different clusters is not less than a threshold value; generating a first replaced image by replacing a portion of each of the acquired images other than a color corresponding to the calculated first base color of each of the clusters with a specific color; performing clustering for each of the generated first replaced images to classify pixels in the first replaced image into clusters of the first cluster number based on pixel values ​​of each pixel in the first replaced image, calculating a second base color for each cluster of the first cluster number thus classified, and generating a second replaced image in which the pixels classified into each cluster are replaced with the second base color of the cluster; An image processing program that causes a computer to execute a process.

[0206] (Additional Note 2) A camera position estimation process is performed to estimate the position and orientation of a camera that captured each of the plurality of images based on each of the generated second replaced images. 2. The image processing program according to claim 1, which causes the computer to execute the process.

[0207] (Additional Note 3) For each of the generated second replaced images, a third replaced image is generated by returning the portion in the second replaced image replaced with the specific color to the color before replacement. 3. The image processing program according to claim 1 or 2, characterized in that the program causes the computer to execute the process.

[0208] (Additional Note 4) A camera position estimation process is performed to estimate a position and an orientation of a camera that captured each of the plurality of images based on each of the generated third replaced images. 4. The image processing program according to claim 3, which causes the computer to execute the process.

[0209] (Additional Note 5) The generated second replaced images are output in association with the plurality of images. 5. The image processing program according to claim 1, wherein the image processing program causes the computer to execute the process.

[0210] (Additional Note 6) The generated third permuted images are output in association with the plurality of images. 5. The image processing program according to claim 3 or 4, which causes the computer to execute the process.

[0211] (Appendix 7) A three-dimensional model of the object is generated by performing MVS (Multi-View Stereo) processing on the plurality of images based on the estimated positions and orientations of the cameras that captured each of the plurality of images. 5. The image processing program according to claim 2 or 4, which causes the computer to execute the process.

[0212] (Appendix 8) The generated 3D model is output in association with the plurality of images. 8. The image processing program according to claim 7, which causes the computer to execute a process.

[0213] (Supplementary Note 9) The image processing program according to any one of Supplementary Notes 1 to 8, wherein the colors corresponding to the first base color include the first base color and colors adjacent to the first base color.

[0214] (Supplementary Note 10) The process of calculating the first cluster number and the first base color includes: performing the clustering while successively increasing the number of clusters from 2 until the minimum value of the color distance of the base colors between different clusters becomes less than the threshold value, thereby determining the number of clusters obtained by subtracting 1 from the number of clusters when the minimum value of the color distance of the base colors between different clusters becomes less than the threshold value as the first number of clusters; 10. The image processing program according to any one of claims 1 to 9, further comprising identifying a first base color of each cluster obtained by classifying a group of pixels in the first image using the determined number of first clusters.

[0215] (Appendix 11) The process of generating the first replaced image includes: The image processing program according to any one of appendices 1 to 10, characterized in that a first replacement image is generated in which parts of a first base color of each cluster other than the specific color and colors corresponding to the base color excluding the specific color and colors neighboring the specific color are replaced with the specific color.

[0216] (Appendix 12) A first image in which a part of an image of an object other than the object is converted to a specific color, and a plurality of images of the object taken from different angles are obtained, performing clustering to classify pixels in the first image into clusters according to the number of clusters while changing the number of clusters based on pixel values ​​of each pixel in the acquired first image, thereby calculating a first cluster number for classifying pixels in the first image and a first base color of each cluster obtained by classifying pixels in the first image by the first cluster number, so that the number of clusters is as large as possible and the minimum value of the color distance of the base colors between different clusters is not less than a threshold value; generating a first replaced image by replacing a portion of each of the acquired images other than a color corresponding to the calculated first base color of each of the clusters with a specific color; performing clustering for each of the generated first replaced images to classify pixels in the first replaced image into clusters of the first cluster number based on pixel values ​​of each pixel in the first replaced image, calculating a second base color for each cluster of the first cluster number thus classified, and generating a second replaced image in which the pixels classified into each cluster are replaced with the second base color of the cluster; An image processing method, the processing being carried out by a computer.

[0217] (Appendix 13) A first image in which a part of an image of an object other than the object is converted to a specific color, and a plurality of images of the object taken from different angles are obtained, performing clustering to classify pixels in the first image into clusters according to the number of clusters while changing the number of clusters based on pixel values ​​of each pixel in the acquired first image, thereby calculating a first cluster number for classifying pixels in the first image and a first base color of each cluster obtained by classifying pixels in the first image by the first cluster number, so that the number of clusters is as large as possible and the minimum value of the color distance of the base colors between different clusters is not less than a threshold value; generating a first replaced image by replacing a portion of each of the acquired images other than a color corresponding to the calculated first base color of each of the clusters with a specific color; performing clustering for each of the generated first replaced images to classify pixels in the first replaced image into clusters of the first cluster number based on pixel values ​​of each pixel in the first replaced image, calculating a second base color for each cluster of the first cluster number thus classified, and generating a second replaced image in which the pixels classified into each cluster are replaced with the second base color of the cluster; An information processing device comprising a control unit. [Explanation of symbols]

[0218] 101 Information processing device 102 Object 110 1st image 120 Multiple Images 130 Multiple First Replacement Images 140 Multiple Secondary Replacement Images 150,160 color images 200 Information Processing Systems 201 Image processing device 202 Client device 210 Network 220 Representative Image Management Table 230 Captured image management table 300 Bus 301 CPU 302 Memory 303 Disk Drive 304 Disk 305 Communication I / F 306 Portable Recording Media I / F 307 Portable recording media 600 Control section 601 Acquisition Department 602 First Calculation Section 603 First Generation Section 604 Second Calculation Section 605 Second Generation 606 Third Generation 607 Execution Control Unit 608 Output section 701, 1301, 1302 Captured images 702 Representative Image 900 captured images 901 First replacement captured image group 1001 First replacement captured image 1100 Second replacement captured image group 1101 Second replacement captured image 1200 Third replacement captured image group 1201 Third replacement image

Claims

1. Obtaining a first image in which a portion of an image of an object other than the object is converted to a specific color, and a plurality of images of the object taken from different angles; performing clustering to classify pixels in the first image into clusters according to the number of clusters while changing the number of clusters based on pixel values ​​of each pixel in the acquired first image, thereby calculating a first cluster number for classifying pixels in the first image and a first base color of each cluster obtained by classifying pixels in the first image by the first cluster number, so that the number of clusters is as large as possible and the minimum value of the color distance of the base colors between different clusters is not less than a threshold value; generating a first replaced image by replacing a portion of each of the acquired images other than a color corresponding to the calculated first base color of each of the clusters with a specific color; performing clustering for each of the generated first replaced images to classify pixels in the first replaced image into clusters of the first cluster number based on pixel values ​​of each pixel in the first replaced image, calculating a second base color for each cluster of the first cluster number thus classified, and generating a second replaced image in which the pixels classified into each cluster are replaced with the second base color of the corresponding cluster; An image processing program that causes a computer to execute a process.

2. performing a camera position estimation process for estimating a position and an orientation of a camera that captured each of the plurality of images based on each of the generated second replaced images; 2. The image processing program according to claim 1, which causes the computer to execute processing.

3. generating a third replaced image by returning the portion of the second replaced image that has been replaced with the specific color to the color before replacement for each of the generated second replaced images; 2. The image processing program according to claim 1, which causes the computer to execute processing.

4. performing a camera position estimation process for estimating a position and an orientation of a camera that captured each of the plurality of images based on each of the generated third replaced images; 4. The image processing program according to claim 3, which causes the computer to execute processing.

5. outputting the generated second replaced images in association with the plurality of images; 2. The image processing program according to claim 1, which causes the computer to execute processing.

6. outputting the generated third permuted images in association with the plurality of images; 4. The image processing program according to claim 3, which causes the computer to execute processing.

7. Obtaining a first image in which a portion of an image of an object other than the object is converted to a specific color, and a plurality of images of the object taken from different angles; performing clustering to classify pixels in the first image into clusters according to the number of clusters while changing the number of clusters based on pixel values ​​of each pixel in the acquired first image, thereby calculating a first cluster number for classifying pixels in the first image and a first base color of each cluster obtained by classifying pixels in the first image by the first cluster number, so that the number of clusters is as large as possible and the minimum value of the color distance of the base colors between different clusters is not less than a threshold value; generating a first replaced image by replacing a portion of each of the acquired images other than a color corresponding to the calculated first base color of each of the clusters with a specific color; performing clustering for each of the generated first replaced images to classify pixels in the first replaced image into clusters of the first cluster number based on pixel values ​​of each pixel in the first replaced image, calculating a second base color for each of the classified clusters of the first cluster number, and generating a second replaced image in which the pixels classified into each cluster are replaced with the second base color of the cluster; An image processing method, the processing being carried out by a computer.

8. Obtaining a first image in which a portion of an image of an object other than the object is converted to a specific color, and a plurality of images of the object taken from different angles; performing clustering to classify pixels in the first image into clusters according to the number of clusters while changing the number of clusters based on pixel values ​​of each pixel in the acquired first image, thereby calculating a first cluster number for classifying pixels in the first image and a first base color of each cluster obtained by classifying pixels in the first image by the first cluster number, so that the number of clusters is as large as possible and the minimum value of the color distance of the base colors between different clusters is not less than a threshold value; generating a first replaced image by replacing a portion of each of the acquired images other than a color corresponding to the calculated first base color of each of the clusters with a specific color; performing clustering for each of the generated first replaced images to classify pixels in the first replaced image into clusters of the first cluster number based on pixel values ​​of each pixel in the first replaced image, calculating a second base color for each of the classified clusters of the first cluster number, and generating a second replaced image in which the pixels classified into each cluster are replaced with the second base color of the cluster; An information processing device comprising a control unit.

Images