Image processing device, display system, image processing method, and image processing program

Abstract

To provide an image processing device or the like capable of appropriately displaying a three-dimensional model even when the model includes a dark portion with low visibility.SOLUTION: An image processing device comprises: three-dimensional point group data acquisition means for acquiring three-dimensional point group data of an object; two-dimensional image acquisition means for acquiring two-dimensional images of the object captured from different directions; two-dimensional image contour extraction means for extracting a contour in each two-dimensional image; two-dimensional image closed region extraction means for extracting the closed region surrounded by the contour; average brightness calculation means for calculating the average brightness of each closed region; color coordinate association means for associating the color of a two-dimensional image with coordinates of three-dimensional point group data; object display direction control means for controlling the display direction of the object; display color data selection means for selecting color data applied to each closed region in display on the basis of the average brightness; and display control means for synthesizing colored closed regions and displaying the image of the object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, a display system, an image processing method, and an image processing program. [Background technology]

[0002] In recent years, digitization of object shapes using 3D scanners has become popular. The data acquired here is a collection of point data called point cloud data (3D point cloud data). With 3D point cloud data, there is a problem that it is difficult for people looking at the displayed image to recognize the measurement target because color cannot be obtained. Therefore, a method of adding color data to 3D point cloud data by using a separately captured 2D image is being studied.

[0003] For example, Patent Document 1 discloses a method of obtaining the relative positional relationship between a camera and a laser scanner from a two-dimensional color image, and corresponding color data of the image to three-dimensional point cloud data based on this positional relationship. In this method, first, three-dimensional point cloud data of a measurement target is obtained by a laser scanner. Also, a two-dimensional color image is obtained by photographing the measurement target. Next, three or more points are arbitrarily selected in the two-dimensional color image. Next, three-dimensional position information based on the three-dimensional point cloud data is given to each of the selected points. Next, the relative positional relationship between the camera and the laser scanner at the time of photographing the measurement target is obtained based on the three-dimensional position information of the selected points. Then, based on the obtained relative positional relationship and the three-dimensional position information of the selected points, the color data of the two-dimensional color image is corresponding to the data of each point of the three-dimensional point cloud data. Through the above-mentioned procedure, color information is added to the three-dimensional point cloud data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-077385 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology of Patent Document 1, a two-dimensional image of a three-dimensional object is captured from one direction, which results in dark, low-visibility areas that become shadows during capture. In order to recognize and properly display the entire object, it is necessary to perform a shadow removal process on the entire low-visibility area.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide an image processing device and the like that can properly display a three-dimensional model even if there are dark areas with low visibility. [Means for solving the problem]

[0007] In order to solve the above problems, the image processing device of the present disclosure includes a 3D point cloud data acquisition means for acquiring 3D point cloud data of an object, a 2D image acquisition means for acquiring a plurality of 2D images of the object photographed from mutually different directions, a 2D image contour extraction means for extracting contours included in each of the 2D images, a 2D image closed region extraction means for extracting a closed region surrounded by the contours in each of the 2D images, an average brightness calculation means for calculating an average brightness value of the brightness in the closed region, and a color data of each point of each of the 2D images and the 3D The system includes a color coordinate matching means for matching the coordinate data of the original point cloud data, an object display direction control means for controlling the direction of the object to be displayed when the color data is added to the 3D point cloud data of the object and displayed, a display color data selection means for selecting, when displaying the object, which of the 2D images the color data of which is to be used as the color data of each of the closed regions included in the display based on the average lightness, and a display control means for synthesizing the color data of each of the selected closed regions to display an image of the object on a display device.

[0008] A display system according to the present disclosure includes the image processing device described above, and a display that displays an image of the object output by the image processing device.

[0009] In addition, the image processing method disclosed herein includes a computer acquiring 3D point cloud data of an object, acquiring multiple 2D images of the object taken from different directions, extracting contours contained in each of the 2D images, extracting closed areas surrounded by the contours in each of the 2D images, calculating the average brightness of the brightness in the closed areas, matching color data of each point in each of the 2D images with coordinate data of the 3D point cloud data, controlling the orientation of the object to be displayed when adding the color data to the 3D point cloud data of the object and displaying it, selecting, based on the average brightness, which of the 2D images the color data of which will be used as the color data of each of the closed areas included in the display, and combining the color data of each of the selected closed areas to display an image of the object on a display device.

[0010] In addition, the image processing program of the present disclosure causes a computer to execute the following processes: acquiring 3D point cloud data of an object; acquiring multiple 2D images of the object photographed from different directions; extracting contours contained in each of the 2D images; extracting closed areas surrounded by the contours in each of the 2D images; calculating an average value of brightness in the closed areas as an average brightness; matching color data of each point in each of the 2D images with coordinate data of the 3D point cloud data; controlling the orientation of the object to be displayed when the color data is added to the 3D point cloud data of the object and displayed; selecting, based on the average brightness, which of the 2D images the color data of which will be used as the color data of each of the closed areas included in the display when displaying the object; and combining the color data of each of the selected closed areas to display an image of the object on a display device. Effect of the Invention

[0011] An advantage of the present invention is that it is possible to provide an image processing device or the like that can properly display a three-dimensional model even if there are dark areas with low visibility. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating an example of an image processing device according to the present disclosure. [Diagram 2] 10 is a flowchart illustrating a portion of an example of the operation of the image processing device of the present disclosure. [Diagram 3] 13 is a flowchart illustrating another part of the example of the operation of the image processing device of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram showing an example of contour fitting according to the present disclosure. [Diagram 5] FIG. 1 is a schematic diagram showing an example of closed region synthesis according to the present disclosure. [Figure 6] FIG. 1 is a block diagram illustrating an example of a display system of the present disclosure. [Figure 7] FIG. 2 is a block diagram showing an example hardware configuration of a display system according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. However, the embodiments described below are limited in a technically preferable manner for carrying out the present invention, but the scope of the invention is not limited to the following. Note that the same components in each drawing are given the same numbers, and the description may be omitted.

[0014] (First embodiment) 1 is a block diagram showing an example of an image processing device 100 of the present disclosure. The image processing device 100 includes a 3D point cloud data acquisition means 110, a 2D image acquisition means 120, a 2D image contour extraction means 130, a 2D image closed region extraction means 140, and an average brightness calculation means 150. The image processing device 100 also includes a color coordinate matching means 160, an object display direction control means 170, a display color data selection means 180, and a display control means 190.

[0015] A three-dimensional point cloud data acquisition means 110 acquires three-dimensional point cloud data of an object.

[0016] A two-dimensional image acquisition means 120 acquires a plurality of two-dimensional images of an object photographed from different directions.

[0017] A two-dimensional image contour extraction means 130 extracts contours contained in each of the two-dimensional images.

[0018] A two-dimensional image closed region extraction means 140 extracts a closed region surrounded by a contour in each of the two-dimensional images.

[0019] The average brightness calculation means 150 calculates the average brightness in the closed region.

[0020] A color coordinate matching means 160 matches the color data of each point in each two-dimensional image with the coordinate data of the three-dimensional point cloud data.

[0021] When color data is added to the three-dimensional point cloud data of an object and the object is displayed, the object display direction control means 170 controls the direction of the object to be displayed.

[0022] When displaying an object, a display color data selection means 180 selects which of the two-dimensional images should use color data for each closed region included in the display based on the average brightness. Then, the color data for each closed region is synthesized. Then, a display control means 190 causes the display device to display the image of the object.

[0023] Next, the operation of the image processing device 100 will be described. Fig. 2 is a flowchart showing a part of an example of the operation of the image processing device of the present disclosure. Fig. 3 is a flowchart showing another part of an example of the operation of the image processing device of the present disclosure. The image processing device 100 performs processing for displaying an image of a three-dimensional model on a display device. Fig. 2 shows the previous part of the operation. And Fig. 3 shows the subsequent part of the operation.

[0024] First, the three-dimensional point cloud data acquisition means 110 acquires three-dimensional point cloud data of the object (S101). The three-dimensional point cloud data of the object is generated by scanning the object with a three-dimensional scanner such as a laser scanner. Then, the three-dimensional point cloud data acquisition means 110 acquires the generated three-dimensional point cloud data. Each point of the three-dimensional point cloud data is associated with a coordinate with a predetermined origin as a reference. Next, the two-dimensional image acquisition means 120 acquires a two-dimensional image of the object. Here, a plurality of two-dimensional images taken from different directions are acquired (S102). The two-dimensional images are, for example, color images. Next, the image processing device 100 associates the two-dimensional images with the three-dimensional point cloud data. This process is executed sequentially for each two-dimensional image. Then, for example, the process is executed as a loop process until the process for the selected two-dimensional image is completed (L1).

[0025] Next, the details of this loop process will be described. First, the two-dimensional image contour extraction means 130 extracts the contours included in the two-dimensional image (S103). Next, the two-dimensional image closed region extraction means 140 extracts the closed region surrounded by the contours (S104). Next, the average brightness calculation means 150 calculates the average brightness for each closed region (S105). Here, the average brightness is the average brightness of each pixel in the closed region. For example, in a color image, the brightness of (R, G, B) is obtained for each pixel. Here, R, G, B represent red, green, and blue. The average brightness is calculated, for example, by the following formula.

[0026] (Average brightness)=(ΣRk+ΣGk+ΣBk) / 3n (Equation 1) Here, k is the identification number of the pixel, Rk, Gk, and Bk are the R, G, and B lightness of the pixel, and n is the number of pixels in the closed region.

[0027] Next, the contour of the two-dimensional image and the contour of the three-dimensional image seen from the shooting direction are fitted (S106). FIG. 4 is a schematic diagram showing an example of contour fitting of the present disclosure. The upper left of FIG. 4 shows a two-dimensional image. In this example, there is a shadow d at the right end of the object o. In this part, the color of the object is not properly reflected in the actual data. In addition, the upper right of FIG. 4 shows three-dimensional point cloud data. Three-dimensional point cloud data has no color. Therefore, the difference in color in the part without unevenness is not reflected. On the other hand, the contour of the outer shape of the object o is clearly shown in the three-dimensional point cloud data. Therefore, by setting a specific part of the contour of the outer shape as a reference contour, it is possible to fit the contour of the two-dimensional image to the contour of the three-dimensional point cloud data.

[0028] In the example of Fig. 4, the contour of the lower left part of the object o is selected. This part in the two-dimensional image is set as the reference contour r. Also, this part in the three-dimensional point cloud data is set as the reference contour s. By fitting the reference contour r and the reference contour s, it becomes possible to associate each point of the two-dimensional image with the coordinates of the three-dimensional point cloud data. Through such an operation, the color coordinate association means 160 associates the color data of the two-dimensional image with the coordinates of the three-dimensional point cloud data (S107).

[0029] In this manner, the coordinates and color are set for each closed region. The average brightness of each closed region is also calculated. The color data with coordinates and the average brightness obtained here are saved (S108).

[0030] Next, the operation of the display control will be described. Fig. 3 shows the operation following Fig. 2(A). The object display direction control means 170 controls the direction of the object to be displayed on the display device (S201). This direction is specified by, for example, a user input.

[0031] The closed region to be displayed is determined by the above-mentioned directional control. According to the result, the display color data selection means 180 extracts two-dimensional images including the closed region to be displayed (S202). Next, the display color data selection means 180 groups the extracted two-dimensional images (S203).

[0032] Next, for one closed region, the display color data selection means 180 compares the average brightness of each closed region of the two-dimensional images (S204). Then, for example, the display color data selection means 180 selects color data of a two-dimensional image whose average brightness is equal to or greater than a predetermined threshold value. The threshold value is determined, for example, based on visibility. When there are multiple candidates, for example, the color data of the two-dimensional image with the highest average brightness in the closed region is adopted as the display data. This determines the color data to be assigned to the three-dimensional point cloud data of the closed region. Then, for the other closed regions to be displayed, color data is similarly assigned to the three-dimensional point cloud data.

[0033] Next, the color data with coordinates of each closed region adopted in the above operation is synthesized by the display control means 190. In other words, the color data with coordinates of each selected closed region is patched together to generate display data of the object. Using this data, the display control means 190 controls the display (S206).

[0034] According to the above operation, color data with coordinates having a high average brightness is added to all closed areas of an object displayed as a 3D model. Therefore, even if there are dark and low-visibility parts in one image, the parts are replaced with a color with high visibility. As a result, the 3D model is displayed appropriately.

[0035] (Example 1) FIG. 5 is a schematic diagram showing an example of closed region synthesis according to the present disclosure. In the two-dimensional image a in the upper left of FIG. 5, a shadow d is present at the right end of the object o, reducing visibility. On the other hand, the same part is clearly captured in the two-dimensional image b in the upper right of FIG. 5. Therefore, the color data of the two-dimensional image b is adopted as the color data of the closed region including that part. As a result, a three-dimensional model with the shadow removed is displayed, as shown in the "synthesized colored point cloud data" in the lower part of FIG. 5.

[0036] (Variation 1) In the above example, the color data of the selected two-dimensional image is directly assigned to the three-dimensional point cloud data of a certain closed region. However, processed color data may be assigned to the closed region. For example, the color data assigned to one closed region may be replaced with an average color obtained by averaging the colors in the closed region. This improves the visibility of the shape. Note that, even in this case, it is not necessary to replace all closed regions with the average color. Some closed regions may be monochromatized with the average color, and the color data of the two-dimensional image may be directly assigned to other closed regions.

[0037] A display system can be configured by combining the image processing device 100 described above with a display. Fig. 6 is a block diagram showing an example of a display system 1000 according to the present disclosure. The display system 1000 includes the image processing device 100 and a display 200. A three-dimensional model is displayed on the display 200 according to display control data output from the image processing device 100.

[0038] FIG. 5 is a block diagram showing an example of a hardware configuration of the display system 1000 of the present disclosure. The display system 1000 is implemented in, for example, a general computer. This computer includes, for example, a processor 1100, a memory 1200, a display unit 1300, an input / output unit 1400, a storage unit 1500, and a communication interface 1600. The processor 1100 performs calculations required for image processing. The memory 1200 stores data used in the calculations. The display unit 1300 displays data and images. The input / output unit 1400 is a device that inputs and outputs data. The input / output unit 1400 includes, for example, a keyboard, a mouse, an audio signal output unit, and a print signal output unit. The storage unit 1500 stores data. The communication interface 1600 performs interface processing for communication. Image processing and display are possible with the above hardware configuration.

[0039] The image processing device 100 and the like of the present disclosure have been described above. The image processing device 100 of the present disclosure includes a three-dimensional point cloud data acquisition means 110, a two-dimensional image acquisition means 120, a two-dimensional image contour extraction means 130, a two-dimensional image closed region extraction means 140, and an average brightness calculation means 150. The image processing device 100 also includes a color coordinate matching means 160, an object display direction control means 170, a display color data selection means 180, and a display control means 190. The three-dimensional point cloud data acquisition means 110 acquires three-dimensional point cloud data of an object. The two-dimensional image acquisition means 120 acquires a plurality of two-dimensional images of the object photographed from different directions. The two-dimensional image contour extraction means 130 extracts contours included in each of the two-dimensional images. The two-dimensional image closed region extraction means 140 extracts closed regions surrounded by contours in each of the two-dimensional images. The average brightness value in the closed regions is calculated as the average brightness by the average brightness calculation means 150. The color data of each point of each two-dimensional image is associated with the coordinate data of the three-dimensional point cloud data by the color coordinate association means 160. When the display device assigns color data to the three-dimensional point cloud data and displays it, the object display direction control means 170 controls the direction of the displayed object. When displaying the object, the display color data selection means 180 selects, based on the average brightness, which two-dimensional image's color data should be used as the color data of each closed region included in the display. Then, the color data of each closed region is synthesized. Then, the display control means 190 causes the display device to display the image of the object.

[0040] In the above configuration, when displaying a 3D model, color data with good visibility is added to the 3D point cloud data that displays each closed area, so that an appropriate 3D model is displayed with poorly visible shadows removed.

[0041] According to one embodiment, in the image processing device 100, the display color data selection means 180 selects color data of a two-dimensional image to be used for each closed region. At this time, color data in which the average brightness of the closed region is equal to or greater than a predetermined threshold value is selected as the color data.

[0042] In the above configuration, for each closed region, color data with an average brightness equal to or greater than a threshold is selected from among multiple two-dimensional images, resulting in an appropriate three-dimensional model with good visibility.

[0043] According to one embodiment, in the image processing device 100, the display color data selection means 180 selects color data of a two-dimensional image to be used for each closed region. At this time, the color data of the two-dimensional image with the highest average brightness of the closed region is selected as the color data. Therefore, an appropriate three-dimensional model with good visibility is displayed.

[0044] According to one embodiment, in the image processing device 100, the display control means 190 controls color data for displaying a closed region. At this time, the color data of the two-dimensional image selected in the closed region is replaced with an average color obtained by averaging the colors in the closed region.

[0045] In the above configuration, each closed region is displayed in its average color, which makes it possible to display the region without shading.

[0046] According to one embodiment, in the image processing device 100, the display control means 190 generates colored three-dimensional point cloud data. In the colored three-dimensional point cloud data, selected color data is assigned to the three-dimensional point cloud data in each closed region.

[0047] In the above configuration, color data of a 2D image with good visibility is selected and the color data is added to the 3D point cloud data, which makes it possible to display an appropriate 3D model with good visibility.

[0048] According to one embodiment, in the image processing device 100, the color coordinate matching means 160 matches the color data with the three-dimensional point cloud data. At this time, the contour extracted from the two-dimensional image and the three-dimensional point cloud data corresponding to the contour are specified. Then, the color data is matched with the three-dimensional point cloud data according to the specified correspondence.

[0049] In the above configuration, the color data and the three-dimensional point cloud data are associated with each other based on the contours of the two-dimensional image, so that the color data of the two-dimensional image is correctly associated with the three-dimensional point cloud data.

[0050] Moreover, the display system of the present disclosure includes an image processing device 100 and a display 200. The display 200 displays an image of an object output by the image processing device.

[0051] In the above configuration, image data from which shading has been removed is output from the image processing device 100 to the display 200. This makes it possible to display a three-dimensional model from which shading has been removed.

[0052] In the image processing method of the present disclosure, the computer acquires three-dimensional point cloud data of the object. The computer acquires a plurality of two-dimensional images of the object captured from different directions. The computer extracts contours included in each of the two-dimensional images. The computer extracts closed regions surrounded by the contours in each of the two-dimensional images. The computer calculates the average value of the brightness in the closed regions as the average brightness. The computer associates color data of each point in each of the two-dimensional images with coordinate data of the three-dimensional point cloud data. The computer assigns color data to the three-dimensional point cloud data and displays it. At this time, the computer controls the direction of the object to be displayed. The computer selects color data of each closed region included in the display. At this time, the computer selects which of the two-dimensional images the color data is to be used based on the average brightness. The color data of each closed region is then synthesized. The computer displays the image of the object with the synthesized color data on a display device.

[0053] In the above configuration, when displaying a 3D model, color data with good visibility is added to the 3D point cloud data that displays each closed area, so that an appropriate 3D model is displayed with poorly visible shadows removed.

[0054] In the image processing program of the present disclosure, the program causes the computer to execute a process of acquiring three-dimensional point cloud data of an object. In addition, two-dimensional images of a plurality of objects are acquired by the computer. These two-dimensional images are photographed from different directions. In addition, the program causes the computer to execute a process of extracting contours included in each of the two-dimensional images. In addition, the program causes the computer to execute a process of extracting closed regions surrounded by contours in each of the two-dimensional images. In addition, the program causes the computer to execute a process of calculating an average value of brightness in the closed regions as an average brightness. In addition, the program causes the computer to execute a process of associating color data of each point of each of the two-dimensional images with coordinate data of the three-dimensional point cloud data. In addition, when the three-dimensional point cloud data is assigned with color data and displayed, the program causes the computer to execute a process of controlling the direction of the object to be displayed. In addition, when the object is displayed, a two-dimensional image to be used as color data of each closed region included in the display is selected. At this time, the two-dimensional image to be adopted is selected based on the average brightness. In addition, the color data of each closed region is synthesized. The program then causes the computer to execute a process of displaying the synthesized image of the object on a display device.

[0055] The scope of the present invention also includes a program for causing a computer to execute the processes of the above-described embodiment and a recording medium storing the program. As the recording medium, for example, a magnetic disk, a magnetic tape, an optical disk, a magneto-optical disk, a semiconductor memory, etc. can be used.

[0056] The present invention has been described above by taking the above-mentioned embodiment as an exemplary example. However, the present invention is not limited to the above-mentioned embodiment. That is, the present invention can be applied in various aspects that can be understood by a person skilled in the art within the scope of the present invention.

[0057] Some or all of the above embodiments may be described as follows: However, the scope of the present disclosure is not limited to the following.

[0058] (Appendix 1) A three-dimensional point cloud data acquisition means for acquiring three-dimensional point cloud data of an object; a two-dimensional image acquisition means for acquiring a plurality of two-dimensional images of the object photographed from different directions; a two-dimensional image contour extraction means for extracting contours included in each of the two-dimensional images; a two-dimensional image closed region extraction means for extracting a closed region surrounded by the contour in each of the two-dimensional images; an average brightness calculation means for calculating an average brightness of the brightness in the closed region; a color coordinate matching means for matching color data of each point of each of the two-dimensional images with coordinate data of the three-dimensional point cloud data; an object display direction control means for controlling a direction of the object to be displayed when the color data is added to the three-dimensional point cloud data and the three-dimensional point cloud data is displayed; a display color data selection means for selecting, based on the average brightness, which of the two-dimensional images the color data of which is to be used as the color data of each of the closed regions included in the display when the object is displayed; a display control means for synthesizing the color data of each of the selected closed regions to display an image of the object on a display device; 13. An image processing device comprising:

[0059] (Appendix 2) The display color data selection means selecting, as the color data to be used for each of the closed regions, the color data of the two-dimensional images whose average brightness is equal to or greater than a predetermined threshold value from among the two-dimensional images; 2. The image processing device according to claim 1 .

[0060] (Appendix 3) The display color data selection means selecting, as the color data to be used for each of the closed regions, the color data of the two-dimensional image having the highest average brightness among the two-dimensional images; 3. The image processing device according to claim 1 or 2.

[0061] (Appendix 4) The display control means replacing the color data of the two-dimensional image selected for displaying the closed region with an average color averaged within the closed region; 4. The image processing device according to claim 1,

[0062] (Appendix 5) The display control means generating colored three-dimensional point cloud data by adding the color data of each of the selected closed regions to the three-dimensional point cloud data; 5. The image processing device according to claim 1,

[0063] (Appendix 6) The color coordinate association means identifying the contour extracted from the two-dimensional image and the three-dimensional point cloud data corresponding to the contour, and associating the color data with the three-dimensional point cloud data; 6. The image processing device according to claim 1,

[0064] (Appendix 7) An image processing device according to any one of claims 1 to 6, a display that displays the image of the object output by the image processing device; A display system comprising:

[0065] (Appendix 8) The computer Acquire 3D point cloud data of the object, acquiring a plurality of two-dimensional images of the object taken from different directions; Extracting contours contained in each of the two-dimensional images; Extracting a closed region surrounded by the contour in each of the two-dimensional images; Calculating an average brightness value in the closed region as an average brightness value; Corresponding color data of each point of each of the two-dimensional images with coordinate data of the three-dimensional point cloud data; When the color data is added to the three-dimensional point cloud data and the three-dimensional point cloud data is displayed, a direction of the object to be displayed is controlled; selecting, based on the average brightness, which of the two-dimensional images the color data of which is to be used as the color data of each of the closed regions included in the display when displaying the object; synthesizing the color data of each of the selected closed regions to display an image of the object on a display device; 13. An image processing method comprising:

[0066] (Appendix 9) A process of acquiring 3D point cloud data of the object; acquiring a plurality of two-dimensional images of the object taken from different directions; A process of extracting contours included in each of the two-dimensional images; A process of extracting a closed region surrounded by the contour in each of the two-dimensional images; A process of calculating an average brightness value in the closed region as an average brightness value; A process of associating color data of each point of each of the two-dimensional images with coordinate data of the three-dimensional point cloud data; A process of controlling a direction of the object to be displayed when the color data is added to the three-dimensional point cloud data and displayed; a process of selecting, based on the average brightness, which of the two-dimensional images the color data of which is to be used as the color data of each of the closed regions included in the display when displaying the object; A process of displaying an image of the object on a display device by synthesizing the color data of each of the selected closed regions; An image processing program characterized by causing a computer to execute the above steps. [Explanation of symbols]

[0067] 100 Image processing device 110 3D point cloud data acquisition means 120 2D image acquisition means 130 2D image contour extraction means 140 2D image closed region extraction means 150 Average brightness calculation means 160 Color coordinate matching means 170 Object display direction control means 180 Display color data selection means 190 Display control means 200 Display 1000 Display System

Claims

1. A three-dimensional point cloud data acquisition means for acquiring three-dimensional point cloud data of an object; a two-dimensional image acquisition means for acquiring a plurality of two-dimensional images of the object photographed from different directions; a two-dimensional image contour extraction means for extracting contours included in each of the two-dimensional images; a two-dimensional image closed region extraction means for extracting a closed region surrounded by the contour in each of the two-dimensional images; an average brightness calculation means for calculating an average brightness of the brightness in the closed region; a color coordinate matching means for matching color data of each point of each of the two-dimensional images with coordinate data of the three-dimensional point cloud data; an object display direction control means for controlling a direction of the object to be displayed when the color data is added to the three-dimensional point cloud data and the three-dimensional point cloud data is displayed; a display color data selection means for selecting, based on the average brightness, which of the two-dimensional images the color data of which is to be used as the color data of each of the closed regions included in the display when the object is displayed; a display control means for synthesizing the color data of each of the selected closed regions to display an image of the object on a display device; 13. An image processing device comprising:

2. The display color data selection means selecting, as the color data to be used for each of the closed regions, the color data of the two-dimensional images whose average brightness is equal to or greater than a predetermined threshold value from among the two-dimensional images; 2. The image processing device according to claim 1,

3. The display color data selection means selecting, as the color data to be used for each of the closed regions, the color data of the two-dimensional image having the highest average brightness among the two-dimensional images; 2. The image processing device according to claim 1,

4. The display control means replacing the color data of the two-dimensional image selected for displaying the closed region with an average color averaged within the closed region; 4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

5. The display control means generating colored three-dimensional point cloud data by adding the color data of each of the selected closed regions to the three-dimensional point cloud data; 4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

6. The color coordinate association means identifying the contour extracted from the two-dimensional image and the three-dimensional point cloud data corresponding to the contour, and associating the color data with the three-dimensional point cloud data; 4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

7. An image processing device according to any one of claims 1 to 3, a display that displays the image of the object output by the image processing device; A display system comprising:

8. The computer Acquire 3D point cloud data of the object, acquiring a plurality of two-dimensional images of the object taken from different directions; Extracting contours contained in each of the two-dimensional images; Extracting a closed region surrounded by the contour in each of the two-dimensional images; Calculating an average brightness value in the closed region as an average brightness value; Corresponding color data of each point of each of the two-dimensional images with coordinate data of the three-dimensional point cloud data; When the color data is added to the three-dimensional point cloud data and the three-dimensional point cloud data is displayed, a direction of the object to be displayed is controlled; selecting, when displaying the object, which of the two-dimensional images the color data of which is to be used as the color data of each of the closed regions included in the display, based on the average lightness; synthesizing the color data of each of the selected closed regions to display an image of the object on a display device; 13. An image processing method comprising:

9. A process of acquiring three-dimensional point cloud data of an object; acquiring a plurality of two-dimensional images of the object taken from different directions; A process of extracting contours included in each of the two-dimensional images; A process of extracting a closed region surrounded by the contour in each of the two-dimensional images; A process of calculating an average brightness value in the closed region as an average brightness value; A process of associating color data of each point of each of the two-dimensional images with coordinate data of the three-dimensional point cloud data; A process of controlling a direction of the object to be displayed when the color data is added to the three-dimensional point cloud data and displayed; a process of selecting, based on the average brightness, which of the two-dimensional images the color data of which is to be used as the color data of each of the closed regions included in the display when displaying the object; A process of displaying an image of the object on a display device by synthesizing the color data of each of the selected closed regions; An image processing program characterized by causing a computer to execute the above steps.

Images