Data processing method and data processing system

Abstract

To obtain a technology for integrating a 3D model based on a visible image and a 3D model based on a thermal image.SOLUTION: There is provided a data processing method for creating an integrated 3D model in which a first 3D model based on a visible captured image and a second 3D model based on a thermal image are integrated. Common markers 101 to 103 are shown in the visible image and the thermal image, the first 3D model and the second 3D model are integrated with the markers 101 to 103 as a reference, the markers 101 to 103 have a stereoscopic structure, and the markers 101-103 are heated and / or cooled.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for integrating a 3D model based on a visible image and a 3D model based on a thermal image. [Background technology]

[0002] Patent Document 1 describes a technique for obtaining a three-dimensional image by superimposing a visible distance image and a far-infrared distance image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-32600 A Summary of the Invention [Problem to be solved by the invention]

[0004] It would be convenient to create a 3D model that integrates a 3D model (three-dimensional model) based on a visible image with a 3D model (three-dimensional model) based on a thermal image. For example, a 3D model based on a thermal image shows the three-dimensional heat distribution, but the correspondence with the actual shape is not clear. With the integrated 3D model, the relationship between the three-dimensional heat distribution and the shape can be visually grasped.

[0005] The technology described in Patent Document 1 is presumably a technology developed for the purpose of constructing the above-mentioned integrated 3D model. However, Patent Document 1 does not clearly explain how to integrate the two 3D models.

[0006] In this context, the present invention aims to provide a technique for integrating a 3D model based on a visible image and a 3D model based on a thermal image. [Means for solving the problem]

[0007] The present invention provides a data processing method for creating an integrated 3D model by integrating a first 3D model based on a visible light image and a second 3D model based on a thermal image, the method comprising: photograph A data processing method in which a common marker is captured in the image and the thermal image, the first 3D model and the second 3D model are integrated based on the marker, the marker has a three-dimensional structure, and the marker is heated and / or cooled.

[0008] In the present invention, the first 3D model is a digital representation of the external shape of the marker in three-dimensional space, and the second 3D model is a digital representation of the heat distribution of the marker in three-dimensional space, and an example of an embodiment is where the first 3D model and the second 3D model are integrated by comparing the shape and the heat distribution.

[0009] In the present invention, There are a plurality of the markers, In one embodiment, the plurality of markers have different sizes and are heated or cooled to different temperatures. The marker is An example of such an embodiment is one in which the device is partially heated or cooled, and has a heat distribution with a thermal gradient.

[0010] In the present invention, Marker The thermal gradient may be formed by having a structure in which a Peltier element is sandwiched between a first member and a second member, and by the Peltier element heating the first member and cooling the second member.

[0011] In the present invention, an embodiment in which the marker has an identification display pattern formed thereon that can be identified by differences in color and thermal radiation, or an embodiment in which the marker has a plurality of surfaces, the plurality of surfaces having different surface temperatures.

[0012] In the present invention, an embodiment is such that a sheet that changes color depending on temperature is attached to the marker, color information of the change is associated with the first 3D model, and temperature information of the sheet is associated with the second 3D model.

[0013] The present invention provides a data processing system for creating an integrated 3D model by integrating a first 3D model based on a visible image and a second 3D model based on a thermal image, the system comprising: photograph The image and the thermal image show a common marker, and an integrated 3D model creation unit is provided that integrates the first 3D model and the second 3D model based on the marker, and the marker has a three-dimensional structure and is a heated and / or cooled data processing system. [Effects of the Invention]

[0014] The present invention provides a technique for integrating 3D models based on visible images and 3D models based on thermal images. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a conceptual diagram of an embodiment. [Figure 2] FIG. 1 is a block diagram of a data processing device. [Figure 3] Photographs showing an RGB visible image (A) and a thermal image (B). [Figure 4] These are photographs that serve as drawings of 3D object models, 3D thermal models, and a 3D model that combines both. [Figure 5] This is an image showing the state in which an object 3D model and a thermal 3D model are displayed on the screen. [Figure 6] FIG. 10 is a diagram showing a marker according to another embodiment. [Figure 7] FIG. 10 is a diagram showing a marker according to another embodiment. [Figure 8] FIG. 10 is a diagram showing a marker according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] 1. First embodiment (overview) A 3D model based on visible images (hereafter referred to as a physical 3D model) is a model of the three-dimensional structure of an object. This can be understood by recalling a 3D model created using CAD. On the other hand, a 3D model based on thermal images (hereafter referred to as a thermal 3D model) is a model of the three-dimensional heat distribution of an object.

[0017] Considering the components of a 3D model, the former is the shape of an object existing in three-dimensional space, and the latter is the heat emitted by that object.

[0018] For example, when integrating two 3D object models, the common 3D structure between them is identified, and then the correspondence between them is determined using that as a starting point. This allows the two 3D object models to be aligned and integrated.

[0019] When integrating a 3D object model with a 3D thermal model, the elements that make up the models are different, as mentioned above, so it is necessary to devise a method for comparing and determining the correspondence. In other words, it is necessary to compare the object and the heat and determine the correspondence, but how to do this becomes a problem.

[0020] Here, an object 3D model and a thermal 3D model are created using markers 101, 102, and 103 that have a three-dimensional shape and generate heat. Because the markers 101 to 103 have a three-dimensional shape, they are modeled in the object 3D model in the same way as a normal 3D model. On the other hand, because the markers 101 to 103 generate heat, their shapes are visualized as heat-generating elements in a thermal image, and the 3D model created based on this (thermal 3D model) shows the shapes of the markers 101 to 103 similar to the object 3D model.

[0021] Therefore, the correspondence between the object 3D model and thermal 3D model of markers 101-103 can be determined in the same way as with normal 3D models, and an integrated 3D model that integrates both can be created. Also, by making the markers 101-103 structures that extend vertically and arranging them apart, spatial information in three orthogonal directions is provided to the stereo thermal image that forms the basis of the thermal image. Therefore, the correspondence between the three-dimensional positions of the object 3D model and the thermal 3D model can be determined with high accuracy. The principle has been explained simply here, but details will be provided later.

[0022] (System Overview) 1 shows a plant 100 that will be the subject of the final 3D model. The plant 100 is an example of the subject, and is not particularly limited.

[0023] Markers 101, 102, and 103 are rectangular pillars with different heights and colors (as seen in a visible image). Each marker is equipped with a heating element 101a, 102a, and 103a, and is heated to a different predetermined temperature. Each marker is made of a material with high thermal conductivity and high emissivity (e.g., ceramics, painted or ceramic-coated metal, etc.).

[0024] In the visible image, the positions of the edges and vertices of each marker are recognized as feature points and are identified by color. In the thermal image, the positions of the edges and vertices of each marker are recognized as feature points on the thermal image. If there is a temperature gradient, it is extracted from the thermal image as a feature point.

[0025] The relationship between the size, color, and temperature of each marker is known. That is, marker 101 has a first size, a first color, and a first temperature, marker 102 has a second size, a second color, and a second temperature, and marker 103 has a third size, a third color, and a third temperature. By using different colors, each marker can be distinguished in the visible image, and by intentionally using different temperatures, each marker can be distinguished as a temperature marker in the thermal image.

[0026] Furthermore, by making the dimensions known and different, the task of determining the positions and orientations of the visible and thermal cameras using stereo images is facilitated and the accuracy is improved.

[0027] Each marker generates a temperature gradient in the longitudinal direction. This temperature gradient is the target of identification in the thermal image. Each marker may be configured by stacking multiple materials with different thermal conductivities in the longitudinal direction to create a more intentional thermal gradient.

[0028] A code that identifies each marker in a visible image may be displayed on each marker. This code may also be made recognizable in a thermal image. For example, if a code display pattern made of thin polystyrene foam or urethane is attached to the side or top of the marker 101, the temperature and radiation characteristics of the code display area will differ from those of the surrounding area when the marker 101 is heated, and this will be recognizable in a thermal image. For example, if a thin polystyrene foam or urethane-processed letter A pattern is attached to the surface of a heated metal, a letter A with different thermal radiation will be formed, and this will be recognizable in a thermal image. Of course, this letter pattern can be identified in a visible image.

[0029] Reference numeral 104 denotes a matching marker, which has a random dot pattern formed on the surface of a rectangular prism that generates heat. The random dot pattern is used when matching between stereo images. The random dot pattern of the matching marker 104 is a dark-colored dot that can be visualized in a visible image, and each dot generates heat, so it is also visualized as a dot pattern in a thermal image. Therefore, the random dot pattern of the matching marker 104 can be used for matching between visible images and between thermal images.

[0030] A specific structure of the matching marker 104 is a heatable painted metal pole, and a heat insulating layer (e.g., a thin layer of polystyrene foam or urethane foam) of a different color from the metal pole, covering the surface of the metal pole and having holes with a random dot pattern.

[0031] 1 shows a visible image stereo camera 110 consisting of visible image cameras 111 and 112. The visible image cameras 111 and 112 are placed at a known distance apart and set to capture overlapping areas, just like a normal stereo camera.

[0032] 1 also shows a thermal imaging stereo camera 120 consisting of thermal imaging cameras 121 and 122. Similar to the visible image stereo camera 110, the thermal imaging stereo camera 120 has the thermal imaging cameras 121 and 122 arranged at a known distance apart and set to capture overlapping ranges.

[0033] A visible light camera is a camera that captures images in the visible light range. A visible light camera can be a digital camera that captures RGB images (color images). It is also possible to capture video and extract still images from it. The same is true for a thermal imaging camera, which is an infrared camera that captures thermal images.

[0034] Although not shown in Figure 1, many more visible and thermal imaging cameras are installed to eliminate blind spots in the imaging range. Adjacent or nearby visible and thermal imaging cameras form a stereo camera. Alternatively, the same effect can be achieved by moving a single camera and capturing multiple images, rather than using a stereo configuration for the visible and thermal imaging cameras. There are also cameras that combine visible and thermal imaging cameras, allowing both to capture images simultaneously. In this case, the same effect can be achieved by moving one camera.

[0035] The plant 100, markers 101 to 103, and matching marker 104 to be measured in 3D may be placed on a rotating table, and visible images and thermal images may be taken while the rotating table is rotated. In this case, visible images and thermal images are taken from many different viewpoints, and many stereo visible images and stereo thermal images are obtained.

[0036] (Data Processing Device) 1 shows a data processing device 200 that performs processing related to the creation of an object 3D model, a thermal 3D model, and an integrated 3D model. The data processing device 200 is configured by a computer. The data processing device 200 can be configured by a PC (personal computer) or a data processing server.

[0037] The following describes the data processing device 200. The data processing device 200 includes an object 3D model creation unit 210, a thermal 3D model creation unit 220, and an integrated 3D model creation unit 230.

[0038] The object 3D model creation unit 210 includes a visible image data receiving unit 211 , a feature point extraction unit 212 , a corresponding point identification unit 213 , a point cloud data creation unit 214 , and an object 3D model creation unit 215 .

[0039] The visible image data receiving unit 211 receives image data of visible images captured by visible image cameras such as the visible image cameras 111 and 112. The feature point extraction unit 212 extracts feature points from the visible images received by the visible image receiving unit 211. The corresponding point identification unit 213 identifies the correspondence between feature points among the multiple visible images from which the feature points have been extracted.

[0040] The point cloud data creation unit 214 identifies the three-dimensional positional relationship between feature points in the stereo visible images and the visible image cameras that captured the stereo visible images, and obtains point cloud data that describes the three-dimensional positions of each feature point. The object 3D model creation unit 215 creates an object 3D model based on the point cloud data derived from the visible images. The creation of the object 3D model will be described later.

[0041] The thermal 3D model creation unit 220 includes a thermal image data receiving unit 221 , a feature point extraction unit 222 , a corresponding point identification unit 223 , a point cloud data creation unit 224 , and a thermal 3D model creation unit 225 .

[0042] The thermal image data receiving unit 221 receives image data of thermal images captured by thermal imaging cameras such as the thermal imaging cameras 121 and 122. The feature point extracting unit 222 extracts feature points from the thermal images received by the thermal image receiving unit. The corresponding point identifying unit 223 identifies the correspondence between feature points among the multiple thermal images from which the feature points have been extracted.

[0043] The point cloud data creation unit 224 identifies the three-dimensional positional relationship between the feature points in the stereo thermal images and the thermal imaging camera that captured the stereo thermal images, and obtains point cloud data describing the three-dimensional positions of each feature point.The thermal 3D model creation unit 225 creates a thermal 3D model based on the point cloud data derived from the thermal images.

[0044] The integrated 3D model creation unit 230 creates an integrated 3D model by integrating the object 3D model and the thermal 3D model. In the integrated 3D model, the three-dimensional positions of the object 3D model and the thermal 3D model are aligned and superimposed in three-dimensional space. In the integrated 3D model, the three-dimensional appearance of the object and the three-dimensional temperature distribution of each part are converted into data.

[0045] By obtaining an integrated 3D model, for example, if the object 3D model is displayed first, the 3D structure of the object can be easily grasped, and if the thermal 3D model is displayed first, the heat distribution of the object can be easily grasped. It is also possible to display one of the 3D models semi-transparently. It is also possible to display the temperature of each part of the object 3D model.

[0046] (Creating 3D models of objects) The 3D model is created using a known method, which is described in, for example, JP 2013-186816 A, WO2011 / 070927 A, JP 2012-230594 A, and JP 2014-35702 A.

[0047] A brief explanation will be given below. Here, to simplify the explanation, we will focus on the marker 101. First, a 3D object model of the marker 101 is created based on visible stereo images captured by the visible image stereo camera 110. Here, since the color of the marker 101 is known, the marker 101 can be identified in the visible image.

[0048] Here, a correspondence relationship is identified using the image of the marker 101 between a visible image 1 captured by a visible image camera 111 and a visible image 2 captured by a visible image camera 112 and showing the marker 101. By identifying the correspondence, the two visible images are matched, and the visible image 1 and the visible image 2 become a stereo visible image.

[0049] This process is performed as follows. Here, a case will be described in which the correspondence between visible image 1 and visible image 2 is identified using the marker 101. First, feature points are extracted from visible image 1 and visible image 2.

[0050] At this time, the feature points of the markers 101 are extracted. Then, a correspondence is identified between the distribution of the feature points of the markers 101 in the visible image 1 and the distribution of the feature points of the markers 101 in the visible image 2. This correspondence is identified using a known image matching technique such as template matching. At this time, the identification information of the markers 101 identified by color or code may be used to roughly align the positions, and then matching may be performed.

[0051] By identifying the correspondence between visible image 1 and visible image 2, it becomes clear which part of visible image 1 corresponds to visible image 2.

[0052] Next, relative orientation is performed to identify the relative positional relationship between each feature point and the visible image cameras 111 and 112. For example, focus on a certain point A. By investigating where this point A appears in the visible image 1, the direction of point A as seen from the visible image camera 111 can be determined. Similarly, the direction of point A as seen from the visible image camera 112 can be determined.

[0053] Therefore, a triangle is determined with the three vertices being point A, visible image camera 111, and visible image camera 112. This is the relative orientation. This is performed for each feature point. In this way, the relative positional relationship between each feature point, visible image camera 111, and visible image camera 112 is determined. The camera position is the projection center of the camera (optical origin: viewpoint).

[0054] Here, if the separation distance between the visible image camera 111 and the visible image camera 112 is given, the dimensions of this triangle are given. Also, if the distance between two separated points is given, the dimensions of the above triangle are given. For example, the dimensions of the marker 101 are known, and by giving these dimensions, the positional relationship between each point and the visible image camera 111 and the visible image camera 112 is determined.

[0055] In this way, the three-dimensional coordinates of the feature points extracted from visible image 1 and visible image 2 are determined. At this stage, the three-dimensional positions are determined in a local coordinate system based on the positions of visible image camera 111 and visible image camera 112. If the camera positions or absolute coordinates of points are given, the positions of the above points are determined in the absolute coordinate system. An absolute coordinate system is a coordinate system used in maps and GNSS.

[0056] In this way, point cloud data is obtained for the marker 101 and the object that appears in both visible image 1 and visible image 2. The point cloud data based on the visible images is a data group that describes the coordinates of each point. A 3D model of the object including the marker 101 is created based on this point cloud data.

[0057] The above processing is performed on the markers 102 and 103, as well as the plant 100. Since the markers 101 to 103 are arranged at intervals, a highly accurate 3D object model can be obtained.

[0058] (Creating a thermal 3D model) A thermal 3D model is created using the same method as for the object 3D model. Figure 3(A) shows an RGB image of a heating element with a rectangular mosaic pattern (checkered pattern), and Figure 3(B) shows the thermal image corresponding to Figure 3(A). This heating element is configured so that the mosaic pattern contains both heated and non-heated areas.

[0059] As can be seen by comparing with FIG. 3(A), the rectangular mosaic pattern is clearly visualized in the thermal image of FIG. 3(B). That is, the borders and edges are clearly recognizable in the thermal image. These parts are effectively extracted as feature points. Therefore, similar to the case where a visible image is used, a thermal 3D model of the markers 101-103 is created using the thermal image. Of course, a thermal 3D model of the plant 100 is also created.

[0060] The process flow is briefly explained below. First, two thermal images are obtained as candidates for the stereo thermal image captured by the thermal imaging stereo camera 120. Then, feature points are extracted from these two thermal images, and the correspondence between these two thermal images is identified. Next, the positional relationship between the thermal imaging stereo camera 120 and each feature point is identified, and the coordinates of each feature point are obtained. In this way, point cloud data based on the thermal images is obtained. Then, a thermal 3D model is created based on the point cloud data derived from these thermal images.

[0061] Point cloud data derived from thermal images consists of temperature information for each point (usually, temperature differences are expressed by color) and 3D coordinates. A thermal 3D model based on this point cloud data is a model that shows the three-dimensional heat distribution of an object.

[0062] The above processing is performed on the markers 102 and 103 as well as the plant 100. Since the markers 101 to 103 are arranged at intervals, a highly accurate thermal 3D model can be obtained.

[0063] (Integration of physical 3D models and thermal 3D models) First, in the thermal 3D model, the markers 101 to 103 can be identified by their temperatures. This is because the markers 101 to 103 are adjusted to different temperatures in advance. Also, they can be identified by providing identification codes that can be identified on the thermal image.

[0064] On the other hand, in the 3D object model, the markers 101 to 103 are also identified by color or identification code display.

[0065] Therefore, the markers 101 to 103 can be identified in the object 3D model and the thermal 3D model, and their approximate correspondence can be determined. However, detailed alignment is required for the orientation and scale.

[0066] The detailed alignment of the object 3D model and the thermal 3D model will be explained below. Figure 5 shows an example of an object 3D model and a thermal 3D model. Figure 5(A) shows the state as seen from an obliquely upward viewpoint, and (B) shows the state as seen from an obliquely downward viewpoint. Figure 5 shows an object 3D model 510 and a thermal 3D model 520 of a plate-shaped object.

[0067] An object 3D model is a digital representation of the external shape of an object in three-dimensional space. For example, the data for object 3D model 510 in Fig. 5 is data that specifies the three-dimensional position of the external shape of a plate-shaped object that is the subject of the 3D model. This data is visualized to produce object 3D model 510 in Fig. 5, for example.

[0068] On the other hand, a thermal 3D model is a digital representation of the heat distribution in three-dimensional space of an object. For example, the data of thermal 3D model 520 in FIG. 5 is data that specifies the three-dimensional heat distribution on the surface of a plate-shaped object that is the subject of the 3D model. In other words, it is a digital representation of the relationship between the position and temperature of each part on the outside of the object. This data is visualized as, for example, thermal 3D model 520 in FIG. 5.

[0069] Since a thermal 3D model is based on the relationship between the position and temperature of each part of the object's exterior, if the temperature distribution of the object can be distinguished from its surroundings, this temperature distribution will correspond to the object's appearance. That is, the shape of the object can be grasped from the three-dimensional distribution of heat, as in thermal 3D model 520 in Figure 5. In other words, the three-dimensional distribution of heat corresponds to the three-dimensional shape of the object.

[0070] In order to make the temperature distribution of the object distinguishable from the surroundings, the markers 101 to 103 in Fig. 1 are made of metal in a rectangular column shape that is easy to grasp and generates heat, and are heated. This makes it easy to compare their appearances, as in the object 3D model 510 and thermal 3D model 520 in Fig. 5.

[0071] In other words, the three-dimensional shape of the object characterized by the physical 3D model can be compared with the three-dimensional heat distribution of the object characterized by the thermal 3D model. This comparison clarifies the correspondence between the physical 3D model and the thermal 3D model. In other words, it clarifies which parts of the physical 3D model correspond to which parts of the thermal 3D model. In this way, the physical 3D model and the thermal 3D model can be aligned and integrated.

[0072] An example of a process for clarifying the correspondence between an object 3D model and a thermal 3D model will be described below. For example, consider overlaying an object 3D model and a thermal 3D model in a virtual three-dimensional space. In this case, one is translated, rotated, or scaled relative to the other to search for conditions under which the object 3D model and the thermal 3D model of the markers 101 to 103 overlap.

[0073] The vertices and edges of the markers 101-103 are converted into data in the object 3D model and the thermal 3D model, so they can be aligned in the same way as when 3D objects are aligned. In particular, the markers 101-103 are intentionally heated so that the vertices and edges of their shapes are easily visualized as feature points in the thermal image. This makes it easy to find the conditions under which the object 3D model and thermal 3D model of the markers 101-103 overlap.

[0074] For example, Figure 3(B) shows a thermal image of a plant leaf. When a thermal 3D model is created based on this thermal image, the temperature difference with the background is not intentionally created, so the edges and borders on the 3D model tend to be unclear. Therefore, it can be difficult to identify the correspondence between the leaf's physical 3D model and its thermal 3D model, and errors are likely to occur.

[0075] By finding the conditions under which the object 3D model and the thermal 3D model overlap, the correspondence between the object 3D model and the thermal 3D model becomes clear. This allows the object 3D model and the thermal 3D model to be aligned (registered). By aligning the positions of the two models, corresponding parts become aligned, and it becomes clear which parts of the object 3D model correspond to which parts of the thermal 3D model. Finally, an integrated 3D model is obtained by integrating the object 3D model and the thermal 3D model.

[0076] A known 3D model matching technique can be used to identify the correspondence between the object 3D model and the thermal 3D model, such as those described in WO2012-141235, JP2014-35702A, JP2015-46128A, and JP2017-15598A.

[0077] Figure 4 shows a photograph of the image of the integrated 3D model obtained by merging the image of the object 3D model (monochrome image on the left) and the image of the thermal 3D model (color image on the right).

[0078] In Figure 4, iron powder that oxidizes and generates heat is scattered around the object as a random dot pattern for matching thermal stereo images. This oxidizing iron powder is visualized as a random dot pattern in the thermal image.

[0079] (superiority) The markers 101 to 103 extend three-dimensionally and function as three-dimensional reference structures in the 3D object model based on the visible image. At the same time, they generate heat themselves and function as three-dimensional reference structures in the thermal 3D model.

[0080] Then, in integrating the physical 3D model and the thermal 3D model, the correspondence between the physical 3D model and the thermal 3D model can be identified using the markers 101 to 103 that are common to both as references.

[0081] Furthermore, since the markers 101 to 103 extend three-dimensionally, the object 3D model and the thermal 3D model can be aligned with high precision.

[0082] For example, alignment in the XY plane may be inaccurate in the Z direction. The markers 101 to 103 extend in the Z direction and have different dimensions in the Z direction. This allows for high-precision three-dimensional alignment.

[0083] 2. Second embodiment Another example of the markers 101 to 103 in Fig. 1 will be described. Fig. 6 shows a marker 500. The marker 500 has a rectangular pillar structure 501 made of metal (aluminum, brass, etc.), on the surface of which a barcode display 504 is formed. The barcode display 504 is formed by attaching a thermal insulating sheet 503 made of black polystyrene foam, urethane foam, or the like to the surface of the rectangular pillar structure 501.

[0084] The rectangular pillar structure 501 is heated by a heater 507. The barcode 504 can be identified from both a visible image and a thermal image. First, in a visible image, the barcode display 504 is identified by its appearance. In a thermal image, the barcode display 504 is identified by the temperature difference between the metal and the thermal insulation sheet 503. The marker 500 is identified by the barcode display 504.

[0085] A checkered positioning mark 505 divided into four rectangles is provided on the top surface of the square pillar structure 501. The positioning mark 505 is composed of a thermally insulated area 506, to which a thin sheet of polystyrene foam that is colored a different color from the base is attached, and an exposed area 507 of the metal base.

[0086] The positioning indicator 505 provides position information to the visible and thermal images by detecting the intersection of the cross 508 from the visible and thermal images.

[0087] In the visible image, the cross intersection 508 is detected by the difference in color between the thermally insulated area 506 and the exposed area 507. In the thermal image, the cross intersection 508 is detected by the difference in temperature between the thermally insulated area 506 and the exposed area 507.

[0088] As a result, the positioning display 505 functions as a reference point target in the visible image and the thermal image. This reference point target can be used as an orientation point in the visible image and the thermal image.

[0089] The barcode display 504 can be identified in both the object 3D model obtained from the visible image and the thermal 3D model obtained from the thermal image, and therefore also functions as a common marker (reference marker) when integrating the object 3D model and the thermal 3D model.

[0090] 6 shows an example of a rectangular prism, but it may also be a circular prism. It is also possible to use a polygonal prism such as a triangular prism, a pentagonal prism, or a hexagonal prism.

[0091] 3. Third embodiment Figure 7 shows markers 600 and 610. Marker 600 is a square prism with a temperature gradient intentionally formed in the longitudinal direction. Marker 600 is composed of a lower metal 601, a Peltier element 602, and an upper metal 603. Peltier element 602 is oriented so that its top surface is hot and its bottom surface is cold.

[0092] The lower metal 601 and the upper metal 602 are colored in different colors, and the color combination makes them distinguishable in a visible image.

[0093] When the Peltier element 602 is operated, the surface of the upper metal 603 that is in contact with the Peltier element 602 is heated. Figure 7 shows an example in which a thermal gradient occurs from the bottom to the top of the upper metal 603.

[0094] When the Peltier element 602 is operated, the surface of the lower metal 601 in contact with the Peltier element 602 is cooled.

[0095] By controlling the voltage applied to the Peltier element 602, the amount of heat generated and absorbed can be adjusted, and the temperature distribution in the vertical direction can be adjusted. Of course, the absolute value of the temperature can also be adjusted. In addition, changing the position of the Peltier element 602 in the vertical direction also changes the shape of the temperature distribution in the vertical direction. This difference in temperature distribution is identified in the thermal image as identification information.

[0096] Marker 610 is an example of a marker that uses two Peltier elements 612 and 614 to provide heat distribution information that can be identified in a more complex thermal image. Marker 610 has a layered structure consisting of a lower metal 611, a Peltier element 612, a middle metal 613, a Peltier element 614, and an upper metal 615.

[0097] The lower metal 611, the middle metal 613 and the upper metal 615 are colored in different colors, and the color combinations make them distinguishable in a visible image.

[0098] By controlling the operation of the Peltier elements 612 and 614, the temperature gradient and temperature distribution in the longitudinal direction of the marker 610 are adjusted, and identification information that can be identified in a thermal image is added.

[0099] 4. Fourth Embodiment 8 shows a case where markers 701, 702, and 703 are placed around a plant 100 that is the target of 3D creation. Note that the visible image camera and thermal image camera for 3D measurement are not shown.

[0100] Each marker is a metal cylinder, and its bottom is heated by heaters 701a, 702a, and 703a. Markers 701 to 703 are set to known different lengths and colors. Setting each marker to a different length makes it easier to identify the correspondence between the object 3D model and the thermal 3D model.

[0101] The center of the circular part on the top surface of each marker is detected as a reference point in the visible image and the thermal image. That is, the top surface of each marker appears as a circle when viewed vertically from above, and as an ellipse when viewed from an oblique direction, and the center can be detected in the image. For example, if it is a circle, the center can be found by finding the center of curvature of the arc. If it is an ellipse, the major and minor axes are found, and the center can be found from their intersection.

[0102] 5. Fifth Embodiment In the markers 101 to 103 in FIG. 1, if two adjacent lateral faces are easily identified (distinguished) on the thermal 3D model, the edges are clearly digitized, making it easier to integrate them with the object 3D model.

[0103] For example, in the case of the marker 101, one method is to cover the two adjacent sides with materials that have different thermal radiation characteristics or thermal conductivity, or to place Peltier elements on each side to make them have different temperatures. In this case, each side will have a different temperature, and the edges that are the boundaries between the surfaces in the thermal image will be easier to extract as feature points, making them clear in the data.

[0104] Other methods include painting adjacent surfaces with different materials, or creating different surface conditions on adjacent surfaces (e.g., one rough surface and the other mirror finish). In this case, differences occur in the radiation characteristics of the radiant heat, making the edges between the surfaces more clearly visible in the thermal image.

[0105] The clearer edges make it easier to understand the three-dimensional structure of the thermal 3D model and to compare it with a physical 3D model.

[0106] 6. Sixth Embodiment The heating surface on the side of the marker 101 may be made checkered. For example, a thermal insulating layer (such as a polystyrene foam layer or a urethane foam layer) can be applied to the checkered pattern, creating a difference in surface temperature between the area with the thermal insulating layer and the area without it, so that the checkered pattern can be visualized in a thermal image. This makes the three-dimensional relationship between the surfaces clearer when a thermal 3D model is created.

[0107] Furthermore, this checkered pattern can be distinguished in the 3D object model by changing the color, etc., so that it can be distinguished in the visible image. Therefore, the checkered pattern also functions as an alignment marker used to identify the correspondence between the 3D object model and the thermal 3D model.

[0108] 7. Seventh Embodiment A temperature indicator is applied to the side of the markers 101 to 103, or a sheet of the temperature indicator is attached. The temperature indicator is a material that uses polymers or liquid crystals and changes color depending on the temperature. Temperature indicator sheets or inks are commercially available.

[0109] In this example, the relationship between the color and temperature indicated by the temperature indicator and the relationship between the color and temperature in the thermal image are investigated in advance. Furthermore, the two relationships are adjusted in advance to be as close as possible (ideally, to match). For example, the colors indicated by the temperature indicator at 15°, 20°, 25°, and 30° are adjusted in advance to be as similar as possible to the colors indicated by 15°, 20°, 25°, and 30° in the thermal image.

[0110] This adjustment is made by (1) selecting the type of temperature indicator, (2) adjusting the settings of the thermal imaging camera, and (3) using both (1) and (2) in combination.

[0111] By using a temperature indicating sheet, thermal gradients and temperature boundaries are visualized in the visible image and extracted as feature points. The 3D object model is created based on the feature points extracted from the visible image, so the temperature displayed on the temperature indicating sheet is reflected in the 3D object model.

[0112] On the other hand, the thermal 3D model is based on feature points extracted from the thermal image, and therefore has an appearance that corresponds to the temperature distribution of the temperature indicator.

[0113] Therefore, the temperature gradient or temperature difference in the temperature indicator portion on the surface of the markers 101 to 103 is visualized in the object 3D model and the thermal 3D model, becoming a characteristic portion. By comparing these characteristic portions, the correspondence between the object 3D model and the thermal 3D model related to the markers 101 to 103 can be identified.

[0114] In this case, the color of each part of the object 3D model is known from the base visible image, and the color can be applied to the display of the object 3D model. On the other hand, in the thermal 3D model, the temperature of each part is known from the base thermal image, and the color corresponding to that temperature can be applied to the display of the thermal 3D model.

[0115] Here, the relationship between the color of the temperature indicator and the color of the thermal image is determined and adjusted in advance as described above. Therefore, by comparing the color of the object 3D model and the color of the thermal 3D model for the markers 101 to 103, it is possible to determine the correspondence between the object 3D model and the thermal 3D model for the markers 101 to 103. Of course, the correspondence can also be determined by comparing the above-mentioned characteristic parts (shape comparison).

[0116] That is, the surfaces of the markers 101 to 103 are provided with a coated surface or sheet of a material that changes color depending on temperature, and the color information of the color change is associated with the object 3D model, and the temperature information of the material that changes color depending on temperature is associated with the thermal 3D model. When the thermal 3D model is visualized, it changes color depending on the temperature. By determining the correspondence between this color change depending on temperature and the color of the material that changes color depending on temperature in the object 3D model, the correspondence between the object 3D model and the thermal 3D model can be determined.

[0117] A specific example will be described below. For simplicity, we will focus on the marker 101 in Fig. 1. Here, it is assumed that the marker 101 has a sheet of temperature indicator attached to its side, and when heated, the sheet changes color in accordance with the detected temperature.

[0118] Now, let us focus on the adjacent first and second side surfaces of the marker 101. In this case, in a visible stereo image of the marker 101, the first and second side surfaces are converted into image data as colors corresponding to the color of the temperature indicator sheet. Here, the color of the temperature indicator sheet corresponds to the temperature detected by the sheet. In other words, it corresponds to the surface temperature of the marker 101.

[0119] Here, color information of the temperature indicator can be given to a 3D object model created based on visible stereo images of the marker 101. For example, the 3D object model can be colored with the color of the temperature indicator. In other words, temperature information can be associated with the 3D object model.

[0120] On the other hand, in the stereo thermal image of the marker 101, the first and second sides are converted into image data as colors according to temperature. The display of the colors according to temperature corresponds to the surface temperature of the marker 101.

[0121] Therefore, the thermal 3D model created based on the stereo thermal images can be colored in the color on the thermal image corresponding to the temperature of the temperature indicator, thereby associating the temperature information of the object with the thermal 3D model.

[0122] Here, if the relationship between the color corresponding to the detected temperature of the temperature indicator and the color corresponding to the temperature of the temperature indicator in the thermal image has been investigated in advance, the correspondence between the object 3D model and the thermal 3D model can be identified.

[0123] For example, consider the case of comparing the color A of the first side of the marker 101 in the object 3D model with the color A' of the first side of the marker 101 in the thermal 3D model, where A is the color of the sheet-like temperature indicator, and A' is the color on the thermal image that corresponds to the temperature of the sheet-like temperature indicator.

[0124] Here, if the relationship between A and A' is known, or if it is adjusted to A=A', the correspondence between the first side (color A) of the marker 101 in the object 3D model and the first side (color A') of the marker 101 in the thermal 3D model can be determined.

[0125] This process is also performed for the second and third aspects, and further for the markers 102 and 103, thereby determining the correspondence between the object 3D model and the thermal 3D model related to the markers 101 to 103. Then, by determining this correspondence, the correspondence between the object 3D model and the thermal 3D model of the plant 100 can also be determined.

[0126] 8.Other One or more of the markers 101 to 103 and 701 to 703 may be cooled.

[0127] 7, it is also possible to configure the central metal 613 sandwiched between two Peltier elements to be heated or cooled by both Peltier elements 612 and 614. In this case, the heating surfaces of the two Peltier elements 612 and 614 both face the central metal 613 side, or the cooling surfaces of the two Peltier elements both face the central metal 613 side.

[0128] It is also possible to arrange an intermediate layer between the Peltier element and the member to be heated.The marker preferably has a three-dimensional structure with edges and vertices.

[0129] Different markers disclosed in this specification may be used in combination. For example, if four markers are used, the first marker may be marker 101 in Fig. 1, the second marker may be marker 500 in Fig. 6, the third marker may be marker 600 in Fig. 7, and the fourth marker may be marker 701 in Fig. 8.

[0130] To enable simultaneous detection and recognition of the marker's position, the marker can be configured with characters or codes on its top surface, with the characters or codes generating or absorbing heat (cooling). In this case, a coded target that can be detected by both visible and thermal images is placed on the top surface of the marker. In this case, the marker is designed so that the temperature of the characters or codes can change.

[0131] A similar function can also be achieved by making the area around the letters or code on the top surface of the marker heat up or absorb heat (cool down).Also, by making the letters or code heat up or absorb heat and making the surrounding area heat up or absorb heat, the letters or code can be made to stand out in the thermal image. [Explanation of symbols]

[0132] 100...Target plant, 101, 102, 103...Marker, heating element 101a, 102a, 103a...Heater element, 110...Visible image stereo camera, 111, 112...Visible image camera, 120...Thermal image stereo camera, 121, 122...Thermal image camera, 200...Data processing device, 500...Marker, 501...Rectangular pillar structure, 503...Thermal insulation sheet, 504...Barcode display, 505...Position Determination display, 506...thermal insulation area, 507...exposed area of ​​underlying metal, 510...object 3D model, 520...thermal 3D model, 600...marker, 601...lower metal, 602...Peltier element, 603...upper metal, 610...marker, 611...lower metal, 612...Peltier element, 614...Peltier element, 615...upper metal, 701, 702, 703...markers, 701a, 702a, 703a...heaters.

Claims

1. 1. A data processing method for creating an integrated 3D model by integrating a first 3D model based on a visible light image and a second 3D model based on a thermal image, comprising: A common marker appears in the visible photographed image and the thermal image, The first 3D model and the second 3D model are integrated using the marker as a reference; the marker has a three-dimensional structure, A data processing method, wherein the marker is heated and / or cooled.

2. the first 3D model is a digital representation of the external shape of the marker in a three-dimensional space; the second 3D model is a digital representation of a heat distribution of the marker in a three-dimensional space; The data processing method according to claim 1 , wherein the first 3D model and the second 3D model are integrated by comparing the shape and the shape of the heat distribution.

3. There are a plurality of the markers, 3. The data processing method according to claim 1, wherein the plurality of markers are of different sizes and are heated or cooled to different temperatures.

4. A data processing method as described in claim 1, wherein the marker is partially heated or cooled and has a thermal distribution with a thermal gradient.

5. the marker has a structure in which a Peltier element is sandwiched between a first member and a second member, 5. The data processing method according to claim 4, wherein the thermal gradient is formed by heating the first member and cooling the second member by the Peltier element.

6. 2. The data processing method according to claim 1, wherein the marker has an identification display pattern formed thereon that can be identified by differences in color and thermal radiation.

7. the marker has a plurality of faces; The data processing method according to claim 1, wherein the surfaces have different surface temperatures.

8. the marker is provided with a thermosensitive portion made of a material that changes color depending on temperature, and color information of the change in color is associated with the first 3D model; The data processing method according to claim 1 , wherein temperature information of the temperature-indicating portion is associated with the second 3D model.

9. A data processing system for creating an integrated 3D model by integrating a first 3D model based on a visible image and a second 3D model based on a thermal image, comprising: A common marker appears in the visible photographed image and the thermal image, an integrated 3D model creation unit that integrates the first 3D model and the second 3D model using the marker as a reference; the marker has a three-dimensional structure, The marker is a heated and / or cooled data processing system.

Images