Glass curtain wall three-dimensional modeling method, device and equipment and storage medium

Abstract

The invention discloses a glass curtain wall three-dimensional modeling method, device and equipment and a storage medium. The method comprises the steps that one or more image groups obtained by shooting a target glass curtain wall are acquired; matching the first visible light image and the first thermal infrared image of the same image group to obtain a second visible light image and a second thermal infrared image which are matched with each other; according to the temperature of each pixel in the second thermal infrared image, identifying the position of a glass panel pixel in the second thermal infrared image; according to the positions of the glass panel pixels in the second thermal infrared image, color correction is carried out on the glass panel pixels in the second visible light image, and a third visible light image is obtained; and performing three-dimensional modeling on the target glass curtain wall by using the one or more third visible light images. According to the embodiment of the invention, a real, clear and complete high-precision glass curtain wall three-dimensional model can be constructed.

Description

Technical Field

[0001] This application relates to the field of 3D modeling technology, and in particular to a method, apparatus, equipment and storage medium for 3D modeling of glass curtain walls. Background Technology

[0002] Currently, the primary technology for 3D real-world modeling of buildings is drone-based oblique photography. Drone-based oblique photography uses a drone equipped with a visible light camera to collect texture information from the building's roof and side facades, generating a 3D model with realistic textures that can quickly cover a large area of ​​the scene.

[0003] Existing technologies are mainly applied to building materials with rough surfaces or no obvious light reflection. However, the current building envelopes of high-rise and super high-rise buildings in cities are made of glass curtain walls. Because the glass panels of the glass curtain wall have the characteristics of light reflection and transmission, when taking oblique photos by drones, the images captured contain reflections or transmissions from the glass panels. In particular, the reflections are more severe, which makes it impossible for the images to reflect the real and stable texture of the curtain wall. As a result, when modeling 3D, the glass curtain wall is prone to distortion, voids or texture disorder, resulting in poor modeling quality of the glass curtain wall, or even modeling failure. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for three-dimensional modeling of glass curtain walls, in order to solve the problem that the modeling quality of glass curtain walls is poor or even fails due to reflection or transmission of glass panels in the prior art.

[0005] To achieve the above objectives, embodiments of this application provide a three-dimensional modeling method for glass curtain walls, including:

[0006] Acquire one or more image sets of the target glass curtain wall by taking pictures, each image set including a first visible light image and a first thermal infrared image taken at the same waypoint;

[0007] The first visible light image and the first thermal infrared image in the same image group are matched to obtain a second visible light image and a second thermal infrared image that match each other.

[0008] Based on the temperature of each pixel in the second thermal infrared image, identify the position of the glass panel pixel in the second thermal infrared image;

[0009] Based on the position of the glass panel pixels in the second thermal infrared image, the glass panel pixels in the second visible light image are color-corrected to obtain a third visible light image;

[0010] The target glass curtain wall is modeled in three dimensions using one or more of the third visible light images.

[0011] As an improvement to the above solution, the step of identifying the position of the glass panel pixels in the second thermal infrared image based on the temperature of each pixel in the second thermal infrared image includes:

[0012] The pixels in the second thermal infrared image are sampled, and the temperature of all sampled points is obtained based on the mapping relationship between pixel color and temperature in the thermal infrared image.

[0013] The temperature of all the sampling points is statistically analyzed to determine the temperature range of the glass panel pixels in the second thermal infrared image;

[0014] The pixel location where the target temperature is located is taken as the location of the glass panel pixel in the second thermal infrared image; wherein, the target temperature is the temperature of the pixel in the second thermal infrared image that belongs to the temperature range.

[0015] As an improvement to the above solution, the step of color-correcting the glass panel pixels in the second visible light image based on the position of the glass panel pixels in the second thermal infrared image to obtain a third visible light image includes:

[0016] The position of the glass panel pixels in the second visible light image is determined based on the position of the glass panel pixels in the second thermal infrared image;

[0017] Determine the primary color of the glass panel in the second visible light image;

[0018] Based on the position of the glass panel pixels in the second visible light image, the color of all the glass panel pixels in the second visible light image is set to the primary color to obtain the third visible light image.

[0019] As an improvement to the above scheme, the step of matching the first visible light image and the first thermal infrared image of the same image group to obtain a mutually matched second visible light image and a second thermal infrared image includes:

[0020] Based on the imaging range of the visible light camera and the imaging range of the thermal infrared camera, calculate the horizontal and vertical offsets between the first thermal infrared image and the first visible light image.

[0021] The cutting dimensions are obtained based on the horizontal and vertical offsets;

[0022] Based on the cropping size, the image to be cropped is cropped to obtain a cropped image; wherein, the larger image between the first visible light image and the first thermal infrared image is used as the image to be cropped; and the smaller image between the first visible light image and the first thermal infrared image is used as the reference image;

[0023] The cropped image is resampled to obtain a matching image that matches the reference image; the matching image has the same resolution as the reference image.

[0024] As an improvement to the above scheme, the step of calculating the horizontal and vertical offsets between the first thermal infrared image and the first visible light image based on the imaging range of the visible light camera and the imaging range of the thermal infrared camera includes:

[0025] The horizontal offset is calculated according to the following formula:

[0026]

[0027] In the formula, W2 represents the horizontal imaging range of the second camera, W1 represents the horizontal imaging range of the first camera, d represents the distance between the UAV and the target glass curtain wall, w2 represents the horizontal dimension of the sensor of the second camera, w1 represents the horizontal dimension of the sensor of the first camera, f2 represents the focal length of the lens of the second camera, f1 represents the focal length of the lens of the first camera, and Dw represents the horizontal distance between the lens mounting positions of the first camera and the second camera.

[0028] The vertical offset is calculated according to the following formula:

[0029]

[0030] In the formula, H2 represents the vertical imaging range of the second camera, H1 represents the vertical imaging range of the first camera, d represents the distance between the UAV and the target glass curtain wall, h2 represents the vertical dimension of the sensor of the second camera, h1 represents the vertical dimension of the sensor of the first camera, f2 represents the focal length of the lens of the second camera, f1 represents the focal length of the lens of the first camera, and Dh represents the vertical distance between the lens mounting positions of the first camera and the second camera.

[0031] The first camera is used to capture the reference image, and the second camera is used to capture the image to be cropped.

[0032] As an improvement to the above solution, obtaining the cutting dimensions based on the horizontal and vertical offsets includes:

[0033] The cutting dimensions are calculated according to the following formula:

[0034]

[0035] In the formula, l represents the number of left-side cropping pixel columns, r represents the number of right-side cropping pixel columns, u represents the number of upper-side cropping pixel rows, b represents the number of lower-side cropping pixel rows, and m*n represents the resolution of the image to be cropped.

[0036] As an improvement to the above solution, the acquisition of one or more image sets obtained by photographing the target glass curtain wall, each image set including a first visible light image and a first thermal infrared image taken at the same waypoint, includes:

[0037] The camera with the smaller imaging range between the camera used to capture the first visible light image and the camera used to capture the first thermal infrared image is selected as the target camera.

[0038] Based on the camera parameters of the target camera, calculate the first spacing between each waypoint in the horizontal direction and the second spacing between each waypoint in the vertical direction:

[0039]

[0040] In the formula, Lx represents the first spacing, k1 represents the preset forward overlap rate, w represents the horizontal dimension of the target camera's sensor, and d represents the distance between the target camera and the target glass curtain wall; Ly represents the second spacing, k2 represents the preset lateral overlap rate, and h represents the vertical dimension of the target camera's sensor.

[0041] Based on the first spacing and the second spacing, one or more waypoints are determined;

[0042] One or more image sets are obtained by taking pictures of the target glass curtain wall at one or more waypoints.

[0043] To achieve the above objectives, this application also provides a three-dimensional modeling device for glass curtain walls, comprising:

[0044] The acquisition module is used to acquire one or more image groups obtained by photographing the target glass curtain wall, each image group including a first visible light image and a first thermal infrared image taken at the same waypoint;

[0045] The matching module is used to match the first visible light image and the first thermal infrared image in the same image group to obtain a second visible light image and a second thermal infrared image that match each other.

[0046] The identification module is used to identify the position of the glass panel pixels in the second thermal infrared image based on the temperature of each pixel in the second thermal infrared image;

[0047] The correction module is used to perform color correction on the glass panel pixels in the second visible light image based on the position of the glass panel pixels in the second thermal infrared image, so as to obtain a third visible light image;

[0048] The modeling module is used to perform three-dimensional modeling of the target glass curtain wall using one or more of the third visible light images.

[0049] To achieve the above objectives, this application also provides a three-dimensional modeling device for glass curtain walls, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the three-dimensional modeling method for glass curtain walls as described above.

[0050] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the glass curtain wall three-dimensional modeling method as described above.

[0051] Compared with the prior art, the embodiments of this application provide a method, apparatus, device, and storage medium for three-dimensional modeling of glass curtain walls. This involves acquiring one or more image sets of a target glass curtain wall, each image set including a first visible light image and a first thermal infrared image taken from the same waypoint; matching the first visible light image and the first thermal infrared image within the same image set to obtain a matching second visible light image and a second thermal infrared image; identifying the position of glass panel pixels in the second thermal infrared image based on the temperature of each pixel; performing color correction on the glass panel pixels in the second visible light image based on the position of the glass panel pixels in the second thermal infrared image to obtain a third visible light image; and using one or more of the third visible light images to perform three-dimensional modeling of the target glass curtain wall. Therefore, this application embodiment incorporates thermal infrared information on the basis of traditional UAV visible light photogrammetry. By identifying the position of glass panel pixels in the second thermal infrared image through the temperature of each pixel, the color correction of glass panel pixels in the second visible light image can be performed, which can eliminate the reflection and transmission texture of the glass curtain wall in the visible light image. Then, three-dimensional reconstruction is performed to construct a real, clear and complete high-precision three-dimensional model of the glass curtain wall. Attached Figure Description

[0052] Figure 1 This is a flowchart of a three-dimensional modeling method for glass curtain walls provided in an embodiment of this application;

[0053] Figure 2 It is a schematic diagram of a 3D model of a glass curtain wall;

[0054] Figure 3 This application provides an embodiment of the imaging range of different cameras at the same waypoint;

[0055] Figure 4 This application provides an embodiment of the imaging range of different cameras in the vertical direction;

[0056] Figure 5 This application provides an embodiment of the imaging range of different cameras in the horizontal direction;

[0057] Figure 6 This application provides a temperature statistical result in its embodiments;

[0058] Figure 7 This is a structural block diagram of a three-dimensional modeling device for glass curtain walls provided in an embodiment of this application;

[0059] Figure 8 This is a structural block diagram of a three-dimensional modeling device for glass curtain walls provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0061] In the description of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0062] In this application description, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0063] In this application description, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." The term "and / or" means at least one of the connected objects, such as A and / or B, indicating three cases: including only A, only B, and both A and B. Unless otherwise stated, the term "multiple" means two or more.

[0064] See Figure 1 , Figure 1 This is a flowchart illustrating a three-dimensional modeling method for glass curtain walls provided in an embodiment of this application. The three-dimensional modeling method for glass curtain walls includes:

[0065] S1. Acquire one or more image groups obtained by photographing the target glass curtain wall, each image group including a first visible light image and a first thermal infrared image taken at the same waypoint;

[0066] S2. Match the first visible light image and the first thermal infrared image in the same image group to obtain a matching second visible light image and a second thermal infrared image;

[0067] S3. Identify the position of the glass panel pixels in the second thermal infrared image based on the temperature of each pixel in the second thermal infrared image;

[0068] S4. Based on the position of the glass panel pixels in the second thermal infrared image, perform color correction on the glass panel pixels in the second visible light image to obtain a third visible light image;

[0069] S5. Using one or more of the third visible light images, perform a three-dimensional modeling of the target glass curtain wall.

[0070] It is worth noting that the UAV is equipped with a visible light camera and a thermal infrared camera, and then takes pictures of the target glass curtain wall at one or more waypoints to obtain an image set for each waypoint; wherein, the image set for each waypoint includes a first visible light image taken by the visible light camera and a first thermal infrared image taken by the thermal infrared camera.

[0071] Because the installation positions, focal lengths, and sensor sizes of visible light and thermal infrared cameras may differ, the imaging ranges of visible light and thermal infrared images taken at the same waypoint will not completely overlap. Therefore, image matching is necessary. Specifically, the first visible light image and the first thermal infrared image are matched to obtain a matched second visible light image and a second thermal infrared image.

[0072] Glass curtain walls are mainly composed of glass panels and frames. By using the temperature difference between the glass panels and the frame, the position of the glass panel pixels in the second thermal infrared image can be identified using the temperature of each pixel in the second thermal infrared image.

[0073] Then, based on the position of the glass panel pixels in the second thermal infrared image, the colors of the glass panel pixels in the second visible light image are corrected to obtain the third visible light image.

[0074] Finally, by utilizing third-party visible light images, a 3D model of the target glass curtain wall is completed. This avoids issues such as distortion, voids, or texture inconsistencies that can easily occur in glass curtain walls, leading to poor modeling quality or even modeling failure. The result is a realistic, clear, and complete high-precision 3D model of the glass curtain wall. Figure 2 , Figure 2 (a) is a glass curtain wall constructed using existing technology. Figure 2 (b) is a glass curtain wall constructed using the embodiments of this application, which has a significantly better effect.

[0075] In an optional embodiment, acquiring one or more image sets obtained by photographing the target glass curtain wall, each image set including a first visible light image and a first thermal infrared image taken at the same waypoint, includes:

[0076] The camera with the smaller imaging range between the camera used to capture the first visible light image and the camera used to capture the first thermal infrared image is selected as the target camera.

[0077] Based on the camera parameters of the target camera, calculate the first spacing between each waypoint in the horizontal direction and the second spacing between each waypoint in the vertical direction:

[0078]

[0079] In the formula, Lx represents the first spacing, k1 represents the preset forward overlap rate, w represents the horizontal dimension of the target camera's sensor, and d represents the distance between the target camera and the target glass curtain wall; Ly represents the second spacing, k2 represents the preset lateral overlap rate, and h represents the vertical dimension of the target camera's sensor.

[0080] Based on the first spacing and the second spacing, one or more waypoints are determined;

[0081] One or more image sets are obtained by taking pictures of the target glass curtain wall at one or more waypoints.

[0082] In this embodiment, based on the principle of close-up photogrammetry, a three-dimensional flight path is designed to be taken close to the surface of the glass curtain wall. The pixel accuracy is designed to be no less than sub-centimeter level, and the flight path adopts a circular or "bow" shape. Starting from a point at the bottom of the glass curtain wall facade, a "bow" shaped flight path is designed, the positions of all waypoints are calculated, and finally, they are connected to form a complete three-dimensional flight path. "Circling" means that the UAV flies around the glass curtain wall from top to bottom or from bottom to top. "Bow" shaped means that the UAV flies in a "bow" shape over each surface of the glass curtain wall. A UAV equipped with a visible light camera and a thermal infrared camera is used to execute the three-dimensional flight path calculation results, ensuring that the distance between the cameras (visible light camera and thermal infrared camera) and the target glass curtain wall remains consistent. First visible light images and first thermal infrared images of the target glass curtain wall are acquired, ensuring that one first visible light image and one first thermal infrared image are captured for each waypoint.

[0083] Specifically, the camera with the smaller imaging range between the camera used to capture the first visible light image and the camera used to capture the first thermal infrared image is selected as the target camera. For example, the imaging range is determined based on the camera lens focal length; the larger the focal length, the smaller the imaging range. Based on the target camera's camera parameters, the first spacing between each waypoint in the horizontal direction is calculated. The second spacing between each waypoint in the vertical direction: In the formula, W represents the horizontal imaging range of the glass curtain wall, and H represents the vertical imaging range of the glass curtain wall. Preferably, both the preset forward overlap rate and the preset lateral overlap rate are not less than 70% to improve modeling quality.

[0084] For example, the focal length of an infrared camera is greater than that of a visible light camera, but its imaging range is smaller. The infrared camera is identified as the target camera. Based on the camera parameters of the target camera, the first and second gaps are calculated, and then waypoints are determined to achieve the imaging of the target glass curtain wall.

[0085] In this embodiment, a camera with a smaller imaging range is selected as the basis for waypoint selection to ensure that the waypoint spacing is set reasonably. This allows for more focused acquisition of key details during the shooting process and avoids missing important information due to excessive waypoint spacing.

[0086] In an optional embodiment, matching the first visible light image and the first thermal infrared image of the same image group to obtain a mutually matched second visible light image and a second thermal infrared image includes:

[0087] Based on the imaging range of the visible light camera and the imaging range of the thermal infrared camera, calculate the horizontal and vertical offsets between the first thermal infrared image and the first visible light image.

[0088] The cutting dimensions are obtained based on the horizontal and vertical offsets;

[0089] Based on the cropping size, the image to be cropped is cropped to obtain a cropped image; wherein, the larger image between the first visible light image and the first thermal infrared image is used as the image to be cropped; and the smaller image between the first visible light image and the first thermal infrared image is used as the reference image;

[0090] The cropped image is resampled to obtain a matching image that matches the reference image; the matching image has the same resolution as the reference image.

[0091] It is worth noting that, since the installation positions, focal lengths, and CMOS sensor sizes of visible light cameras and thermal infrared cameras may not be the same, the imaging ranges of visible light images and thermal infrared images taken at the same waypoint will not completely overlap, so image matching is required.

[0092] First, the smaller image from the first thermal infrared image and the first visible light image is used as the reference image. Based on the horizontal and vertical offsets between the two images, the larger image (the image to be cropped) is cropped so that the cropped image (the cropped image) has the same size as the reference image. Next, the cropped image is resampled so that the resampled image (the matching image) has the same resolution as the reference image. This ensures that pixels at the same location in the second visible light image and the second thermal infrared image correspond one-to-one in spatial order, providing support for subsequent location of glass panel pixels in the second visible light image.

[0093] In an optional embodiment, calculating the horizontal and vertical offsets between the first thermal infrared image and the first visible light image based on the imaging range of the visible light camera and the imaging range of the thermal infrared camera includes:

[0094] The horizontal offset is calculated according to the following formula:

[0095]

[0096] In the formula, W2 represents the horizontal imaging range of the second camera, W1 represents the horizontal imaging range of the first camera, d represents the distance between the UAV and the target glass curtain wall, w2 represents the horizontal dimension of the sensor of the second camera, w1 represents the horizontal dimension of the sensor of the first camera, f2 represents the focal length of the lens of the second camera, f1 represents the focal length of the lens of the first camera, and Dw represents the horizontal distance between the lens mounting positions of the first camera and the second camera.

[0097] The vertical offset is calculated according to the following formula:

[0098]

[0099] In the formula, H2 represents the vertical imaging range of the second camera, H1 represents the vertical imaging range of the first camera, d represents the distance between the UAV and the target glass curtain wall, h2 represents the vertical dimension of the sensor of the second camera, h1 represents the vertical dimension of the sensor of the first camera, f2 represents the focal length of the lens of the second camera, f1 represents the focal length of the lens of the first camera, and Dh represents the vertical distance between the lens mounting positions of the first camera and the second camera.

[0100] The first camera is used to capture the reference image, and the second camera is used to capture the image to be cropped.

[0101] In one optional embodiment, obtaining the cutting dimensions based on the horizontal and vertical offsets includes:

[0102] The cutting dimensions are calculated according to the following formula:

[0103]

[0104] In the formula, l represents the number of left-side cropping pixel columns, r represents the number of right-side cropping pixel columns, u represents the number of upper-side cropping pixel rows, b represents the number of lower-side cropping pixel rows, m*n represents the resolution of the image to be cropped, m is the number of horizontal pixels, and n is the number of vertical pixels.

[0105] For example, in photogrammetry, visible light cameras typically use wide-angle lenses, so it is assumed that the focal length of the thermal infrared camera is greater than that of the visible light camera, such as... Figure 3In the Cartesian coordinate system OXY shown, the imaging range is R, and the visible light imaging range is G; G > R. Therefore, the image to be cropped is the first visible light image, and the reference image is the first thermal infrared image, which means that the first visible light image is cropped and resampled. Then, the cropped and resampled first visible light image and the first thermal infrared image match each other; that is, the cropped and resampled first visible light image is the matching image and also the second visible light image. The first thermal infrared image is the reference image and also the second thermal infrared image. The first camera is a thermal infrared camera, and the second camera is a visible light camera. Due to the different camera installation positions, R will not be exactly in the center of G, but will have a certain offset, assuming the horizontal offset is Sx and the vertical offset is Sy.

[0106] like Figure 4 As shown, assuming the vertical distance between the mounting positions of the first camera and the second camera lenses is Dh, then Combination We can obtain:

[0107]

[0108] like Figure 5 As shown, assuming the horizontal distance between the mounting positions of the first camera and the second camera lenses is Dw, then Combination We can obtain:

[0109]

[0110] Assuming the resolution of the image to be cropped is m*n (m is the number of horizontal pixels, n is the number of vertical pixels), the image is cropped by cutting 1 column of pixels from the left, 1 / 2 columns of pixels from the right, 1 / 3 rows of pixels from the top, and 1 / 4 rows of pixels from the bottom. This ensures that the cropped image area matches the baseline image area. Based on the resolution of the image to be cropped, we can obtain:

[0111]

[0112] By resampling the cropped image so that the matching image, i.e. the resampled cropped image, has the same resolution as the reference image, pixels at the same location in the second visible light image and the second thermal infrared image can be matched one-to-one with the same spatial position.

[0113] Similarly, if the thermal infrared camera has a larger imaging range and a smaller focal length, then the image to be cropped is the first thermal infrared image, and the reference image is the first visible light image. That is, the first thermal infrared image is cropped and resampled. Then, the cropped and resampled first thermal infrared image and the first visible light image are matched with each other. In other words, the cropped and resampled first thermal infrared image is the matched image and also the second thermal infrared image. The first visible light image is the reference image and also the second visible light image. The first camera is a visible light camera and the second camera is a thermal infrared camera, which will not be elaborated further here.

[0114] The calculation can still be performed as follows: the calculated horizontal offset is Sx, the vertical offset is Sy, and the number of left cropped pixel columns is l, the number of right cropped pixel columns is r, the number of upper cropped pixel rows is u, and the number of lower cropped pixel rows is b. If these numbers are negative, it indicates that the thermal infrared image has been cropped. In other words, if the number of left cropped pixel columns is l, the number of right cropped pixel columns is r, the number of upper cropped pixel rows is u, and the number of lower cropped pixel rows is b, the visible light image has been cropped; if they are negative, the thermal infrared image has been cropped. Subsequently, the cropped image is resampled to match the uncropped and resampled images.

[0115] In one optional embodiment, identifying the position of the glass panel pixels in the second thermal infrared image based on the temperature of each pixel in the second thermal infrared image includes:

[0116] The pixels in the second thermal infrared image are sampled, and the temperature of all sampled points is obtained based on the mapping relationship between pixel color and temperature in the thermal infrared image.

[0117] The temperature of all the sampling points is statistically analyzed to determine the temperature range of the glass panel pixels in the second thermal infrared image;

[0118] The pixel location where the target temperature is located is taken as the location of the glass panel pixel in the second thermal infrared image; wherein, the target temperature is the temperature of the pixel in the second thermal infrared image that belongs to the temperature range.

[0119] It is worth noting that in thermal infrared images, since the glass panel is mainly composed of structural adhesive and aluminum alloy frame structure, the temperature of the glass panel is often different from that of the edge structural adhesive and frame due to the influence of indoor and outdoor temperatures. In particular, it may be higher or lower due to the influence of indoor temperature. Therefore, the pixels belonging to the glass panel in the thermal infrared image can be identified by temperature.

[0120] First, pixels in the second thermal infrared image are sampled. Based on the mapping relationship between pixel color and temperature in the thermal infrared image, the temperature of all sampled points is obtained. Preferably, pixels in the second thermal infrared image are sampled uniformly. In the thermal infrared image, different colors represent different temperatures, and there is a mapping relationship between them. By analyzing the thermal infrared image, the temperature of each sampled point can be obtained.

[0121] Because the area of ​​the glass panel is much larger than the area occupied by the structural adhesive and frame in the image, the number of sampling points representing the temperature of the glass panel is much larger than the number of sampling points for the structural adhesive and frame. Simultaneously, because the surface temperature of the glass panel is relatively stable, statistical analysis of the temperature of the sampling points shows that the number of sampling points within a certain temperature range is significantly higher. The difference between the number of sampling points in this temperature range and the number of sampling points in other temperature ranges exceeds a preset difference threshold. This temperature range is the temperature range of the glass panel pixels in the second thermal infrared image. Figure 6 As shown, the number of sampling points in the 36° to 38° range is significantly larger. Therefore, the temperature range of 36° to 38° is taken as the temperature range of the glass panel pixels in the second thermal infrared image.

[0122] Once the temperature range corresponding to the glass panel in the second thermal infrared image is determined, it can be determined whether each pixel in the second thermal infrared image belongs to the glass panel. Specifically, the pixel position where the target temperature is located is taken as the position of the glass panel pixel in the second thermal infrared image; the target temperature is the temperature of the pixel in the second thermal infrared image that belongs to that temperature range. For example, for a certain pixel in the second thermal infrared image, if the temperature of the pixel belongs to that temperature range, then the pixel is a glass panel pixel; otherwise, it is not a glass panel pixel.

[0123] In an optional embodiment, the step of color-correcting the glass panel pixels in the second visible light image based on the position of the glass panel pixels in the second thermal infrared image to obtain a third visible light image includes:

[0124] The position of the glass panel pixels in the second visible light image is determined based on the position of the glass panel pixels in the second thermal infrared image;

[0125] Determine the primary color of the glass panel in the second visible light image;

[0126] Based on the position of the glass panel pixels in the second visible light image, the color of all the glass panel pixels in the second visible light image is set to the primary color to obtain the third visible light image.

[0127] It is understandable that after determining whether each pixel in the second thermal infrared image belongs to the glass panel, the pixel positions in the second visible light image and the second thermal infrared image are matched one-to-one. Therefore, the position of the glass panel pixel in the second visible light image can be obtained from the position of the glass panel pixel in the second thermal infrared image, thereby determining the range of the glass panel in the second visible light image. That is, based on the position of the glass panel pixel in the second thermal infrared image, the same position is found in the second visible light image as the position of the glass panel pixel in the second visible light image.

[0128] Next, the primary color of the glass panel in the second visible light image is determined. This primary color is the color of the glass without reflection or transmission, which can be determined by the color of the glass without reflection or transmission in the visible light image of the target glass curtain wall. Let's assume the primary color of the glass panel is (r, g, b). Based on the position of the glass panel pixels in the second visible light image determined in the previous step, the color of all corresponding glass panel pixels in the second visible light image is set to (r, g, b) to eliminate reflection and transmission textures in the second visible light image, thus obtaining the third visible light image.

[0129] Using one or more third visible light images, a 3D model of the target glass curtain wall is completed. Specifically, aerial triangulation is performed based on UAV POS (position and orientation system) data and one or more third visible light images to generate a dense point cloud, construct a TIN (irregular triangle) network, and perform texture mapping to finally obtain a realistic 3D model of the target glass curtain wall.

[0130] See Figure 7 , Figure 7 This is a structural block diagram of a three-dimensional modeling device 10 for glass curtain walls provided in an embodiment of this application. The three-dimensional modeling device for glass curtain walls includes:

[0131] The acquisition module is used to acquire one or more image groups obtained by photographing the target glass curtain wall, each image group including a first visible light image and a first thermal infrared image taken at the same waypoint;

[0132] The matching module is used to match the first visible light image and the first thermal infrared image in the same image group to obtain a second visible light image and a second thermal infrared image that match each other.

[0133] The identification module is used to identify the position of the glass panel pixels in the second thermal infrared image based on the temperature of each pixel in the second thermal infrared image;

[0134] The correction module is used to perform color correction on the glass panel pixels in the second visible light image based on the position of the glass panel pixels in the second thermal infrared image, so as to obtain a third visible light image;

[0135] The modeling module is used to perform three-dimensional modeling of the target glass curtain wall using one or more of the third visible light images.

[0136] Optionally, identifying the position of the glass panel pixels in the second thermal infrared image based on the temperature of each pixel in the second thermal infrared image includes:

[0137] The pixels in the second thermal infrared image are sampled, and the temperature of all sampled points is obtained based on the mapping relationship between pixel color and temperature in the thermal infrared image.

[0138] The temperature of all the sampling points is statistically analyzed to determine the temperature range of the glass panel pixels in the second thermal infrared image;

[0139] The pixel location where the target temperature is located is taken as the location of the glass panel pixel in the second thermal infrared image; wherein, the target temperature is the temperature of the pixel in the second thermal infrared image that belongs to the temperature range.

[0140] Optionally, the step of color-correcting the glass panel pixels in the second visible light image based on the position of the glass panel pixels in the second thermal infrared image to obtain a third visible light image includes:

[0141] The position of the glass panel pixels in the second visible light image is determined based on the position of the glass panel pixels in the second thermal infrared image;

[0142] Determine the primary color of the glass panel in the second visible light image;

[0143] Based on the position of the glass panel pixels in the second visible light image, the color of all the glass panel pixels in the second visible light image is set to the primary color to obtain the third visible light image.

[0144] Optionally, matching the first visible light image and the first thermal infrared image within the same image group to obtain a matching second visible light image and a second thermal infrared image includes:

[0145] Based on the imaging range of the visible light camera and the imaging range of the thermal infrared camera, calculate the horizontal and vertical offsets between the first thermal infrared image and the first visible light image.

[0146] The cutting dimensions are obtained based on the horizontal and vertical offsets;

[0147] Based on the cropping size, the image to be cropped is cropped to obtain a cropped image; wherein, the larger image between the first visible light image and the first thermal infrared image is used as the image to be cropped; and the smaller image between the first visible light image and the first thermal infrared image is used as the reference image;

[0148] The cropped image is resampled to obtain a matching image that matches the reference image; the matching image has the same resolution as the reference image.

[0149] Optionally, calculating the horizontal and vertical offsets between the first thermal infrared image and the first visible light image based on the imaging range of the visible light camera and the imaging range of the thermal infrared camera includes:

[0150] The horizontal offset is calculated according to the following formula:

[0151]

[0152] In the formula, W2 represents the horizontal imaging range of the second camera, W1 represents the horizontal imaging range of the first camera, d represents the distance between the UAV and the target glass curtain wall, w2 represents the horizontal dimension of the sensor of the second camera, w1 represents the horizontal dimension of the sensor of the first camera, f2 represents the focal length of the lens of the second camera, f1 represents the focal length of the lens of the first camera, and Dw represents the horizontal distance between the lens mounting positions of the first camera and the second camera.

[0153] The vertical offset is calculated according to the following formula:

[0154]

[0155] In the formula, H2 represents the vertical imaging range of the second camera, H1 represents the vertical imaging range of the first camera, d represents the distance between the UAV and the target glass curtain wall, h2 represents the vertical dimension of the sensor of the second camera, h1 represents the vertical dimension of the sensor of the first camera, f2 represents the focal length of the lens of the second camera, f1 represents the focal length of the lens of the first camera, and Dh represents the vertical distance between the lens mounting positions of the first camera and the second camera.

[0156] The first camera is used to capture the reference image, and the second camera is used to capture the image to be cropped.

[0157] Optionally, obtaining the cutting size based on the horizontal offset and the vertical offset includes:

[0158] The cutting dimensions are calculated according to the following formula:

[0159]

[0160] In the formula, l represents the number of left-side cropping pixel columns, r represents the number of right-side cropping pixel columns, u represents the number of upper-side cropping pixel rows, b represents the number of lower-side cropping pixel rows, and m*n represents the resolution of the image to be cropped.

[0161] Optionally, the acquisition of one or more image sets obtained by photographing the target glass curtain wall, each image set including a first visible light image and a first thermal infrared image taken at the same waypoint, includes:

[0162] The camera with the smaller imaging range between the camera used to capture the first visible light image and the camera used to capture the first thermal infrared image is selected as the target camera.

[0163] Based on the camera parameters of the target camera, calculate the first spacing between each waypoint in the horizontal direction and the second spacing between each waypoint in the vertical direction:

[0164]

[0165] In the formula, Lx represents the first spacing, k1 represents the preset forward overlap rate, w represents the horizontal dimension of the target camera's sensor, and d represents the distance between the target camera and the target glass curtain wall; Ly represents the second spacing, k2 represents the preset lateral overlap rate, and h represents the vertical dimension of the target camera's sensor.

[0166] Based on the first spacing and the second spacing, one or more waypoints are determined;

[0167] One or more image sets are obtained by taking pictures of the target glass curtain wall at one or more waypoints.

[0168] It is worth noting that the working process of each module in the glass curtain wall three-dimensional modeling device 10 described in this application embodiment can refer to the working process of the glass curtain wall three-dimensional modeling method described in the above embodiment, and will not be repeated here.

[0169] This application provides a 3D modeling device 10 for glass curtain walls. It acquires one or more image sets of a target glass curtain wall, each image set including a first visible light image and a first thermal infrared image taken from the same waypoint. The first visible light image and the first thermal infrared image from the same image set are matched to obtain a matching second visible light image and a second thermal infrared image. The position of the glass panel pixels in the second thermal infrared image is identified based on the temperature of each pixel. Color correction is performed on the glass panel pixels in the second visible light image based on their position to obtain a third visible light image. One or more of the third visible light images are then used to perform a 3D model of the target glass curtain wall. Therefore, this application, based on traditional UAV visible light photogrammetry, incorporates thermal infrared information. By identifying the position of the glass panel pixels in the second thermal infrared image based on the temperature of each pixel, and then performing color correction on the glass panel pixels in the second visible light image, it can eliminate reflection and transmission textures of the glass curtain wall in the visible light image. Further 3D reconstruction is then performed to construct a realistic, clear, and complete high-precision 3D model of the glass curtain wall.

[0170] Furthermore, this application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the glass curtain wall three-dimensional modeling method as described in any of the above embodiments.

[0171] Furthermore, this application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the glass curtain wall three-dimensional modeling method as described in any of the above embodiments.

[0172] See Figure 8 , Figure 8 This is a structural block diagram of a 3D modeling device 20 for glass curtain walls provided in an embodiment of this application. The 3D modeling device 20 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described embodiments of the 3D modeling method for glass curtain walls. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments.

[0173] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the glass curtain wall 3D modeling device 20.

[0174] The 3D modeling device 20 for glass curtain walls may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the 3D modeling device 20 for glass curtain walls and does not constitute a limitation on the 3D modeling device 20. It may include more or fewer components than shown in the diagram, or combine certain components, or use different components. For example, the 3D modeling device 20 for glass curtain walls may also include input / output devices, network access devices, buses, etc.

[0175] The processor 21 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the glass curtain wall 3D modeling equipment 20, connecting all parts of the equipment via various interfaces and lines.

[0176] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the glass curtain wall 3D modeling device 20 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0177] The modules / units integrated into the glass curtain wall 3D modeling equipment 20, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0178] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0179] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A method for three-dimensional modeling of glass curtain walls, characterized in that, include: Acquire one or more image sets of the target glass curtain wall by taking pictures, each image set including a first visible light image and a first thermal infrared image taken at the same waypoint; The first visible light image and the first thermal infrared image in the same image group are matched to obtain a second visible light image and a second thermal infrared image that match each other. Based on the temperature of each pixel in the second thermal infrared image, identify the position of the glass panel pixel in the second thermal infrared image; Based on the position of the glass panel pixels in the second thermal infrared image, the glass panel pixels in the second visible light image are color-corrected to obtain a third visible light image; The target glass curtain wall is modeled in three dimensions using one or more of the third visible light images.

2. The three-dimensional modeling method for glass curtain walls as described in claim 1, characterized in that, The step of identifying the position of the glass panel pixels in the second thermal infrared image based on the temperature of each pixel in the second thermal infrared image includes: The pixels in the second thermal infrared image are sampled, and the temperature of all sampled points is obtained based on the mapping relationship between pixel color and temperature in the thermal infrared image. The temperature of all the sampling points is statistically analyzed to determine the temperature range of the glass panel pixels in the second thermal infrared image; The pixel location where the target temperature is located is taken as the location of the glass panel pixel in the second thermal infrared image; wherein, the target temperature is the temperature of the pixel in the second thermal infrared image that belongs to the temperature range.

3. The three-dimensional modeling method for glass curtain walls as described in claim 1, characterized in that, The step of color-correcting the glass panel pixels in the second visible light image based on the position of the glass panel pixels in the second thermal infrared image to obtain a third visible light image includes: The position of the glass panel pixels in the second visible light image is determined based on the position of the glass panel pixels in the second thermal infrared image; Determine the primary color of the glass panel in the second visible light image; Based on the position of the glass panel pixels in the second visible light image, the color of all the glass panel pixels in the second visible light image is set to the primary color to obtain the third visible light image.

4. The three-dimensional modeling method for glass curtain walls as described in claim 1, characterized in that, The step of matching the first visible light image and the first thermal infrared image of the same image group to obtain a matching second visible light image and a second thermal infrared image includes: Based on the imaging range of the visible light camera and the imaging range of the thermal infrared camera, calculate the horizontal and vertical offsets between the first thermal infrared image and the first visible light image. The cutting dimensions are obtained based on the horizontal and vertical offsets; Based on the cropping size, the image to be cropped is cropped to obtain a cropped image; wherein, the larger image between the first visible light image and the first thermal infrared image is used as the image to be cropped; and the smaller image between the first visible light image and the first thermal infrared image is used as the reference image; The cropped image is resampled to obtain a matching image that matches the reference image; the matching image has the same resolution as the reference image.

5. The three-dimensional modeling method for glass curtain walls as described in claim 4, characterized in that, The step of calculating the horizontal and vertical offsets between the first thermal infrared image and the first visible light image based on the imaging range of the visible light camera and the imaging range of the thermal infrared camera includes: The horizontal offset is calculated according to the following formula: In the formula, W2 represents the horizontal imaging range of the second camera, W1 represents the horizontal imaging range of the first camera, d represents the distance between the UAV and the target glass curtain wall, w2 represents the horizontal dimension of the sensor of the second camera, w1 represents the horizontal dimension of the sensor of the first camera, f2 represents the focal length of the lens of the second camera, f1 represents the focal length of the lens of the first camera, and Dw represents the horizontal distance between the lens mounting positions of the first camera and the second camera. The vertical offset is calculated according to the following formula: In the formula, H2 represents the vertical imaging range of the second camera, H1 represents the vertical imaging range of the first camera, d represents the distance between the UAV and the target glass curtain wall, h2 represents the vertical dimension of the sensor of the second camera, h1 represents the vertical dimension of the sensor of the first camera, f2 represents the focal length of the lens of the second camera, f1 represents the focal length of the lens of the first camera, and Dh represents the vertical distance between the lens mounting positions of the first camera and the second camera. The first camera is used to capture the reference image, and the second camera is used to capture the image to be cropped.

6. The three-dimensional modeling method for glass curtain walls as described in claim 5, characterized in that, The process of obtaining the cutting dimensions based on the horizontal and vertical offsets includes: The cutting dimensions are calculated according to the following formula: In the formula, l represents the number of left-side cropping pixel columns, r represents the number of right-side cropping pixel columns, u represents the number of upper-side cropping pixel rows, b represents the number of lower-side cropping pixel rows, and m*n represents the resolution of the image to be cropped.

7. The three-dimensional modeling method for glass curtain walls as described in claim 1, characterized in that, The acquisition of one or more image sets obtained by photographing the target glass curtain wall, each image set including a first visible light image and a first thermal infrared image taken at the same waypoint, includes: The camera with the smaller imaging range between the camera used to capture the first visible light image and the camera used to capture the first thermal infrared image is selected as the target camera. Based on the camera parameters of the target camera, calculate the first spacing between each waypoint in the horizontal direction and the second spacing between each waypoint in the vertical direction: In the formula, Lx represents the first spacing, k1 represents the preset forward overlap rate, w represents the horizontal dimension of the target camera's sensor, and d represents the distance between the target camera and the target glass curtain wall; Ly represents the second spacing, k2 represents the preset lateral overlap rate, and h represents the vertical dimension of the target camera's sensor. Based on the first spacing and the second spacing, one or more waypoints are determined; One or more image sets are obtained by taking pictures of the target glass curtain wall at one or more waypoints.

8. A three-dimensional modeling device for glass curtain walls, characterized in that, include: The acquisition module is used to acquire one or more image groups obtained by photographing the target glass curtain wall, each image group including a first visible light image and a first thermal infrared image taken at the same waypoint; The matching module is used to match the first visible light image and the first thermal infrared image in the same image group to obtain a second visible light image and a second thermal infrared image that match each other. The identification module is used to identify the position of the glass panel pixels in the second thermal infrared image based on the temperature of each pixel in the second thermal infrared image; The correction module is used to perform color correction on the glass panel pixels in the second visible light image based on the position of the glass panel pixels in the second thermal infrared image, so as to obtain a third visible light image; The modeling module is used to perform three-dimensional modeling of the target glass curtain wall using one or more of the third visible light images.

9. A three-dimensional modeling device for glass curtain walls, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the three-dimensional modeling method for glass curtain walls as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the three-dimensional modeling method for glass curtain walls as described in any one of claims 1 to 7.

Images