Method, image stitching system, computer program (image stitching for high resolution scans)

Abstract

To provide a computer-implemented method, image stitching system and computer program for image stitching for a plurality of digital images of an infrastructure surface for defect detection.SOLUTION: A method 100 includes a step 102 of providing a plurality of partial digital images of an infrastructure surface, a step 104 of extracting global positioning system meta-data from data corresponding to the partial digital images, a step 108 of determining feature descriptions of features in the partial digital images, and a step 110 of, for the partial digital images based on the extracted global positioning system meta-data, determining an affinity matrix using the feature descriptions of adjacent partial digital images to incrementally position each of the partial digital images such that an overview image of the infrastructure surface is produced by stitching the partial digital images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates generally to methods for image stitching of multiple digital images, and more particularly to a computer-implemented method for image stitching of multiple digital images of an infrastructure surface for defect detection. The present disclosure further relates to an image stitching system, and a computer program product for image stitching of multiple digital images of an infrastructure surface for defect detection. [Background technology]

[0002] Public infrastructure around the world is becoming increasingly aging, and its reliability can weaken over time. For example, constant manual human inspection of public infrastructure components, such as buildings, bridges, roads, pipelines, power grid pylons, wind turbines, solar power systems, and dams, is expensive and often dangerous. Therefore, automated methods for monitoring the structural health of public infrastructure have become widespread in recent years. The availability of automated drones with high-quality imaging materials to easily access and scan the surfaces of walls, bridges, or roads can facilitate the evaluation of structural surfaces by locating, identifying, and evaluating defects such as cracks, rust, or algae. However, the interpretation of defects can be improved by relying at least on high-resolution magnified views of the defects, as well as their location and size to better assess their impact on the entire structure.

[0003] To thoroughly inspect a large infrastructure component, a single digital image of the surface is not sufficient. In fact, multiple digital images of the large surface area of ​​interest may be required. To reconstruct an accurate overview of the infrastructure component, the multiple digital images need to be mapped to each other. Some known methods for attempting to solve this problem are based on panoramic scenes created from a single photograph. However, such methods are laborious, time-consuming, and often result in inconsistencies. Unfortunately, such problems also arise in known techniques currently available for seamlessly mapping multiple digital images. Additionally, such known techniques may be disadvantageous in terms of the accuracy and speed of feature detection during the image stitching process. This may result in inaccurate detection of potential defects. Summary of the Invention [Problem to be solved by the invention]

[0004] Therefore, it is desirable to provide an image stitching technique to overcome the limitations of existing techniques for image stitching and provide accurate results for use in public infrastructure health monitoring based on improved deterministic algorithms. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, a computer-implemented method for image stitching of multiple digital images of an infrastructure surface for defect detection may be provided. The method may include providing multiple partial digital images of the infrastructure surface. The method may include extracting global positioning system metadata from data corresponding to the partial digital images. The method may include determining characterizations of features in one or more partial digital images. The method may include performing a scheduled processing sequence on the partial digital images based on the extracted global positioning system metadata. This includes determining a similarity matrix using the characterizations of adjacent partial digital images to progressively position each of the partial digital images such that an overview image of the infrastructure surface is generated by digitally stitching the multiple partial digital images together in an iterative manner.

[0006] According to another aspect of the present disclosure, an image stitching system for image stitching of multiple digital images of an infrastructure surface for defect detection may be provided. The system may include a memory for storing program code portions, the memory coupled to a processor. The processor, when executing the program code portions, may enable the processor to receive multiple partial digital images of the infrastructure surface. The processor may be further capable of extracting global positioning system metadata from data corresponding to the partial digital images. The processor, when executing the program code portions, may be further capable of determining characterizations of features in one or more partial digital images. The processor may be further capable of performing a scheduled processing sequence on the multiple partial digital images based on the extracted global positioning system metadata. This includes determining a similarity matrix using the characterizations of adjacent partial digital images to progressively position each of the partial digital images such that an overview image of the infrastructure surface is generated by digitally stitching the multiple partial digital images together in an iterative manner.

[0007] The methods and systems for image stitching of multiple digital images of infrastructure surfaces for defect detection disclosed herein may provide multiple features, technical effects, contributions, or improvements, or combinations thereof.

[0008] For example, embodiments of the methods and systems disclosed herein may include using innovative algorithms to estimate robust and accurate transformation matrices for planar services, such as scans of public infrastructure components.

[0009] Additionally, embodiments of the methods and systems disclosed herein may use Global Positioning System (GPS) data for initial positioning of partial digital images to define an error-resilient processing order. For example, according to at least some embodiments of the present disclosure, GPS data may be used to define a schedule for processing partial digital images. In such embodiments, the disclosed methods and systems may scale linearly with the number of images, thereby facilitating computational speed. Thus, in such embodiments, results may be better than those achieved using conventional techniques, such as techniques that include few or no non-overlapping zones in overview images. This may be primarily due to the use of GPS coordinates in combination with visual alignment, as purely visual approaches often have difficulty aligning images with similar, repetitive, homogeneous structures, such as dozens or hundreds of identical windows in a tall building.

[0010] Furthermore, embodiments of the methods and systems disclosed herein may be resilient to effects resulting from outlier images that may be present in multiple captured partial digital images. Such outlier images may occur when a camera on an airborne object, such as a drone, is used. For example, the camera may be out of focus or may not have a viewing angle perpendicular to the surface. For example, if there is an obstacle that visually blocks a portion of the surface, the camera may not be able to adjust its orientation to have a viewing angle perpendicular to the surface. Additionally, reflections may occur on the imaged digital surface or descent may occur due to camera movement. Embodiments of the methods and systems disclosed herein may be resilient to all of these issues and may self-correct such errors.

[0011] Resiliency to outlier images that may occur during capture of a partial digital image may be primarily addressed by the feature detection, feature matching, and transformation matrix estimation aspects of at least some embodiments of the methods and systems disclosed herein. Additionally, embodiments of the methods and systems disclosed herein may include validation that may check properties of the estimated matrix for acceptance or rejection.

[0012] To determine a processing schedule, which may be referred to as a stitching sequence, for fast cumulative stitching, embodiments of the methods and systems disclosed herein may rely on the fact that images with small Euclidean distances between GPS locations may have large overlaps. Furthermore, in embodiments of the methods and systems disclosed herein, the initiation of the stitching process follows a heuristic approach in that it starts at the center or central (e.g., from a GPS-based context) of multiple captured partial digital images of a particular scene. This strategy may provide the most natural appearance and minimize potential error propagation paths.

[0013] Additional embodiments of the present disclosure applicable to the methods and systems disclosed herein may facilitate additional advantages.

[0014] According to one optional embodiment, the method may also include rendering an overview image and mapping at least one identified defect, at least one annotation, or at least one measurement onto the overview image. The defects may be detected, for example, by a neural network system, such as a deep neural network, by various methods. Based on this, associated annotations or measurements determined from the detected elements in the overview image may also be superimposed on the overview image. As an example, the size of the defects, or the number of defects (e.g., per area), or the average distance between defects of a predetermined size may be determined and also displayed on the rendered overview image.

[0015] According to one optional embodiment, the method may also include performing at least one of scene registration, image warping (or dewarping), and pixel-use comparison for time evolution evaluation on multiple partial digital images of the infrastructure surface having different time stamps. This feature may facilitate determining evolutionary changes of defects or cracks in public infrastructure elements. Scene registration and additional activities in this context may be more suitable than single-image comparison, such as those performed by conventional techniques.

[0016] According to another optional embodiment, the method may also include generating a plurality of partial digital images using a camera on an unmanned vehicle. Such an embodiment facilitates imaging the facade of a tall building or, for example, a bridge pillar over a river. The unmanned vehicle (e.g., UAV) may be a pilotless aircraft, a drone, a helicopter, a model airplane, a dirigible, a balloon, a camera mounted on a vehicle that slides along a surface, such as a building cleaning lift, or the like. However, it should be possible to operate the UAV by remote control or an automatic control and positioning system.

[0017] According to another optional embodiment, determining the characterization may include using a scale-invariant feature transformation (SIFT) method, which may be used to determine local features in the image. Thus, when different captured images are generated at different viewing angles, local features may still be detectable and may be matched when seen in different, especially partially overlapping, images.

[0018] According to another optional embodiment, determining the characterization may include detecting at least one point of interest in the captured partial image. In such an embodiment, the SIFT method may be supported, and the stitching process may be accelerated.

[0019] According to another optional embodiment, determining the similarity matrix may include progressively locating each of the partial digital images using a RANSAC (e.g., known as Random Sample Consensus) method, which is a robust estimation procedure that may use a minimal set of randomly sampled correspondences in images to estimate image transformation parameters and find a solution with the best consensus on whether existing data, e.g., captured partial images, can be matched.

[0020] According to another optional embodiment, the progressive positioning of each of the partial digital images is performed by: i The existing intermediate spliced ​​image S i To splice into the transformation matrix H i In such an embodiment, the existing intermediate stitched image S i may be the result of past iterations or cycles of image stitching of captured partial digital images. A general perspective transformation matrix may thereby be a 3x3 matrix, while an affine transformation is a special case of perspective where the third row is [0,0,1]. An affine transformation matrix is ​​a combination of rotation, shear, translation, and scaling (explained in more detail below).

[0021] According to another optional embodiment, scheduling a processing sequence for the images may include sorting the partial digital images by increasing distance (such as Euclidean distance) from a central image of the infrastructure surface and starting the progressive positioning of each of the partial digital images from the central image. This approach provides a more robust method for the accurate successive stitching process than, for example, corner detector-based algorithms, because detector-based algorithms are not resilient to increasing deformations.

[0022] According to another optional embodiment, providing the partial digital image may include capturing the partial digital image at an angle parallel or near-parallel to the surface of interest of the infrastructure. In other words, the viewing angle of the camera for the drone may be perpendicular or near-perpendicular to the surface of interest. This may reduce the requirement for significant transformations, such as deskewing, of the partial digital image before it can be used in the stitching process.

[0023] According to another optional embodiment, providing the partial digital image may include capturing the partial digital image such that the image plane is not parallel to the imaged surface, and digitally deskewing the partial digital image to align the partial digital image parallel to the imaged surface. In such an embodiment, it may be possible to use digital images in which the camera's viewing angle is not perpendicular to the infrastructure surface, for example, when a visual obstruction prevents a clear view of the surface.

[0024] According to another optional embodiment, providing the partial digital images may include capturing the partial digital images on a virtual regular grid of the infrastructure surface. In such an embodiment, the transformation required before stitching the partial digital images together with the overview image may be reduced. Thus, optical misalignments in the final overview image may be significantly reduced or avoided entirely.

[0025] Furthermore, embodiments of the present disclosure may take the form of an associated computer program product accessible from a computer usable or computer readable medium providing program code for use with, by, or in connection with a computer or any instruction execution system. For purposes of this description, a computer usable or computer readable medium may be any apparatus that may include means for storing, communicating, propagating, or transporting a program for use with, by, or in connection with an instruction execution system, apparatus, or device. [Brief explanation of the drawings]

[0026] It should be noted that the disclosed embodiments are described with reference to different subject matters. In particular, some embodiments are described with reference to method-type claims, while other embodiments are described with reference to apparatus-type claims. However, in view of the above and following explanations, those skilled in the art will recognize that, unless otherwise specified, any combination of features belonging to one type of subject matter, as well as any combination between features relating to different subject matters, in particular between features of method-type claims and features of apparatus-type claims, is considered to be disclosed herein.

[0027] The above-defined and further aspects of the present disclosure will be apparent from and will be elucidated with reference to the example embodiments described hereinafter, to which the disclosure is not limited. Preferred embodiments of the disclosure are described, by way of example only, with reference to the following drawings, in which:

[0028] [Figure 1] FIG. 1 illustrates a block diagram of an embodiment of a computer-implemented method for image stitching of multiple digital images of an infrastructure surface for defect detection, according to an embodiment of the present disclosure.

[0029] [Figure 2] 1 shows a schematic diagram of an embodiment of an implementation of the present disclosure, according to an embodiment of the present disclosure.

[0030] [Figure 3] 1 shows a schematic diagram of an overview of an implementation of the present disclosure, according to an embodiment of the present disclosure.

[0031] [Figure 4] 1 shows a schematic diagram of an example embodiment of camera movement and a sequence of images used for stitching, according to an embodiment of the present disclosure.

[0032] [Figure 5A] 10 shows an image after performing a portion of the process of stitching together captured partial digital images according to an embodiment of the present disclosure.

[0033] [Figure 5B] 10 shows an image after performing a portion of the process of stitching together captured partial digital images according to an embodiment of the present disclosure.

[0034] [Figure 5C] 10 shows an image after performing a portion of the process of stitching together captured partial digital images according to an embodiment of the present disclosure.

[0035] [Figure 5D] 10 shows an image after performing a portion of the process of stitching together captured partial digital images according to an embodiment of the present disclosure.

[0036] [Figure 5E] 10 shows an image after performing a portion of the process of stitching together captured partial digital images according to an embodiment of the present disclosure.

[0037] [Figure 5F] 10 shows an image after performing a portion of the process of stitching together captured partial digital images according to an embodiment of the present disclosure.

[0038] [Figure 5G]10 shows an image after performing a portion of the process of stitching together captured partial digital images according to an embodiment of the present disclosure.

[0039] [Figure 6] 10 shows an example of time-dependent captured images being compared in single image mode, according to an embodiment of the present disclosure.

[0040] [Figure 7] 1 illustrates a block diagram of multiple captured images using scene comparison to determine defect evolution, according to an embodiment of the present disclosure.

[0041] [Figure 8] FIG. 1 illustrates a block diagram of an embodiment of an image stitching system for image stitching of multiple digital images of an infrastructure surface for defect detection, according to an embodiment of the present disclosure.

[0042] [Figure 9] 9 shows a schematic diagram of an embodiment of a computing system including the system of FIG. 8, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0043] In the context of the present description, the following conventions, terms or expressions, or combinations thereof, may be used.

[0044] The term "image stitching" may refer to the process of combining digital images of multiple photographs, possibly with overlapping areas, to provide a segmented panoramic or overview image in high resolution format.

[0045] The term "infrastructure surface" may refer to any man-made public infrastructure surface such as buildings, bridges, pipelines, wind turbine masts, cable masts for electrical energy transmission, roads, and similar structures.

[0046] The term "partial digital image" may refer to a digital image, for example, a digital image taken by a digital camera on an unmanned aerial vehicle, that can be spliced ​​together with other images to construct an overview image having the same or similar resolution as the partial digital image.

[0047] The term "Global Positioning System (GPS)" may refer to a global satellite-based radio navigation system based on a Global Navigation Satellite System (GNSS) that provides geographic location and time information to a GPS receiver. The GPS receiver may be part of a camera that captures a partial digital image so that GPS data, such as latitude, longitude, and height data (also referred to as altitude data), can be integrated as metadata into the dataset of the captured digital image.

[0048] The term "processing sequence" may refer to a process that processes partial digital images to determine rules for stitching them together. The stitching process may specifically start with a starting image that may be in the center of the overview image to be generated. Furthermore, images stitched to the starting image follow a rule of increasing distance from the center image. After two or more images have been stitched together, they may be referred to as "intermediate images."

[0049] The term "characterization" may refer to specific markers used to identify features detected in a digital image.

[0050] The term "similarity matrix" may refer to a mathematical matrix in which related features in adjacent digital images may be related to one another so that they overlap in a seamless manner during the stitching process.

[0051] The term "overview image" may refer to multiple captured partial digital images that are stitched together in a seamless manner in accordance with the methods and systems disclosed herein, whereby the overview image may have the same resolution (e.g., pixels per inch) as each of the partial digital images.

[0052] The term "identified defects" may refer to inconsistencies (mechanical or otherwise) in the digitally captured images. Such defects may relate to cracks or other problem areas in public infrastructure components that may sooner or later cause malfunction of the associated infrastructure component. To identify defects, various trained neural network systems may use one or more of the captured partial digital images as input.

[0053] The term "scene registration" may refer to the registration of an overview image generated from multiple captured partial digital images stitched together. The registered scene at a particular time (particularly, a time having timestamp X) may be used as a reference for comparison with another scene at a later time (particularly, a later time having timestamp Y).

[0054] The term "image warping" may generally refer to the process of digitally manipulating a digital image. Shapes and forms depicted in the image may undergo one or more transformations that may result in significant distortion compared to the captured image. The term "image dewarping" may be used herein to refer to the transformation of an image that may not have been captured under ideal conditions, such as when the viewing angle is perpendicular to the imaged surface.

[0055] The term "pixel-based comparison" may refer to a comparison process of digital images that uses techniques based on comparing corresponding pixels in the digital images.

[0056] The term "SIFT" may refer to a known feature detection algorithm for detecting and describing local features and images. The abbreviation SIFT stands for Scale Invariant Feature Transform. This technique may be useful for object recognition and, in particular, image stitching. First, SIFT keypoints of an object may be extracted from a set of reference images and stored in a database. Later, an object may be recognized in a new image by individually comparing each feature from the new image with images in the database and finding candidate matches based on the Euclidean distance of their feature vectors. In one example, a 128 floating-point value vector may be used here.

[0057] The term "transformation matrix" may refer to a matrix in the mathematical sense that is used to transform another matrix into another representation. For example, a transformation matrix may be used to transform a matrix that describes a digital image or a portion thereof into another representation.

[0058] Detailed descriptions of the figures are provided herein. All instructions in the figures are schematic. First, a block diagram of an embodiment of a computer-implemented method for image stitching of multiple digital images of an infrastructure surface for defect detection is provided. Further, additional embodiments and embodiments of an image stitching system for image stitching of multiple digital images of an infrastructure surface for defect detection are described.

[0059] Before proceeding to a detailed description of the present disclosure, the general concepts underlying the disclosure are explained below.

[0060] First, a substantially flat and stationary surface, the surface of interest, is assumed to be captured as multiple digital images. The entire complex structure is decomposed into multiple simple scenes. For example, each face of a rectangular bridge pillar or pylon is treated separately. Primitive elements such as building facades, ridges, and road surfaces are simple examples of faces. Each face generally defines a plane of interest within which the face is considered to reside. During the scanning process, a flying object, e.g., a drone, moves along a parallel plane so that the camera direction remains perpendicular to the plane of interest.

[0061] Mission planning is performed such that the drone's path is controlled along a regular grid pattern to scan the surface of interest. Images are captured at regular intervals and automatically associated with GPS metadata that is enhanced using real-time kinematic techniques (ITK) relying on reference base station signals. The raw images with GPS annotations are the only input data considered for subsequent processing.

[0062] The image sensor resolution and the distance of the drone from the surface of interest determine the ground sampling distance. Mission planning ensures a minimum target ground sampling distance. Because the drone moves during image capture, flight speed may be limited to limit the maximum image sensor exposure to minimize motion blur seen in the collected data. For example, flight speed may be limited to 0.4 meters per second, or a similar rate.

[0063] For feature matches, the intermediate spliced ​​image S i and the corresponding current image I i At iteration i, feature matching is performed between S and S. The features of the considered image pair may be matched, for example, by the following procedure: According to at least some embodiments of the present disclosure, a feature matching algorithm S based on the Euclidean distance between the features is used. i A k-NN matching algorithm with k=2 may be used, corresponding each feature to the top two features from

[0000] . Additionally, Lowe's ratio test may be used to remove false positive matches.

[0064] In further actions, I use the matches from the past actions. i S i In this method, an estimation of an affine transformation matrix for transforming a pixel into an accordion in a system is performed. According to at least one embodiment of the present disclosure, a RANSAC (Random Sample Consensus) algorithm may be used to perform this operation. RANSAC is a robust estimation procedure that uses a minimal set of randomly sampled correspondences to estimate image transformation parameters and find a solution that has the best consensus with the data. It removes outliers and returns the best transformation matrix. At least three non-collinear points may need to be matched to be able to estimate an accurate transformation matrix.

[0065] A general perspective transformation matrix is ​​a 3x3 matrix, while an affine transformation matrix is ​​a special case of perspective where the third row is [0,0,1]. An affine transformation matrix is

number

[0066] Assuming an ideal setup, we can only predict the translation between images, as explained above. Henceforth, the ideal transformation matrix is ​​structured as follows:

number

[0067] However, rotation, shear, and dilation up to a certain threshold are also possible to account for distortions due to drone and camera position. Transformations beyond the threshold are marked as outliers and excluded from further processing to avoid error propagation.

[0068] This technique is effective in eliminating some typical errors in a single image, such as keypoints that are out of focus, have motion blur, or result in incorrect matches.

[0069] The figures are now described below in the context explained above.

[0070] FIG. 1 shows a block diagram of a preferred embodiment of a computer-implemented method 100 for image stitching of multiple digital images of an infrastructure surface for defect detection. The multiple digital images may be derived from high-resolution regular scans of the infrastructure surface for defect detection. The infrastructure may be, for example, a building, a bridge, or another public infrastructure. The multiple digital images may also be derived from scans for location identification and evolution-based severity assessment. In operation 102 of method 100, multiple partial digital images of the infrastructure surface are provided. According to at least some embodiments of the present disclosure, the partial digital images also include GPS metadata.

[0071] At operation 104, method 100 further comprises extracting global positioning system metadata from the partial digital image data. According to at least some embodiments of the present disclosure, the metadata is extracted from a GPS or GNSS system such as Galileo, Navstar, GLONASS, or Beidou, for example.

[0072] In operation 106, the method 100 further comprises scheduling a processing sequence for the image based on the extracted global positioning system metadata.

[0073] At operation 108, method 100 further comprises determining a feature representation in one or more partial digital images. According to at least some embodiments of the present disclosure, the feature representation may be determined, for example, by using 128-dimensional floating-point vectors from the SIFT method or other methods.

[0074] At operation 110, the method 100 further comprises executing the scheduled processing sequence. By executing the scheduled processing sequence, a similarity matrix using characterizations of adjacent partial digital images can be determined to progressively locate each of the partial digital images. In this manner, a synoptic image of the infrastructure surface can be progressively generated by digitally stitching together multiple partial digital images.

[0075] FIG. 2 shows a schematic diagram of an exemplary embodiment 200 of an implementation of the present disclosure. In the exemplary embodiment 200, the infrastructure object of interest may be a bridge 202, a building 204, a ground electric cable mast, a wind turbine mast, or another similar structure. According to the exemplary embodiment 200, a camera installed on an airborne vehicle may capture an image 206 having a given resolution. In this captured image 206, a particular region or sub-image 208 may be of greater interest due to possible damage, such as small cracks, surface scratches, or similar defects. However, the resolution of the captured image may not be high enough to allow the region of interest to be identified. Therefore, a region of the image 206 is scanned to generate multiple partial digital images (not shown here).

[0076] In some implementations of the present disclosure, such as this example, it may be necessary to span up to four orders of magnitude of resolution to accurately detect and locate defects. For example, a region of interest in bridge 202 may be approximately 10 meters wide. This region may be represented by overview image 206 taken by a camera. However, a defect crack in that region of interest may have a width less than approximately 1 millimeter. Thus, overview image 206 typically has too low a resolution to accommodate both the width of the region of interest (approximately 10 meters) and the width of the crack (approximately less than 1 millimeter).

[0077] FIG. 3 shows a schematic diagram of an example embodiment of an overview 300 of an implementation of the present disclosure. A camera 302 is mounted on an airborne object 304, and the camera 302 may capture a plurality of partial digital images 306. According to at least one embodiment of the present disclosure, the airborne object 304 is a drone. Each digital image in the plurality of partial digital images 306 includes metadata, such as GPS data. The GPS data includes coordinates such as latitude, longitude, and height (which may also be referred to as altitude). According to at least some embodiments of the present disclosure, the metadata for each digital image in the plurality of digital images 306 may also include an associated timestamp. Using current technology, GPS data may have an accuracy of ±5 centimeters. However, this is not accurate enough to enable high-quality stitching of the partial digital images 306 to detect infrastructure defects.

[0078] Based on the captured images 306 and the GPS data, a stitching order 308 is determined. The captured images 306 and the stitching order 308 are then used as input to a system 310 (which implements one or more algorithms), including SIFT interest point detection and delineation 312, interest point matching 314, transformation matrix estimation 316, and defect localization or measurement results 318, or a combination thereof. The system 310 outputs a rendered fully stitched image 320, which includes the defect localization or measurement data, or a combination thereof. The fully stitched image 320 may also be referred to as a generated overview image.

[0079] 4 shows a schematic diagram 400 of an example embodiment of camera movement and the sequence of images used for stitching. In diagram 400, cube 402 may represent an example object of interest, such as a bridge pylon. Camera 302 of flying object 304 may fly along path 404 in front of a surface of interest 406 on the pylon. Assuming flying object 304 and camera 302 operate in a typical manner, camera 302 captures images from within a relatively short distance of the pylon. In this example embodiment, by capturing these images, camera 302 provides multiple partial digital images, which are stored in memory system 408.

[0080] Based on the GPS data associated with each of the captured images, a chart 412 may be formed in which each image is represented by a dot. According to at least some embodiments of the present disclosure, the chart 412 is oriented such that relative longitude positions are on the x-axis and relative latitude positions are on the y-axis, and the dots are organized on the chart 412 in a manner that represents their positions relative to one another. In the context of the chart 412, the partial digital images are sorted according to increasing geographic distance from a central image 410 that is determined to be at or near the center or center of the overview image. Geographic distance may also be referred to as physical distance or geolocation distance.

[0081] According to at least some embodiments of the present disclosure, dots representing images may be connected by lines that may indicate the sequence in which the images were captured. According to at least some embodiments of the present disclosure, dots representing images may be labeled with numbers adjacent to the dots. In such embodiments, the numbers may represent a determined or scheduled sequence of the stitching process. As can be seen by FIG. 4, according to at least some embodiments of the present disclosure, the sequence in which images were captured and the sequence in which the images are stitched together to generate the overview image are typically not the same.

[0082] 5A-5G show a series of results of performing various operations of the process described herein for stitching together captured partial digital images. In FIG. 5A, each of the dots 502 represents a corresponding reference point (e.g., center, top left corner, center of top border, or the like) of the captured partial digital image 504 based on GPS metadata. However, in FIG. 5A, the image 504 is only roughly positioned because the GPS metadata is not accurate enough for proper matching and stitching.

[0083] As already mentioned in the context of FIG. 4, the stitching process starts in the center. This is shown in FIG. 5B, where center image 504a is highlighted. As shown in FIG. 5C, a second partial image 504b is stitched to the left of center image 504a, the first image mentioned in FIG. 5B. Additional images 504c and 504d are added, as shown in FIGS. 5D and 5E. At each iteration stage, the multiple partial digital images stitched together may be referred to as a "so far" intermediate stitched image or an "in progress" intermediate stitched image until all of the available partial digital images have been stitched together. Such intermediate stitched images are referred to herein as "S" intermediate stitched images. i " can be referenced at

[0084] Figure 5F shows the result of the iterative progression of the stitching process after more partial digital images have been stitched into intermediate stitched image 506. Figure 5G shows the result of all available partial digital images stitched together to generate overview image 508 as a result of the processing described herein.

[0085] FIG. 6 shows a flow 600 of captured images associated with data representing the time they were captured and the three-dimensional spatial conditions in which they were captured. In this example, image 602 was captured with timestamp X and shows a defect 604 on a surface and another artifact 606 on the surface. Image 608 was captured with timestamp Y, which is later than timestamp X. The recognized defect 604 has changed in image 608 compared to image 602. The viewing angles of the images are also different. More specifically, the viewing angle of image 608 is rotated clockwise relative to the viewing angle of image 602. Additionally, the right edge of defect 604 is larger in image 608 than in image 602. In contrast, artifact 606 appears unchanged in image 608 relative to image 602.

[0086] For the purposes of this example, the elapsed time between timestamp X and timestamp Y is relatively large. To provide a reliable assessment of infrastructure components, it is important to be able to observe defects or potential defects over an extended period of time. Assessing the evolution of a defect allows for a better assessment of its severity. However, comparing cracks over an extended period of time is difficult because it requires snapshots of images of the same defect for multiple points in time, requires precise positioning, the same defect needs to be compared, and requires precise measurements of changes in the crack. The simple solution of using a stationary camera for precise observation is often not feasible because the area to be scanned is too large or the image material comes from a drone, which can be difficult to assess.

[0087] Image 610 shows the result of images 602 and 608 being overlaid on top of each other. Image 610 therefore allows for direct comparison, as well as image registration, warping (or dewarping), and final direct comparison. However, image registration may require highly visible reference points that may be difficult to access in the absence of any markers. For example, because concrete is a self-similar material, it is difficult to locate reliable match points. Therefore, achieving the desired result shown in image 612, which shows an exact match, may be difficult.

[0088] Therefore, instead of using a single image comparison for the method disclosed herein, the scene comparison uses the image stitching procedure disclosed herein. To this end, FIG. 7 shows a related example diagram 700.

[0089] In diagram 700, multiple images 702 of a scene are captured at a first time having timestamp X, and multiple images 704 of the same scene are captured at a second time having timestamp Y. As described above, timestamp Y occurs after timestamp X. In operation 706, each of the set of images of the same scene are stitched together using the methods described above, resulting in overview images 708 and 710, respectively.

[0090] Diagram 700 further includes a partial flowchart 716 illustrating two operations performed using overview images 708 and 710. First, in operation 712, scene registration is performed. Next, in operation 714, image registration, refinement, and comparison are performed. The technique illustrated in FIG. 7 provides a high-quality assessment of crack evolution because scenes are likely to contain highly visible reference points (such as boundaries or lines), and registration accuracy depends on the accuracy of stitching together images of each scene. Therefore, automatic detection of inherent growing defects versus new defects is facilitated.

[0091] FIG. 8 shows a schematic diagram of an example embodiment of an image stitching system 800 for image stitching of multiple digital images of an infrastructure surface for defect detection. The system 800 includes a memory 802 for storing program code portions. The memory 802 is coupled to a processor 804. When executing the program code portions, the processor 804 is capable of receiving multiple partial digital images of the infrastructure surface, extracting global positioning system metadata from data of the partial digital images, and scheduling a processing sequence for the images based on the extracted global positioning system metadata. In accordance with at least some embodiments of the present disclosure, the processor 804 may receive the multiple partial digital images of the infrastructure surface using a receiver 806. In accordance with at least some embodiments of the present disclosure, the processor 804 may extract global positioning system metadata from the partial digital images using a GPS extraction unit 808. In accordance with at least some embodiments of the present disclosure, the processor 804 may schedule a processing sequence for the images using a scheduler 810.

[0092] When processor 804 executes the program code, it also determines feature descriptions in one or more partial digital images and performs a scheduled processing sequence, thereby enabling it to determine a similarity matrix using feature descriptions in adjacent partial digital images and to progressively position each of the partial digital images. In this manner, a schematic image of the infrastructure surface is progressively generated by digitally stitching multiple partial digital images together. According to at least some embodiments of the present disclosure, processor 804 may use determination module 812 to determine feature descriptions in one or more partial digital images. According to at least some embodiments of the present disclosure, processor 804 may use extraction module 814 to perform the scheduled processing sequence.

[0093] It should be understood that all functional units, modules, and functional blocks, such as the receiver 806, the GPS extraction unit 808, the scheduler 810, the determination module 812, and the extraction module 814, may be implemented as hardware units, software modules, or a combination thereof, and they may be communicatively coupled to each other for signal or message exchange in a selected 1:1 manner. Alternatively, the functional units, modules, and functional blocks may be coupled to a system internal bus system 816 for selective signal or message exchange.

[0094] The disclosed embodiments may be implemented with virtually any type of computer suitable for storing and / or executing program code, regardless of platform. For example, Figure 9 illustrates an exemplary embodiment comprising a computing system 900 suitable for executing program code associated with the methods disclosed herein.

[0095] Computing system 900 is merely one example of a suitable computer system and is not intended to suggest any limitation on the scope of use or functionality of the embodiments described herein. The exemplary computer system 900 is capable of implementing and / or performing any of the functions described above. In accordance with at least some embodiments of the present disclosure, computer system 900 may be a server. There are components in computer system 900 that operate with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing system environments or configurations or combinations thereof that may be suitable for use with computer system 900 include, but are not limited to, personal computer systems, server computer systems, thin client, thick client, handheld, or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments, which may include any of the systems or devices described above, and the like.

[0096] The computer system / server 900 may be described in the general context of computer system-executable instructions, such as program modules, executed by the computer system 900. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 900 may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0097] 9, computer system / server 900 is a general-purpose computing device. Components of computer system / server 900 may include, but are not limited to, one or more processors or processing units 902, a system memory 904, and a bus 906 that couples various system components, including the system memory 904, to the processor 902. Bus 906 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0098] Computer system / server 900 typically includes a variety of computer system-readable media. Such media can be any available media that can be accessed by computer system / server 900 and includes both volatile and nonvolatile media, removable and non-removable media.

[0099] The system memory 904 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 908 or cache memory 910, or a combination thereof. The computer system / server 900 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 912 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive (e.g., a floppy disk) for reading from and writing to a removable, non-volatile magnetic disk and an optical disk drive for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media, may be provided. In such cases, each may be connected to the bus 906 by one or more data medium interfaces. As further shown and described below, the system memory 904 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the disclosed embodiments.

[0100] As a non-limiting example, a set of program modules 916 having programs / utilities, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 904. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. The program modules 916 generally perform the functions or methods, or combinations thereof, of the disclosed embodiments as described herein.

[0101] The computer system / server 900 may also communicate with one or more external devices 918, such as a keyboard, pointing device, display 920, one or more devices that allow a user to interact with the computer system / server 900, or any device (e.g., a network card, modem, etc.) that allows the computer system / server 900 to communicate with one or more other computing devices. Such communication may occur via an input / output (I / O) interface 914. The computer system / server 900 may also communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), via a network adapter 922. As shown, the network adapter 922 may communicate with other components of the computer system / server 900 via a bus 906. Although not shown, it should be understood that other hardware or software components, or combinations thereof, may be used in cooperation with the computer system / server 900. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0102] Additionally, an image stitching system 800 for image stitching of multiple digital images of infrastructure surfaces for defect detection may be added to the bus system 906 .

[0103] The description of various embodiments of the present disclosure has been provided for illustrative purposes and is not intended to be exhaustive or limiting of the disclosed embodiments. It will be apparent to those skilled in the art that many modifications and variations can be made without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles, practical applications, or technical improvements of the embodiments relative to technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0104] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions that cause a processor to perform aspects of the present invention.

[0105] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanical coding devices such as punch cards or raised structures in grooves with instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0106] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0107] Computer-readable program instructions for carrying out operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, or the like, and procedural programming languages ​​such as the “C” programming language or similar. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0108] Aspects of the present invention will now be described with reference to flowchart and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention, it being understood that each block of the flowchart and / or block diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus that causes a machine, and the instructions execute via the computer's processor or other programmable data processing apparatus to cause means for implementing the functions / acts specified in a block or blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, and a computer-readable storage medium having instructions stored thereon includes an article of manufacture containing instructions that implement aspects of the functions / acts specified in a block or blocks of the flowcharts and / or block diagrams.

[0110] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device so that a series of operational steps are executed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, and the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in a block or blocks of the flowcharts and / or block diagrams.

[0111] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be accomplished as a single step, or may be executed simultaneously, substantially simultaneously, partially, or fully in a time-overlapping manner, or the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a special-purpose hardware-based system that performs the specified functions or actions or executes a combination of special-purpose hardware and computer instructions.

[0112] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used herein, the singular forms a, an, and the are intended to include the plural forms unless the context clearly dictates otherwise. It is further understood that the terms "comprises" and / or "comprises," when used herein, specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof.

[0113] Corresponding structure, materials, acts, and equivalents of all means or step and functional elements in the claims below are intended to include any structure, material, or acts for performing the function as specifically claimed or in combination with other claimed elements. The description of the present disclosure has been presented for purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the disclosure. The embodiments are chosen and described to best explain the principles and practical applications of the disclosure and to enable those skilled in the art to understand the disclosure for various embodiments with various modifications suitable for the particular uses contemplated.

[0114] To summarize, several aspects of the present disclosure can be illustrated using the following non-limiting list of exemplary embodiments:

[0115] Exemplary embodiment 1 is a computer-implemented method for image stitching of multiple digital images of an infrastructure surface for defect detection. The method comprises providing multiple partial digital images of the infrastructure surface. The method further comprises extracting global positioning system metadata from data of the partial digital images. The method further comprises scheduling a processing sequence for the partial digital images based on the extracted global positioning system metadata. The method further comprises determining characterizations in one or more partial digital images. The method further comprises executing the scheduled processing sequence, thereby determining a similarity matrix using the characterizations of adjacent partial digital images and progressively positioning each of the partial digital images, thereby progressively generating an overview image of the infrastructure surface by digitally stitching together the multiple partial digital images.

[0116] Exemplary embodiment 2 includes the method of embodiment 1, including or excluding optional features. In this exemplary embodiment, the method also includes the steps of rendering an overview image and mapping identified defects, annotations, or measurement results onto the overview image.

[0117] Exemplary embodiment 3 includes the method of any one of exemplary embodiments 1 or 2, including or excluding optional features. In this exemplary embodiment, the method also includes performing, on a plurality of partial digital images of the infrastructure surface having different timestamps, at least one activity selected from the group including scene registration for time evolution assessment, image warping, and pixel usage comparison.

[0118] Exemplary embodiment 4 includes the method of any one of exemplary embodiments 1 to 3, including or excluding optional features. In this exemplary embodiment, the method also includes generating a plurality of partial digital images using a camera of the unmanned vehicle.

[0119] Exemplary embodiment 5 includes the method of any one of exemplary embodiments 1 to 4, including or excluding optional features. In this exemplary embodiment, determining the characterization uses the SIFT method.

[0120] Exemplary embodiment 6 includes the method of any one of exemplary embodiments 1 to 5, including or excluding optional features. In this exemplary embodiment, determining the characterization includes detecting points of interest.

[0121] Exemplary embodiment 7 includes the method of any one of exemplary embodiments 1 to 6, including or excluding optional features. In this exemplary embodiment, determining the similarity matrix includes using a RANSAC method to progressively position each of the partial digital images.

[0122] Exemplary embodiment 8 includes the method of exemplary embodiment 7, including or excluding optional features. In this exemplary embodiment, the progressively positioning each of the partial digital images includes: i The existing intermediate spliced ​​image S i To splice into the transformation matrix H i This includes determining:

[0123] Exemplary embodiment 9 includes the method of any one of exemplary embodiments 1 to 8, including or excluding optional features. In this exemplary embodiment, scheduling the processing sequence for the images includes sorting the partial digital images by ascending distance from a central image of the infrastructure surface, and commencing incremental positioning of each of the partial digital images from the central image.

[0124] Exemplary embodiment 10 includes the method of any one of exemplary embodiments 1-9, including or excluding optional features. In this exemplary embodiment, providing a plurality of partial digital images includes capturing the partial digital images parallel to the infrastructure surface.

[0125] Exemplary embodiment 11 includes the method of any one of exemplary embodiments 1 to 10, including or excluding optional features. In this exemplary embodiment, providing the plurality of partial digital images includes capturing the partial digital images such that the image plane is not parallel to the imaged surface, and digitally deskewing the partial digital images to align the partial digital images parallel to the imaged surface.

[0126] Exemplary embodiment 12 includes the method of any one of exemplary embodiments 1 to 11, including or excluding optional features. In this exemplary embodiment, providing the plurality of partial digital images includes capturing the partial digital images on a regular grid of the infrastructure surface.

[0127] Exemplary embodiment 13 is an image stitching system for image stitching of multiple digital images of an infrastructure surface for defect detection. The system includes a memory for storing program code portions. The memory is coupled to a processor, which, when executing the program code portions, is capable of receiving multiple partial digital images of the infrastructure surface. When executing the program code portions, the processor is further capable of extracting global positioning system metadata from data of the partial digital images. When executing the program code portions, the processor is further capable of scheduling a processing sequence for the images based on the extracted global positioning system metadata. When executing the program code portions, the processor is further capable of determining characterizations in one or more partial digital images. When executing the program code portions, the processor is further capable of executing the scheduled processing sequence, thereby determining a similarity matrix using the characterizations of adjacent partial digital images and progressively positioning each of the partial digital images, thereby progressively generating an overview image of the infrastructure surface by digitally stitching together the multiple partial digital images.

[0128] Exemplary embodiment 14 includes the system of exemplary embodiment 13, including or excluding optional features. In this exemplary embodiment, the processor, when executing the program code portions, is also capable of rendering an overview image and mapping identified defects, annotations, or measurements onto the overview image.

[0129] Exemplary embodiment 15 includes the system of any one of exemplary embodiments 13 or 14, including or excluding optional features. In this exemplary embodiment, the processor, when executing the program code portions, is also capable of performing at least one activity selected from the group including scene registration, image warping, and pixel usage comparison for time evolution assessment on multiple partial digital images of the infrastructure surface having different timestamps.

[0130] Exemplary embodiment 16 includes the system of any one of exemplary embodiments 13 to 15, including or excluding optional features. In this exemplary embodiment, the plurality of partial digital images are captured by a camera of an unmanned vehicle.

[0131] Exemplary embodiment 17 includes the system of any one of exemplary embodiments 13-15, including or excluding optional features. In this exemplary embodiment, the processor is also enabled to detect points of interest when executing the program code portions and when performing the characterization determination.

[0132] Exemplary embodiment 18 includes the system of any one of exemplary embodiments 13 to 17, including or excluding optional features. In this exemplary embodiment, the processor, when executing the program code portion, is also enabled to perform a RANSAC method to progressively locate each of the partial digital images when determining the similarity matrix.

[0133] Exemplary embodiment 19 includes the system of exemplary embodiment 13, including or excluding optional features. In this exemplary embodiment, the processor, when executing the program code portions, is also capable of capturing the partial digital image such that the image plane is not parallel to the imaged surface when providing the partial digital image, and digitally deskewing the partial digital image to align the partial digital image parallel to the imaged surface.

[0134] Exemplary embodiment 20 includes a computer program product for image stitching of multiple digital images of an infrastructure surface for defect detection. The computer program product includes a computer-readable storage medium having program instructions embodied thereon. The program instructions are executable by one or more computing systems or controllers to cause the one or more computing systems to receive multiple partial digital images of the infrastructure surface. The program instructions are executable to cause the one or more computing systems to extract global positioning system metadata from data of the partial digital images. The program instructions are executable to cause the one or more computing systems to schedule a processing sequence for the images based on the extracted global positioning system metadata. The program instructions are executable to cause the one or more computing systems to determine characterizations in one or more partial digital images. The program instructions are executable to cause the one or more computing systems to execute the scheduled processing sequence, thereby determining a similarity matrix using the characterizations of adjacent partial digital images and progressively positioning each of the partial digital images, thereby progressively generating a synoptic image of the infrastructure surface by digitally stitching together the multiple partial digital images.

Claims

1. 1. A computer-implemented method for image stitching for a plurality of digital images of an infrastructure surface for defect detection, comprising: providing a plurality of partial digital images of an infrastructure surface; extracting global positioning system metadata from data corresponding to the partial digital image; scheduling a processing sequence for said images, which includes sorting said partial digital images in ascending order of distance from a central image of said infrastructure surface; determining feature descriptions of features in the plurality of partial digital images; performing the scheduled processing sequence on the partial digital images based on the extracted global positioning system metadata, the processing sequence including determining a similarity matrix using the feature descriptions of adjacent partial digital images and progressively positioning each of the partial digital images such that a synoptic image of the infrastructure surface is generated by iteratively digitally stitching the partial digital images together; Equipped with each of the partial digital images is positioned based on the central image; How to do it.

2. The plurality of partial digital images are captured in a capture sequence; The computer-implemented method comprises: generating a chart of the plurality of partial digital images, wherein each partial digital image is represented by a dot on the chart, a plurality of the dots being organized in a manner that represents a relative position for the plurality of partial digital images; sorting said plurality of partial digital images according to ascending geographic distance from a central image determined to be at the center of the overview image; labeling a plurality of the dots on the chart with numbers indicating a sequence of the stitching process based on the sorting of the stitching of the plurality of partial digital images; Further provided with Executing the scheduled processing sequence includes: defining a scheduled processing sequence for processing the plurality of partial digital images based on the chart and the extracted global positioning system metadata; performing the scheduled processing sequence on the partial digital image to determine a similarity matrix using the feature descriptions of adjacent partial digital images; progressively positioning each of the partial digital images based on the similarity matrix includes determining a transformation to apply to each partial digital image, the determining a transformation including discarding the transformation if the transformation exceeds a threshold; The computer-implemented method comprises: iteratively digitally stitching the positioned partial digital images together to generate a synoptic image of the infrastructure surface, the positioned partial digital images being stitched together in a stitching sequence, the stitching sequence being different from the capture sequence; Further comprising: The method of claim 1.

3. rendering the overview image; mapping at least one of the identified defects, annotations, or measurements onto the overview image; The method of claim 1 or 2, further comprising:

4. 4. The method of claim 1, further comprising performing at least one of scene registration, image warping, or pixel usage comparison for time evolution assessment on a plurality of partial digital images of the infrastructure surface having different time stamps.

5. The method of claim 1 , wherein providing the plurality of partial digital images comprises using a camera on an unmanned vehicle.

6. The method of claim 1 , wherein determining the characterization comprises using the SIFT method.

7. The method of claim 1 , wherein determining the characterization comprises detecting points of interest.

8. 8. The method of claim 1, wherein determining the similarity matrix further comprises progressively positioning each of the partial digital images using a RANSAC method.

9. The step of progressively positioning each of the partial digital images includes a transformation matrix H ｉ and calculate the current partial digital image I ｉ the existing intermediate stitched image S ｉ 9. The method of claim 8, including the step of splicing.

10. The method of claim 1 , wherein providing the partial digital image comprises capturing the partial digital image at an angle approximately parallel to the infrastructure surface.

11. The step of providing a partial digital image comprises: capturing the partial digital image such that the image plane is not parallel to the imaged surface; digitally deskewing the partial digital image to align the partial digital image parallel to the imaged surface; 11. The method of claim 1, comprising:

12. 12. The method of claim 1, wherein providing the partial digital images comprises capturing the partial digital images on a regular grid of the infrastructure surface.

13. 1. An image stitching system for image stitching of a plurality of digital images of an infrastructure surface for defect detection, comprising: a memory for storing program code portions, said memory coupled to a processor, said processor, when executing said program code portions, receiving a plurality of partial digital images of an infrastructure surface; extracting global positioning system metadata from data corresponding to the partial digital image; scheduling a processing sequence for the images, the processing sequence including sorting the partial digital images in ascending order of distance from a central image of the infrastructure surface; determining feature descriptions of features in the plurality of partial digital images; performing the scheduled processing sequence on the plurality of partial digital images based on the extracted global positioning system metadata, including determining a similarity matrix using the feature descriptions of adjacent partial digital images and progressively positioning each of the partial digital images such that a synoptic image of the infrastructure surface is generated by iteratively digitally stitching the plurality of partial digital images together; It becomes possible, Each of the partial digital images is positioned based on the central image.

14. The plurality of partial digital images are captured in a capture sequence; The processor also, when executing the program code portions: generating a chart of the plurality of partial digital images, wherein each partial digital image is represented by a dot on the chart, a plurality of the dots being organized in a manner that represents a relative position for the plurality of partial digital images; sorting said plurality of partial digital images according to ascending geographic distance from a central image determined to be at the center of the overview image; labeling the dots on the chart with numbers indicating a sequence of stitching based on the sorting of the stitching of the plurality of partial digital images; Executing the scheduled processing sequence includes: defining a scheduled processing sequence for processing the plurality of partial digital images based on the chart and the extracted global positioning system metadata; performing the scheduled processing sequence for the plurality of partial digital images to determine a similarity matrix using the feature descriptions of adjacent partial digital images; progressively positioning each of the partial digital images based on the similarity matrix includes determining a transformation to apply to each partial digital image, wherein determining the transformation includes discarding the transformation if the transformation exceeds a threshold; The processor also, when executing the program code portions: iteratively digitally stitching the positioned partial digital images together to generate an overview image of the infrastructure surface, the positioned partial digital images being stitched together in a stitching sequence, wherein the stitching sequence is different from the capture sequence; and The system of claim 13, wherein:

15. 15. The system of claim 13 or 14, wherein the processor, when executing the program code portions, is also capable of rendering the overview image and mapping identified defects, annotations or measurements onto the overview image.

16. 16. The system of claim 13, wherein the processor, when executing the program code portions, is also capable of performing at least one of scene registration, image warping, and pixel usage comparison for time evolution assessment on a plurality of partial digital images of the infrastructure surface having different timestamps.

17. 17. The system of claim 13, wherein the plurality of partial digital images are taken by a camera on an unmanned vehicle.

18. 18. The system of any one of claims 13 to 17, wherein the processor, when executing the program code portions, is also capable of detecting points of interest when determining characterizations.

19. 19. The system of claim 13, wherein the processor, when executing the program code portions, is also capable of performing a RANSAC method for progressively positioning each of the partial digital images when determining the similarity matrix.

20. 20. The system of claim 13, wherein the processor, when executing the program code portions, is also capable of, when providing the partial digital images, capturing each partial digital image such that the image plane is not parallel to the imaged surface, and digitally deskewing each partial digital image to align the partial digital image parallel to the imaged surface.

21. 1. A computer program for image stitching of a plurality of digital images of an infrastructure surface for defect detection, the computer program comprising program instructions executable by one or more computing systems or controllers, the one or more computing systems comprising: receiving a plurality of partial digital images of an infrastructure surface; extracting global positioning system metadata from data corresponding to the partial digital image; scheduling a processing sequence for the images, the processing sequence including sorting the partial digital images in ascending order of distance from a central image of the infrastructure surface; determining feature descriptions of features in the plurality of partial digital images; performing the scheduled processing sequence on the partial digital images based on the extracted global positioning system metadata, including determining a similarity matrix using the feature descriptions of adjacent partial digital images and progressively positioning each of the partial digital images such that a synoptic image of the infrastructure surface is generated by iteratively digitally stitching the plurality of partial digital images together; Let them do this, each of the partial digital images is positioned based on the central image; Computer program.

22. The plurality of partial digital images are captured in a capture sequence; The program instructions are executable by one or more computing systems or controllers, and the one or more computing systems include: generating a chart of the plurality of partial digital images, wherein each partial digital image is represented by a dot on the chart, a plurality of the dots being organized in a manner that represents a relative position for the plurality of partial digital images; sorting said plurality of partial digital images according to ascending geographic distance from a central image determined to be at the center of the overview image; labeling a plurality of the dots on the chart with numbers indicating a sequence of the stitching process based on the sorting of the stitching of the plurality of partial digital images; Executing the scheduled processing sequence includes: defining a scheduled processing sequence for processing the plurality of partial digital images based on the chart and the extracted global positioning system metadata; performing the scheduled processing sequence on the partial digital image to determine a similarity matrix using the feature descriptions of adjacent partial digital images; progressively positioning each of the partial digital images based on the similarity matrix includes determining a transformation to apply to each partial digital image, wherein determining the transformation includes discarding the transformation if the transformation exceeds a threshold; The program instructions are executable by one or more computing systems or controllers, and the one or more computing systems include: iteratively digitally stitching the positioned partial digital images together to generate an overview image of the infrastructure surface, the positioned partial digital images being stitched together in a stitching sequence, the stitching sequence being different from the capture sequence; Further carry out 22. A computer program according to claim 21.

Images