Crawling method for creating and automatically annotating custom datasets

Abstract

The invention relates to a crawling method for creating and automatically annotating a custom data set, and provides a method for creating a custom data set. The method includes capturing, with a camera, a sequence of images of a surface in a specified crawling direction by a plurality of successive crawling steps representing a change in camera position. For each crawling step, the method matches a first image and a second image captured at successive camera positions. The method models each crawling step as paired secondary transformations between the first image and the second image according to a plurality of respective anchor points in each image selected as optimal candidates for fitting the regression model.

Description

Technical Field

[0001] The present disclosure relates generally to visual inspection and, more particularly, to creating custom data sets for computer vision-based structural health monitoring. Background Art

[0002] Building computer vision-based models for structural health monitoring requires large datasets that capture a wide range of conditions on surfaces. Existing datasets are usually generated in controlled laboratory settings with nearly ideal environmental conditions. Therefore, they are not suitable for real-life industrial applications. Creating custom datasets traditionally requires manual annotation. Summary of the invention

[0003] The illustrative embodiments provide a computer-implemented method for creating a custom dataset. The method includes capturing a sequence of images of a surface with a camera in a specified crawling direction through a plurality of consecutive crawling steps representing changes in camera position. For each crawling step, the method matches a first image and a second image captured at consecutive camera positions. The method models each crawling step as a pairwise quadratic transformation between the first image and the second image according to a plurality of corresponding anchor points in each image that are selected as best candidates for fitting a regression model.

[0004] Another illustrative embodiment provides a system for creating a custom data set. The system includes a storage device storing program instructions and one or more processors operably connected to the storage device and configured to execute the program instructions so that the system: captures a sequence of images of a surface in a specified crawling direction with a camera through a plurality of consecutive crawling steps representing changes in camera position; for each crawling step, matches a first image and a second image captured at consecutive camera positions; and models each crawling step as a pairwise quadratic transformation between the first image and the second image according to a plurality of corresponding anchor points in each image selected as the best candidate for fitting a regression model.

[0005] Another illustrative embodiment provides a computer program product for creating a custom dataset. The computer program product includes a non-volatile computer-readable storage medium having program instructions stored thereon to perform the following operations: capture a sequence of images of a surface at a specified crawling direction with a camera through a plurality of consecutive crawling steps representing changes in camera position; for each crawling step, match a first image and a second image captured at consecutive camera positions; and model each crawling step as a pairwise quadratic transformation between the first image and the second image according to a plurality of corresponding anchor points in each image selected as best candidates for fitting a regression model.

[0006] The features and functions can be achieved independently in various embodiments of the present disclosure or may be combined in yet other embodiments in which further details can be seen with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The novel features which are believed to be characteristic of the illustrative embodiments are set forth in the appended claims. However, the illustrative embodiments together with the preferred mode of use, further objects and features thereof will be best understood by reference to the following detailed description of illustrative embodiments of the present disclosure when read in conjunction with the accompanying drawings, in which:

[0008] Figure 1 is an illustration of a block diagram of a visual inspection crawling system according to an illustrative embodiment;

[0009] Figure 2 depicts a computer vision imaging setup according to an illustrative embodiment;

[0010] Figure 3 depicts computer vision imaging under different lighting conditions according to an illustrative embodiment;

[0011] Figure 4 shows pairwise matching of consecutive images in accordance with an illustrative embodiment;

[0012] Figure 5 shows determination of anchor points in an image according to an illustrative embodiment;

[0013] Figure 6 shows a filter for an anchor point in accordance with an illustrative embodiment;

[0014] Figure 7 shows a bounding box of an object in an image according to an illustrative embodiment;

[0015] Figure 8 shows an example of accurate bounding box tracking across a sequence of images in accordance with an illustrative embodiment;

[0016] Fig. 9 depicts a flow chart of a process for creating a custom data set in accordance with an illustrative embodiment;

[0017] Fig.10 depicts a flow chart of a process for matching a first image and a second image for each crawling step in accordance with an illustrative embodiment;

[0018] Fig.11 depicts a flowchart of a process for propagating a bounding box in accordance with an illustrative embodiment; and

[0019] Fig.12 is an illustration of a block diagram of a data processing system in accordance with an illustrative embodiment. DETAILED DESCRIPTION

[0020] The illustrative embodiments recognize and take into account that existing datasets used for computer vision-based modeling are typically generated in controlled laboratory settings with nearly ideal environmental conditions that are not suitable for real-life industrial applications.

[0021] The illustrative embodiments recognize and take into account that creating custom datasets is time consuming because annotation is often performed manually, relying on human annotators to mark objects in images. If objects are captured under various lighting conditions, all corresponding images also need to be manually annotated. In addition to the time and cost associated with these manual methods, proprietary data may need to be shared with third-party vendors.

[0022] The illustrative embodiments provide a method for collecting computer vision data in a step-by-step crawling manner. This crawling method propagates the bounding boxes of labeled objects by estimating the transformation between images, thereby bypassing the manual labeling step. By using the crawling method, one only needs to focus on replicating the variable conditions under which the images were taken. The real data of one image can be automatically tracked in all other images.

[0023] Reference now Figure 1 , an illustration of a block diagram of a visual inspection crawling system is depicted in accordance with an illustrative embodiment.

[0024] The visual inspection crawling system 100 includes a frame 102 that houses a camera 104 and a plurality of lights 106 that illuminate the surface to be inspected. A crawling drive system 108 moves the frame 102 in a stepwise manner in a crawling direction 128 (see FIG. Figure 2 ).

[0025] The camera 104 continuously captures images 110 in a crawling direction 128. Each image 112 in the sequence of images 110 is taken at a particular crawling step 114, which represents a camera position along the crawling direction 128. The lights 106 may be used to create a plurality of different lighting conditions 116 for each image 112. For example, at each crawling step 114, the camera 104 may take pictures in which the lights 106 illuminate the surface only from the right side, only from the left side, and in which the lights are off (see FIG. 1 ). Figure 3 ). In this way, an object of interest 122 can be captured in various light settings. At each crawling step 114, the object 122 is captured for each lighting condition 116. The images 110 taken under one lighting condition constitute a crawling sequence.

[0026] The images 110 are organized into consecutive pairs of matches 130. The change in camera position represented by each crawling step 114 is modeled as a pairwise quadratic transformation 138, i.e., the change between two consecutive images 110 in the sequence can be fit to a quadratic regression model 136 (see Figure 4 ). Fitting the regression model 136 requires a set of related and independent anchor points 120. Since the crawling direction 128 and the crawling step 114 are known, these anchor points 120 are generated using tile-based template matching (see Figure 5 ).

[0027] Each image 112 is divided into a plurality of tiles 118. The use of tiles 118 reduces the search space, thereby speeding up the process. The matching results of the two images from all different lighting conditions 116 are combined. These results are subjected to statistics and quadrant-based filtering according to a normal distribution 132 and a quadrant distribution 134 to generate the best anchor points 120 for a quadratic regression model 136 (see Figure 6 ).

[0028] This matching process is repeated for each consecutive pair of images in the sequence.

[0029] The model estimated by the secondary transform 138 is used to predict the vertices 126 of the bounding box 124 around the object of interest 122 (see Figure 7 ). Only one image in a sequence needs to provide ground truth data. The quadratic regression model 138 predicts the trajectory 140 of the bounding box vertices in the crawling direction 128, thereby automatically annotating all sequences in all lighting conditions 116 (see Figure 8 ). Each trajectory 140 is characterized by a score reflecting the correctness of the data propagation, further ensuring the robustness of the dataset.

[0030] The visual inspection crawling system 100 can be implemented with software, hardware, firmware, or a combination thereof. When software is used, the operations performed by the visual inspection crawling system 100 can be implemented with program code, which is configured to run on hardware (e.g., a processor unit). When firmware is used, the operations performed by the visual inspection crawling system 100 can be implemented with program code and data, and stored in a persistent memory to run on a processor unit. When hardware is used, the hardware can include circuits that operate to perform the operations in the visual inspection crawling system 100.

[0031] In an illustrative example, the hardware can take the form of at least one selected from the following: a circuit system, an integrated circuit, an application specific integrated circuit (ASIC), a programmable logic device, or some other suitable type of hardware configured to perform multiple operations. In the case of a programmable logic device, the device can be configured to perform multiple operations. The device can be reconfigured at a later time, or can be permanently configured to perform multiple operations. Programmable logic devices include, for example, programmable logic arrays, programmable array logic, field programmable logic arrays, field programmable gate arrays, and other suitable hardware devices. Additionally, these processes can be implemented in organic components integrated with inorganic components, and can completely include organic components (excluding humans). For example, these processes can be implemented as circuits in organic semiconductors.

[0032] Computer system 150 is a physical hardware system and includes one or more data processing systems. When there is more than one data processing system in computer system 150, these data processing systems communicate with each other using a communication medium. The communication medium may be a network. The data processing system may be selected from at least one of the following: a computer, a server computer, a tablet computer, or some other suitable data processing system.

[0033] As depicted, computer system 150 includes multiple processor units 152, which are capable of executing program code 154 that implements the process in the illustrative example. As used herein, the processor unit in multiple processor units 152 is a hardware device, and includes hardware circuits (such as those on integrated circuits), which respond to and process instructions and program codes of operating computers. When multiple processor units 152 execute program code 154 for a process, multiple processor units 152 are one or more processor units that can be on the same computer or on different computers. In other words, the process can be distributed between processor units on the same or different computers in the computer system. In addition, multiple processor units 152 can be processor units of the same type or different types. For example, multiple processor units can be selected from at least one of the following: a single-core processor, a dual-core processor, a multi-processor core, a general-purpose central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP) or a processor unit of some other type.

[0034] Figure 2 A computer vision imaging setup is depicted in accordance with an illustrative embodiment. Setup 200 includes a frame 202 on which is mounted a camera 204 for imaging a surface 210. In this example, surface 210 is the exterior surface of an aircraft fuselage, but the principles of the illustrative embodiments may be applicable to other types of surfaces.

[0035] A plurality of lights 206 are also mounted on both sides of the frame 202. When the frame crawls along the surface 210, the lights 206 can be turned on and off to illuminate the surface 210 from different directions (see FIG. Figure 3 ), thereby allowing camera 204 to capture images under different lighting conditions at each crawling step (camera position) along surface 210.

[0036] Figure 3 Computer vision imaging under different lighting conditions is depicted in accordance with an illustrative embodiment. Figure 3 shows the use of a computer vision imaging setup (e.g. Figure 2 Two image sequences 302, 304 captured by the computer vision imaging setup 200).

[0037] In this example, the image sequence 302, 304 is a sequence of images of a forward portion 306 of an aircraft fuselage near the nose. The image sequence 302 includes images captured under lighting conditions in the forward direction of the fuselage toward the rear (aft). In the context of the images shown, the lighting conditions have lights on the left side of the image, and the lights on the right are off. The image sequence 304 is exactly the opposite, where the lighting is from the rear to the front, i.e., with lights on the right side of the image, and the lights on the left are off.

[0038] The image sequences 302, 304 cover the same portion of the fuselage surface only under different lighting conditions which change the way surface features and objects of interest appear in the images.

[0039] Figure 4 Paired matching of consecutive images according to an illustrative embodiment is shown. Images 402, 404, 406 represent a set of crawl images taken through multiple crawling steps. As shown, these images may be captured in a forward path or a reverse path.

[0040] The method of the illustrative embodiment is used to map each image in a sequence to the next in the sequence of images in a pairwise manner through the entire sequence. The goal is to propagate the bounding box of the object of interest using manually labeled bookend images (image 1 402, image n 406). The pairwise mapping between images is performed according to a function that maps the coordinates between the images:

[0041] (x * ,y * )=f(x,y)

[0042] where x,y are the coordinates of the bounding box in the source image, and x * ,y * are the coordinates of the bounding box in the target image.

[0043] Figure 5 Determination of anchor points in an image according to an illustrative embodiment is shown. Image 502 is divided into a plurality of tiles. Using tiles allows matching specific areas between images, rather than matching the entire image to the entire image in the sequence.

[0044] In this example, tile 506 is being matched between image 502 and image 504. The center of tile 506 defines an anchor point that exists in both images 502, 504. In a pairwise match, the anchor point in the first image 502 may be referred to as a source anchor point, and the anchor point in the second image may be referred to as a target anchor point. Matching source and target anchor points between images takes into account that the camera and lights will not necessarily move in a perfectly straight line from one crawling step to the next.

[0045] Once the source and target anchors are identified in matched pairs of consecutive images, a quadratic function defining the mapping between the images can be estimated.

[0046] The above matching process can be applied to other patches in the image to identify some corresponding source anchor points and target anchor points in consecutive images.

[0047] Figure 6 Filters for anchor points according to an illustrative embodiment are shown. Two filtering steps are used to identify the most feasible source and target anchor points between consecutive images. The first filtering step uses statistical filtering according to a normal distribution, while the second filtering step relies on a quadrant distribution related to the perceived movement direction of the anchor points between images 602, 604.

[0048] The filtering process can rely on some sequence of data. In this example, the sequence (channel) of data includes a sequence under a first lighting condition (e.g., lit on the left), a sequence under a second lighting condition (e.g., lit on the right), and a difference image generated from the difference between the other two lighting conditions. The filtering process attempts to determine how and where the corresponding anchor point moves between images under different lighting conditions.

[0049] Anchor points with associated position data outside a defined range within the normal distribution (e.g., more than one or two standard deviations) are eliminated. In such cases, the apparent positions of these anchor points in the second image deviate too far from the area where they would be expected to be found based on the movement of the camera from one image to the next (allowing for some degree of imperfect drift of the camera and lights).

[0050] Quadrant distribution filtering is related to the direction of movement of the candidate target anchor points relative to the source anchor points, as shown by the arrow in image 604. Since the direction of camera movement (crawling direction) between images 602, 604 is known, the anchor points should appear to move in the same direction. In addition, the anchor points should also be expected to move within a limited range of angles (again taking into account potential drift of the camera between crawling steps). Therefore, anchor points with obvious movement directions and angles outside the specified direction quadrant are also eliminated.

[0051] Figure 7 704. Bounding boxes of objects in images according to an illustrative embodiment are shown. In this example, image 702 has multiple objects of interest (surface features) that are surrounded by corresponding bounding boxes 706, 708, 710. If the secondary mapping between images is unreliable, the bounding boxes 706, 708, 710 in the subsequent image 704 will not correctly delimit the objects because the trajectory of the bounding box vertices across the image cannot be accurately predicted.

[0052] Figure 8 An example of accurate bounding box tracking across a sequence of images in accordance with an illustrative embodiment is shown. In this particular example, a bounding box 802 is placed around a particular rivet and then tracked across the sequence of images as the camera gradually moves over the surface in a crawling direction. Because the secondary mapping between the images is accurate in this example, the bounding box 802 is maintained around the rivet in question, allowing accurate tracking across the sequence of images.

[0053] Fig. 9 A flow chart of a process for creating a custom data set according to an illustrative embodiment is depicted. Process 900 may be implemented in hardware, software, or both. When implemented in software, the process may take the form of program code that is executed by one or more processor units in one or more hardware devices located in one or more computer systems. For example, process 900 may be executed in Figure 2 The visual inspection crawling system 100 is implemented in the crawling system 100.

[0054] Process 900 begins with capturing a sequence of images of a surface in a specified crawling direction with a camera through a plurality of consecutive crawling steps representing a change in camera position (operation 902). The images captured at each crawling step may be captured under a plurality of different lighting conditions.

[0055] For each crawling step, process 900 matches the first image and the second image captured at consecutive camera positions (operation 904).

[0056] The process 900 models each crawling step as a pairwise quadratic transformation between the first image and the second image according to a plurality of corresponding anchor points in each image that are selected as the best candidates for fitting the regression model (operation 906). The anchor points may be selected according to a statistical filter for the position distribution of the anchor points across different lighting conditions. The anchor points may also be selected according to the distribution of the movement directions of the anchor points across different lighting conditions.

[0057] Process 900 receives an input of a bounding box around an object on a surface (operation 908) and predicts a trajectory of bounding box vertices over a sequence of images based on the crawling step modeling (operation 910). The trajectory of the specified bounding box vertices can automatically annotate a sequence of images under all lighting conditions, where only one image in the sequence needs to provide ground truth data. The unique identifier of the object can be used to filter any duplicate images. Process 900 then ends.

[0058] Fig.10 A flow chart of a process for matching a first image and a second image for each crawling step is depicted in accordance with an illustrative embodiment. Process 1000 is Fig. 9 Detailed example of operation 904 in FIG.

[0059] The process 1000 begins by dividing each image into a plurality of tiles (operation 1002), and matching the tiles (operation 1004). The process 1000 then generates anchor points (operation 1006). The process 1000 then ends.

[0060] Fig.11 A flow chart of a process for propagating bounding boxes according to an illustrative embodiment is depicted. Process 1100 is Fig. 9 Detailed example of operation 910 in FIG.

[0061] Process 1100 begins by determining a position of an object relative to a sequence of images (operation 1102 ).

[0062] In response to determining that the object is located at the beginning of the sequence of images, process 1100 performs a forward propagation (operation 1104 ).

[0063] In response to determining that the object is located at the end of the sequence of images, process 1100 performs back propagation (operation 1104 ).

[0064] In response to determining that the object is located at a location other than the beginning or the end of the sequence of images, process 1100 performs forward and backward propagation (operation 1106). Process 1100 then ends.

[0065] The flow chart and block diagram in the embodiment of different descriptions show the framework, function and operation of some possible implementations of device and method in illustrative embodiment.In this respect, each block in flow chart or block diagram can represent at least one of the part of module, fragment, function or operation or step.For example, one or more blocks can be implemented as a combination of program code, hardware or program code and hardware.When implemented with hardware, hardware can, for example, take the form of integrated circuits, and these integrated circuits are manufactured or configured to perform one or more operations in flow chart or block diagram.When implemented as a combination of program code and hardware, the embodiment can take the form of firmware.Each block in flow chart or block diagram can use the combination of special hardware system or special hardware and the program code run by special hardware to perform different operations to realize.

[0066] In some alternative implementations of the illustrative embodiments, one or more functions noted in the blocks may not appear in the order noted in the figure. For example, in some cases, two blocks shown in succession may be performed substantially simultaneously, or the blocks may sometimes be performed in reverse order, depending on the functions involved. In addition, in addition to the blocks shown in the flow chart or block diagram, other blocks may be added.

[0067] Now go to Fig.12 , depicts an illustration of a block diagram of a data processing system according to an illustrative embodiment. Data processing system 1200 may be used to implement Figure 1 12. In this illustrative example, data processing system 1200 includes communications framework 1202, which provides communications between processor unit 1204, memory 1206, persistent storage 1208, communications unit 1210, input / output (I / O) unit 1212, and display 1214. In this example, communications framework 1202 takes the form of a bus system.

[0068] Processor unit 1204 is used to execute instructions for software that may be loaded into memory 1206. Processor unit 1204 may be multiple processors, a multi-processor core, or some other type of processor, depending on the particular implementation. In one embodiment, processor unit 1204 includes one or more conventional general-purpose central processing units (CPUs). In an alternative embodiment, processor unit 1204 includes one or more graphics processing units (GPUs).

[0069] Memory 1206 and persistent storage 1208 are examples of memory devices 1216. A memory device is any piece of hardware capable of storing information (such as, for example, but not limited to, at least one of data, program code in functional form, or other suitable information) on a temporary basis, on a permanent basis, or both on a temporary basis and on a permanent basis. In these illustrative examples, memory devices 1216 may also be referred to as computer-readable memory devices. In these examples, memory 1206 may be, for example, a random access memory or any other suitable volatile or non-volatile memory device. Persistent storage 1208 may take various forms, depending on the particular implementation.

[0070] For example, persistent storage 1208 may include one or more components or devices. For example, persistent storage 1208 may be a hard drive, a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination thereof. The media used by persistent storage 1208 may also be removable. For example, a removable hard drive may be used for persistent storage 1208. In these illustrative examples, communications unit 1210 provides for communications with other data processing systems or devices. In these illustrative examples, communications unit 1210 is a network interface card.

[0071] Input / output unit 1212 allows for input and output of data with other devices that may be connected to data processing system 1200. For example, input / output unit 1212 may provide a connection for user input through at least one of a keyboard, a mouse, or some other suitable input device. Additionally, input / output unit 1212 may send output to a printer. Display 1214 provides a mechanism for displaying information to a user.

[0072] Instructions for at least one of an operating system, applications, or programs may be located in memory devices 1216, which are in communication with processor unit 1204 through communications framework 1202. The processes of the different embodiments may be performed by processor unit 1204 using computer-implemented instructions, which may be located in a memory, such as memory 1206.

[0073] These instructions are referred to as program code, computer usable program code, or computer readable program code, which can be read and executed by a processor in processor unit 1204. The program code in different embodiments may be embodied on different physical or computer readable storage media (e.g., memory 1206 or persistent storage 1208).

[0074] Program code 1218 is located in functional form on computer readable media 1220, which is selectively removable and can be loaded or transferred to data processing system 1200 for execution by processor unit 1204. In these illustrative examples, program code 1218 and computer readable media 1220 form computer program product 1222. In one example, computer readable media 1220 may be computer readable storage media 1224 or computer readable signal media 1226.

[0075] In these illustrative examples, computer-readable storage media 1224 is a physical or tangible memory device for storing program code 1218 rather than a medium that propagates or transmits program code 1218. Computer-readable storage media 1224, as used herein, should not be construed as transitory signals, per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses traveling through fiber optic cables), or electrical signals transmitted through wires.

[0076] Alternatively, program code 1218 may be transmitted to data processing system 1200 using computer readable signal media 1226. Computer readable signal media 1226 may be, for example, a propagated data signal containing program code 1218. For example, computer readable signal media 1226 may be at least one of an electromagnetic signal, an optical signal, or any other suitable type of signal. These signals may be transmitted via at least one of a communication link, such as a wireless communication link, a fiber optic cable, a coaxial cable, an electrical wire, or any other suitable type of communication link.

[0077] The different components shown for data processing system 1200 are not meant to provide architectural limitations to the manner in which different embodiments may be implemented. The different illustrative embodiments may be implemented with a data processing system that includes components in addition to or in place of those shown for data processing system 1200. Fig.12 Other components shown in the example may be different from the illustrative examples shown. Different embodiments may be implemented using any hardware device or system capable of running program code 1218.

[0078] As used herein, when used with a list of items, the phrase "at least one of" means that different combinations of one or more of the listed items may be used, and only one of each item in the list may be required. In other words, "at least one of" means that any combination of items and numbers of items from the list may be used, but not all items in the list are required. An item may be a specific object, thing, or category.

[0079] For example, but not limited to, "at least one of item A, item B, or item C" may include item A, item A and item B, or item B. This example may also include item A, item B, and item C, or item B and item C. Of course, any combination of these items may exist. In some illustrative examples, "at least one of..." may be, for example, but not limited to, two of item A; one of item B; and ten of item C; four of item B and seven of item C; or other suitable combinations.

[0080] As used herein, "a plurality of" when used with reference to items means one or more items. For example, "a plurality of different types of networks" is one or more different types of networks. In the illustrative examples, "a set of" when used with reference to items means one or more items. For example, a set of metrics is one or more metrics.

[0081] The description of different illustrative embodiments has been presented for the purpose of illustration and description and is not intended to be exhaustive or limited to the embodiments of the disclosed forms. Different illustrative examples describe components that perform actions or operations. In the illustrative embodiments, the components can be configured to perform the described actions or operations. For example, the components can have a configuration or design for a structure that provides the components with the ability to perform the actions or operations described as being performed by the components in the illustrative examples. In addition, in the context of using the terms "comprises," "comprising," "having," "containing" and variations thereof in this article, such terms are intended to be inclusive in a manner similar to the term "comprises" as an open transition word, without excluding any additional or other elements.

[0082] Many modifications and variations will be apparent to those of ordinary skill in the art. In addition, different illustrative embodiments may provide different features than other desired embodiments. The selected embodiment or embodiments are selected and described in order to best explain the principles of the embodiments, the practical application, and to enable others of ordinary skill in the art to understand the disclosure of various embodiments with various modifications as are suitable for the particular use under consideration.

Claims

1. A computer-implemented method for creating a custom data set, the method comprising: The plurality of processors (152) are used to perform the following operations: capturing (902) a sequence of images (110) of a surface in a specified crawling direction (128) with a camera (104) through a plurality of consecutive crawling steps (114) representing changes in camera position; For each crawling step, matching (904) a first image and a second image captured at consecutive camera positions; as well as Each crawling step is modeled (906) as a pairwise quadratic transformation (138) between the first image and the second image based on a plurality of corresponding anchor points (120) in each image that are selected as best candidates for fitting a regression model (136).

2. The method of claim 1, wherein matching the first image and the second image further comprises: Dividing (1002) each image into a plurality of tiles (118); matching (1004) the tiles; and The anchor point is generated (1006).

3. The method of claim 1, wherein the images captured at each crawling step are captured under a plurality of different lighting conditions (116). The method of claim 3 , wherein the anchor points are selected according to a statistical filter for the distribution of their positions across the different lighting conditions. The method of claim 3 , wherein the anchor points are selected according to a distribution of movement directions of the anchor points across the different lighting conditions.

6. The method according to claim 1, further comprising: receiving (908) input of a bounding box (124) around an object on the surface; and Based on the crawling step modeling, a trajectory (140) of the vertices (126) of the bounding box over the sequence of images is predicted (910).

7. The method of claim 6, further comprising using the object's unique identifier to filter any duplicate images.

8. The method of claim 6, wherein the locus of the vertices specifying the bounding box automatically annotates the sequence of images under all lighting conditions, and wherein ground truth data need only be provided for one image in the sequence.

9. The method according to claim 6, further comprising: In response to determining that the object is located at the beginning of the sequence of images, performing a forward propagation (1104); In response to determining that the object is located at an end of the sequence of images, performing back propagation (1106); and In response to determining that the object is located at a location other than the beginning or the end of the sequence of images, forward and backward propagation (1108) is performed.

10. A system for creating a custom data set, the system comprising: a memory device (1216) storing program instructions; One or more processors (1204) operably connected to the memory device and configured to execute the program instructions so that the system: capturing (902) a sequence of images (110) of a surface in a specified crawling direction (128) with a camera (104) through a plurality of successive crawling steps (114) representing changes in camera position; For each crawling step, matching a first image and a second image captured at consecutive camera positions; as well as Each crawling step is modeled (906) as a pairwise quadratic transformation (138) between the first image and the second image based on a plurality of corresponding anchor points (120) in each image that are selected as best candidates for fitting a regression model (136).

11. The system of claim 10, wherein matching the first image and the second image further comprises: Dividing (1002) each image into a plurality of tiles (118); matching (1004) the tiles; and The anchor point is generated (1006).

12. The system of claim 10, wherein the images captured at each crawling step are captured under a plurality of different lighting conditions (116).

13. The system of claim 12, wherein the anchor points are selected according to a statistical filter for the distribution of their positions across the different lighting conditions.

14. The system of claim 12, wherein the anchor points are selected based on a distribution of movement directions of the anchor points across the different lighting conditions.

15. The system of claim 10, wherein the processor further executes program instructions so that the system: receiving (908) input of a bounding box (124) around an object on the surface; and Based on the crawling step modeling, a trajectory (140) of the vertices (126) of the bounding box on the sequence of images is predicted (910).

16. The system of claim 15, wherein the processor further executes program instructions to cause the system to filter any duplicate images using the unique identifier of the object.

17. The system of claim 15, wherein the locus of the vertices specifying the bounding box automatically annotates the sequence of images under all lighting conditions, and wherein ground truth data need only be provided for one image in the sequence.

18. The system of claim 15, wherein the processor further executes program instructions so that the system: In response to determining that the object is located at the beginning of the sequence of images, performing a forward propagation (1104); In response to determining that the object is located at an end of the sequence of images, performing back propagation (1106); and In response to determining that the object is located at a location other than the beginning or the end of the sequence of images, forward and backward propagation (1108) is performed.

19. A computer program product for creating a custom data set, the computer program product comprising: A computer readable storage medium (1224) having program instructions embodied thereon to perform the following operations: capturing (902) a sequence of images (110) of a surface in a specified crawling direction (128) with a camera (104) through a plurality of consecutive crawling steps (114) representing changes in camera position; For each crawling step, matching (904) a first image and a second image captured at consecutive camera positions; as well as Each crawling step is modeled (906) as a pairwise quadratic transformation (138) between the first image and the second image based on a plurality of corresponding anchor points (120) in each image that are selected as best candidates for fitting a regression model (136).

20. The computer program product of claim 19, further comprising instructions for: receiving (908) input of a bounding box (124) around an object on the surface; and Based on the crawling step modeling, a trajectory (140) of the vertices (126) of the bounding box over the sequence of images is predicted (910).

Images