System for determining defect in physical object, and defect detector module

Abstract

To determine a defect of physical objects such as railway components.SOLUTION: In one embodiment, a method includes the steps of receiving, by a defect detector module, an image of a physical object and classifying, by the defect detector module, one or more first features from the image of the physical object into one or more first classifications using one or more machine learning algorithms. The method further includes the steps of analyzing, by the defect detector module, the one or more first classifications and determining, by the defect detector module, that the physical object comprises a defect based on the step of analyzing the one or more first classifications.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to defect determination, and more specifically, to defect determination in physical objects. [Background technology]

[0002] Physical objects are used in various industries to perform business objectives. For example, railroad cars move freight along tracks. The railroad cars and tracks include a combination of several physical objects. Over time, the physical objects may wear out, break, or have other defects, and repairs to the defects may be required for the continued safe operation of the railroad cars and tracks. Typically, physical objects such as railroad components are manually inspected by technicians to identify defects. Summary of the Invention [Means for solving the problem]

[0003] According to one embodiment, a method includes receiving, by a defect detector module, an image of a physical object, and classifying, by the defect detector module, one or more first features from the image of the physical object into one or more first classifications using one or more machine learning algorithms, The method further includes analyzing, by the defect detector module, the one or more first classifications, and determining that the physical object includes a defect based on the analyzing, by the defect detector module, the one or more first classifications.

[0004] According to another embodiment, a system includes one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including receiving, by a defect detector module, an image of a physical object, and classifying, by the defect detector module, one or more first features from the image of the physical object into one or more first classifications using one or more machine learning algorithms, The operations further include analyzing, by the defect detector module, the one or more first classifications, and determining that the physical object includes a defect based on analyzing, by the defect detector module, the one or more first classifications.

[0005] According to yet another embodiment, one or more computer-readable storage media embodied with instructions that, when executed by a processor, cause the processor to perform operations including receiving, by a defect detector module, an image of a physical object, and classifying, by the defect detector module, one or more first features from the image of the physical object into one or more first classifications using one or more machine learning algorithms, The operations further include analyzing, by the defect detector module, the one or more first classifications, and determining that the physical object includes a defect based on analyzing, by the defect detector module, the one or more first classifications.

[0006] Technical advantages of certain embodiments of the present disclosure may include one or more of the following: The disclosed image capture system automatically captures images of physical objects, eliminating or reducing manual data collection and human labor, saving time and money. The image capture system can capture images of physical objects, such as railroad components, from angles that cannot be identified by manual inspection, which may increase the accuracy of detecting defects in the physical objects. The disclosed defect detector module can automatically detect defects in physical objects using machine learning algorithms, which may eliminate the manual effort of scanning and labeling images of potential defects. The systems and methods described in this disclosure may be generalized across different transportation infrastructures, including rail, road, and waterways.

[0007] Other technical advantages will be readily apparent to one skilled in the art from the following figures, descriptions, and claims. Additionally, while certain advantages have been enumerated above, various embodiments may include all, some, or none of the enumerated advantages.

[0008] For an aid in understanding the present disclosure, reference is made to the following descriptions taken in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0009] [Figure 1] 1 illustrates an example system for determining defects in a physical object. [Diagram 2] 1 illustrates an example method for determining defects in a physical object. [Diagram 3] 4 illustrates another example method for determining defects in a physical object. [Figure 4] 1 illustrates an exemplary image capture system. [Diagram 5] 5 illustrates an exemplary module that may be used by the image capture system of FIG. 4. [Figure 6] 1 illustrates an exemplary method for tagging features with labels. [Figure 7]1 illustrates an exemplary computer system that may be used with the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Physical objects, such as railroad components, are used in various industries to perform business objectives. Over time, physical objects can become damaged or develop defects that may require repair in order to allow the continued safe operation of rail vehicles. Typically, physical objects, such as railroad components, are manually inspected by technicians to identify defects.

[0011] Condition and defect analysis algorithms may leverage one or more traditional computer vision techniques (e.g., region of interest analysis, filtering, thresholds, blob techniques, edge analysis, contour extraction, histogram analysis, etc.) and common assets and / or defect heuristics to automate the detection of various conditions. Such techniques are effective when very specific and consistent, but are often brittle under real-world conditions and can require significant time for maintenance, correction, and enhancement.

[0012] The use of machine learning algorithms for image classification and segmentation tasks involves a fundamentally different approach to image analysis techniques than traditional computational methods: instead of algorithmically filtering, warping, grouping, segmenting, and computing on pixel matrices, machine learning algorithms (e.g., neural networks) make predictions about the content of an image using a series of trained network layers through which an image or a portion of an image is passed.

[0013] Embodiments of the present disclosure use deep machine learning to capture and analyze visual images. The disclosed systems and methods recognize physical objects (e.g., railroad components) in the context of a particular hardware configuration of a moving vehicle (e.g., a vehicle moving on rails, a vehicle moving along rails, etc.) and interpret the visual data based on the amount, variety, and accuracy of continuously captured, pooled, and weighted labeled training data. The disclosed systems and methods perform the interpretation by using models trained on captured image data of the physical objects to obtain higher accuracy and a higher level of analysis of the various captured physical objects. Machine learning algorithms are custom trained for each hardware configuration to achieve optimal sequential capture and analysis for each of the various physical objects. The algorithms are used to classify specific features from the images of the physical objects. The classification is then used to determine defects in the physical objects.

[0014] 1-7 illustrate example systems and methods for determining defects in a physical object. FIG. 1 illustrates an example system for determining defects in a physical object. FIGS. 2 and 3 illustrate example methods for determining defects in a physical object. FIG. 4 illustrates an example image capturing system. FIG. 5 illustrates example modules that may be used by the image capturing system of FIG. 4. FIG. 6 illustrates an example method for tagging features with labels. FIG. 7 illustrates an example computer system that may be used by the systems and methods described herein.

[0015] FIG 1 illustrates an exemplary system 100 for determining defects in a physical object. The system 100 of FIG 1 includes a network 110, a defect detector module 120, and an image capture system 170. The system 100, or portions thereof, may include or be associated with any entity, such as a business, company (e.g., railroad company, transportation company, etc.), or government agency (e.g., traffic department, public safety department, etc.) that can determine defects in a physical object. Elements of the system 100 may be implemented using any suitable combination of hardware, firmware, and software.

[0016] Network 110 may be any type of network that facilitates communication between components of system 100. Network 110 may connect defect detector module 120 and image capture system 170 of system 100. Although this disclosure illustrates network 110 being a particular type of network, this disclosure contemplates any suitable network. One or more portions of network 110 may include an ad-hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular network, a 3G network, a 4G network, a 5G network, a Long Term Evolution (LTE) cellular network, a combination of two or more of these, or any other suitable type of network. One or more portions of network 110 may include one or more access (e.g., mobile access), core, and / or edge networks. The network 110 may be any communication network, such as a private network, a public network, a connection via the Internet, a mobile network, a WI-FI network, a Bluetooth network, etc. The network 110 may include one or more network nodes. A network node is a connection point that can receive, generate, store, and / or transmit data via the network 110. The network 110 may include cloud computing capabilities. One or more components of the system 100 may communicate via the network 110. For example, the defect detector module 120 may communicate via the network 110, including receiving information from the image capture system 170.

[0017] The defect detector module 120 of the system 100 represents any suitable computing component that may be used to determine defects 156 in a physical object. A defect 156 is a flaw that potentially destroys the usefulness of a physical object. A physical object is an identifiable collection of material. Physical objects may include transportation infrastructure components such as roads, railroads, airways, waterways, canals, pipelines, and terminal components. Examples of railroad components include joints, switches, frogs, rail heads, anchors, fasteners, gauge plates, ballast, sleepers (e.g., concrete sleepers and wooden sleepers), and the like.

[0018] The defect detector module 120 includes an interface 122, a memory 124, and a processor 126. The interface 122 of the defect detector module 120 represents any suitable computing element that may receive information from the network 110, transmit information over the network 110, perform suitable processing of the information, communicate with other components of the system 100 of Figure 1 (e.g., components of the image capture system 170), or any combination thereof. The interface 122 represents any port or connection, actual or virtual, including any suitable combination of hardware, firmware, and software, including protocol conversion and data processing capabilities for the system 100 of Figure 1 to communicate over a LAN, WAN, or other communication system for exchanging information between components of the system 100.

[0019] The memory 124 of the defect detector module 120 stores, permanently and / or temporarily, received and transmitted information as well as system software, control software, other software for the defect detector module 120, and various other information. The memory 124 may store information for execution by the processor 126. The memory 124 includes any one or combination of volatile or non-volatile local or remote devices suitable for storing information. The memory 124 may include random access memory (RAM), read only memory (ROM), magnetic storage, optical storage, or any other suitable information storage device, or combination of these devices. The memory 124 may include any suitable information for use in the operation of the defect detector module 120. Additionally, the memory 124 may be a component external (or partially external) to the defect detector module 120. The memory 124 may be located in any location suitable for the memory 124 to be in communication with the defect detector module 120. The memory 124 of the defect detector module 120 may store the image collection engine 130, the classification engine 132, the defect detector engine 134, the reporting engine 136, the labeling engine 138, and the training engine 140. As another example, the image collection engine 130, the classification engine 132, the defect detector engine 134, the reporting engine 136, the labeling engine 138, and the training engine 140 may be external to the memory 124 and / or the defect detector module 120.

[0020] The image collection engine 130 of the defect detector module 120 is an application that collects images 152 of physical objects. The image collection engine 130 may receive one or more images 152 of one or more physical objects from an image capture system 170 over the network 110. The image collection engine 130 may receive the images 152 of the physical objects in real time or near real time as the physical objects are captured by the image capture system 170. The image collection engine 130 may receive the images 152 of the physical objects according to a schedule (e.g., minutely, hourly, weekly, etc.). The image collection engine 130 may combine (e.g., stitch) one or more images 152 to generate a combined image 152. The image collection engine 130 may group the images 152 by any suitable combination, such as by the physical object by which the image 152 was captured by the image capture system 170, the time the image 152 was received by the image collection engine 130, and / or the location 158 (e.g., geographic location 158) at which the image 152 was captured.

[0021] The classification engine 132 of the defect detector module 120 is an application that classifies features 154 from one or more images 152 of one or more physical objects into one or more classifications 155. Each feature 154 may be a feature of an image 152. For example, the features 154 may include railroad components such as a bolt head. The features 154 may represent the physical object itself or one or more components of the physical object. For example, a first feature 154 may represent a physical object (e.g., a rail joint) and one or more second features 154 may represent components of the rail joint (e.g., a bolt head, a square nut, a hex nut, a round nut, a hole, etc.).

[0022] The classification engine 132 analyzes data from the images 152 by applying one or more machine learning algorithms. The machine learning algorithms may be associated with one or more neural networks (e.g., deep neural networks), one or more deep learning algorithms, one or more convolutional neural networks (CNNs), artificial intelligence (AI), other suitable applications, or any suitable combination of the above. Each machine learning algorithm may be trained on the labeled image data to recognize one or more features 154 from the one or more images 152. The one or more machine learning algorithms used by the classification engine 132 are trained to recognize specific physical objects (e.g., railroad components) seen by one or more components (e.g., one or more cameras) of the image capture system 170. For example, the one or more CNNs 166 may use a series of convolutions, pooling, reduction, and fully connected layers through which the image 152 or a portion of the image 152 passes to determine a prediction about the content of the image 152. The one or more machine learning algorithms output one or more classifications 155 that describe the one or more features 154 of the one or more images 152 within a level of certainty. The level of certainty may vary depending on the number of classifications 155, the desired accuracy, the availability of sample training data for each classification 155, etc. The level of certainty may be expressed as a probability.

[0023] One or more classifications 155 are used to identify one or more features 154. For example, a first classification 155 may indicate a first feature 154 of a rail joint, and one or more second classifications 155 may indicate one or more second features 154 related to the rail joint, such as a bolt head, a square nut, a hex nut, a round nut, a hole, etc. The classifications 155 may include a location of the feature 154 within the image 152. For example, a "bolt head" classification 155 may be part of a physical object (e.g., a rail joint) and may include a location of a bolt head within the image 152.

[0024] One or more of the classifications 155 may include one or more defects 156. A defect 156 includes any attribute of a physical object that is a defect or deficiency. A defect 155 may include defects such as cracks, breaks, holes, mismatches, chips, wear, and the like. For example, rail joint defects 155 may include missing bolts, breaks, cracks, sleeper condition (e.g., sleeper deterioration), rail mismatch, rail end splits, rail end crushed, ballast condition (e.g., ballast deterioration), properly sized joint bars, and the like. As another example, railroad switch defects 156 may include crushed points, broken points, missing fasteners (e.g., base rail), bent switch rods, rail run at switch points, improper sleeper spacing, improper rod spacing, improper flangeway width, and the like. Railroad frog defects 156 may include crushed points, broken points, tread wear, broken gauge plates, bent gauge plates, missing fasteners (e.g., missing frog bolts), missing plates, broken guard rails, and the like.

[0025] Rail head defects 156 may include rail corrugation, rail spalling, rail shelling, broken rails, and the like. Railroad anchor and / or fastener defects 156 may include missing anchors and / or fasteners, improper anchor pattern, missing clips, missing spikes, and the like. Railroad gauge plate defects 156 may include bent gauge plates, broken gauge plates, and the like. Railroad ballast defects 156 may include insufficient ballast, dirty ballast, rotten ballast, and the like. Railroad concrete sleeper defects 156 may include broken sleepers, cracked sleepers, deteriorated sleepers, insufficient distance between sleepers, and the like. Railroad wooden sleeper defects 156 may include plate cuts, wheel cuts, insufficient distance between sleepers, rotten sleepers, hollow sleepers, and the like. Other defects 156 may be identified or defined based on the physical objects being photographed.

[0026] The defect detector engine 134 of the defect detector module 120 is an application that determines whether a physical object includes one or more defects 156 based on an analysis of the output of one or more machine learning algorithms (e.g., CNN 166). For example, the defect detector engine 134 may analyze one or more classifications 155 output by the one or more machine learning algorithms. The defect detector engine 134 may apply the one or more algorithms to the detection results (e.g., classifications 155) to determine (e.g., optically identify) the one or more defects 156.

[0027] The defect detector engine 134 may analyze one or more classifications 155 to identify one or more defects 156. For example, the defect detector engine 134 may determine that a physical object (e.g., a rail joint) includes a defect 156 if the classification 155 associated with the physical object is the defect 156 (e.g., a break). The defect detector engine 134 may determine a location 158 of the defect 156 on the physical object relative to other features 154 of the physical object to generate additional information for the defect 156. For example, the defect detector engine 134 may identify a break in a joint bar based on results of one or more machine learning algorithms including a "break" and a "joint bar" classification 155. The defect detector engine 134 may determine that the joint bar includes a bolt based on a "bolt" classification 155. The defect detector engine 134 may determine a location 158 of the break relative to the bolt. The defect detector engine 134 may generate additional information for the break based on its relative location 158. For example, the defect detector engine 134 may classify a break as a "center break defect" if the break is between two middle bolts. As another example, the defect detector engine 134 may classify a break as a "quarter break defect" if the break is outside of the two middle bolts.

[0028] The defect detector engine 134 may determine a geographic location 158 of the defect 156 in the physical object. The defect detector engine 134 may determine the geographic location 158 using information received from the image collection engine 130. The image collection engine 130 may capture information (e.g., latitude and longitude) indicative of the one or more geographic locations 158 associated with the one or more images 152 and transmit this information to the defect detector engine 134. The defect detector engine 134 may convert the information received from the image collection engine 130 into any suitable format. For example, the defect detector engine 134 may convert the latitude and longitude received from the image collection engine 130 into track type, platform, line segment, milepost information, etc. The defect detector engine 134 may identify the geographic location 158 on a map (e.g., an existing map of the railroad track) using one or more algorithms (e.g., a closest point algorithm).

[0029] In certain embodiments, the defect detector engine 134 may detect welds (e.g., thermite welds) in the images 152 of the rail and use the weld location to identify the milepost. Each image 152 may represent "n" feet (e.g., 5 feet) of rail that is "n" feet (e.g., 5 feet) wide, where n represents any suitable number. The defect detector engine 134 may determine the geographic location 158 of the physical object by determining footage from the images 152 of the physical object associated with the milepost. The defect detector engine 134 may determine the geographic location 158 of the physical object by footage from the physical object in an up and / or down direction of the milepost. For example, the defect detector engine 134 may determine that the physical object is a certain number of feet (e.g., 1000 feet) past milepost 100, which may be represented (e.g., output) as "MP 100.0+1000 feet." As another example, the defect detector engine 134 may determine that the physical object is a particular number of feet (e.g., 1000 feet) in front of the milepost 137, which may be represented (e.g., output) as "MP 100.0-1000 feet." In particular embodiments, the defect detector engine 134 may represent the geographic location 158 as Global Positioning System (GPS) coordinates.

[0030] The defect detector engine 134 may determine the geographic location 158 based at least in part on the location 158 of the defect 156 relative to the physical object. For example, the defect detector engine 134 may determine the geographic location 158 of a break in a rail joint based in part on the location 158 of the break relative to the rail joint. The defect detector engine 134 may determine the geographic location 158 based at least in part on one or more sensors (e.g., position sensors) located within the system 100 (e.g., the image capture system 170). The geographic location 158 may be defined by coordinates (e.g., longitude and latitude).

[0031] The defect detector engine 134 may also determine an identification mark of the physical object. The identification mark of the physical object may include any character (e.g., numbers, letters, etc.) suitable for identifying a physical object. The identification mark of the physical object may identify an owner and / or manufacturer (e.g., a railroad manufacturer) of the physical object. The identification mark of the physical object may be used to trace the origin of the physical object. For example, the identification mark may include a batch code that can trace the physical object to a particular manufacturing batch. The defect detector engine 134 may determine the identification mark of the physical object based on the geographic location 158 of the physical object and / or by analyzing one or more images 152 of the physical object.

[0032] The reporting engine 136 of the defect detector module 120 is an application that generates one or more reports 160. The reporting engine 136 may generate the report 160 indicating that the physical object is free of defects if the defect detector engine 134 determines that the physical object does not include defects 156. The reporting engine 136 may generate the report 160 indicating that the physical object is defective if the defect detector engine 134 determines that the physical object includes defects 156. The report 160 may be in any suitable form (e.g., written and / or verbal) for communicating the defects 156. The report 160 may include information related to the physical object, the features 154, the classification 155, the location 158, the label 162, etc. For example, the report 160 may include the label 162, the location 158 (e.g., the geographic location 158), and / or an identifying mark shown in FIG. 6. The report 160 may include one or more charts, tables, lists, graphs, and / or any other suitable format for communicating information. The reporting engine 136 may communicate the report 160 to one or more components of the system 100 (eg, the image capture system 170) and / or to a user of the system 100 (eg, an administrator, technician, etc.).

[0033] The labeling engine 138 of the defect detector module 120 is an application that generates one or more labels 162. The labels 162 provide visual information related to the feature 154, the classification 155, the defect 156, and / or the location 158. The labeling engine 138 tags one or more physical object features 154 with a label 162. Each label 162 may include information related to the physical object. Each label 162 may include the size (e.g., diameter, area, etc.) of the feature 154, probability regarding the feature 154, classification 155, location 158, etc. For example, the label 162 may indicate the feature 154 as "Bolt Head (0.77)" to describe the classification 155 of "Bolt Head" and a probability of 0.77 (i.e., 77%) that the feature 154 is correctly identified as a bolt head.

[0034] The labeling engine 138 may insert one or more portions of a label 162 indicating a feature 154 of a physical object (e.g., a rail joint) into a drawing of the physical object. The labeling engine 138 may insert one or more portions of the label 162 on the drawing at a location 158 where the associated feature 154 occurs on the physical object. For example, for a bolt head feature 154, the labeling engine 138 may insert one or more portions of the label 162 on the drawing of the physical object at a location 158 where the bolt head is located on the physical object. The label 162 may indicate the feature 154 with any suitable shape or character. For example, the label 162 indicating the bolt head feature 154 may be a bounding box having a square or rectangular shape. The defect detector engine 134 may use the label 162 to further classify the defect 156, as described in FIG. 6 below.

[0035] The training engine 140 of the defect detector module 120 is an application that trains one or more machine learning algorithms (e.g., CNN 166). The training engine 140 may use training data (e.g., training images 164) to train the machine learning algorithms, thereby adjusting the weights to accurately recognize the physical objects. The machine learning algorithms may be trained for each particular camera angle of the image capture system 170 to provide the highest accuracy for each classification 155. The training engine 140 may train the one or more machine learning algorithms by collecting sample data (e.g., training images 164) that are indicative of each classification 155. The training engine 140 may label the sample data with labels 162 and use the labeled data to train the one or more machine learning algorithms to recognize each classification 155. The training engine 140 may use a subset of the labeled images to test the accuracy of each machine learning algorithm.

[0036] The training engine 140 may receive initial training data from an administrator of the system 100. The amount and variety of training data (e.g., training images 164) used by one or more machine learning algorithms varies depending on the number of classifications 155, the desired accuracy, and the availability of sample data for each classification 155.

[0037] The memory 124 may store a database 150. The database 150 may store a particular type of information related to the defect detector module 120. For example, the database 150 may store images 152, features 154, classifications 155, defects 156, locations 158, reports 160, labels 162, training images 164, and machine learning algorithms (e.g., CNN 166). The database 150 may be any one or combination of volatile or non-volatile local or remote devices suitable for storing information. The database 150 may include RAM, ROM, magnetic storage, optical storage, or any other suitable information storage device, or combination of these devices. The database 150 may be a component external to the defect detector module 120. The database 150 may be located in any location suitable for the database 150 to store information for the defect detector module 120. For example, the database 150 may be located in a cloud environment.

[0038] The processor 126 of the defect detector module 120 controls certain operations of the defect detector module 120 by processing information received from the interface 122 and the memory 124 or accessed by the processor 126. The processor 126 is communicatively coupled to the interface 122 and the memory 124. The processor 126 may include any hardware and / or software operable to control and process information. The processor 126 may be a programmable logic device, a microcontroller, a microprocessor, any suitable processing device, or any suitable combination thereof. Additionally, the processor 126 may be a component external to the defect detector module 120. The processor 126 may be located in any location suitable for the processor 126 to communicate with the defect detector module 120. The processor 126 of the defect detector module 120 controls the operation of the image acquisition engine 130, the classification engine 132, the defect detector engine 134, the reporting engine 136, the labeling engine 138, and the training engine 140.

[0039] One or more components of the defect detector module 120 may operate in a cloud. The cloud may provide different services (e.g., applications, servers, and storage) to the defect detector module 120 over the network 110. The cloud may be implemented using any suitable combination of hardware, firmware, and software. For example, the cloud may be implemented using one or more components of the computer system of FIG. 7.

[0040] The image capture system 170 of the system 100 illustrates a system for capturing images 152 of a physical object. The image capture system includes a subframe 172, a beam 174, one or more modules 176, a lighting system 178, a heating, ventilation, and air conditioning (HVAC) system 180, a data system 182, and one or more controllers 184. The components of the image capture system 180 are mounted (e.g., physically attached) to a vehicle (e.g., a locomotive). The image capture system 170 may capture one or more images 152 of a physical object while the vehicle moves relative to the physical object. For example, the image capture system 170 may be mounted to a locomotive, and the image capture system 170 may capture images 152 of a rail joint while the locomotive moves along a railroad track. The image capture system 170 may operate similarly to a flatbed document scanner, except that the image capture system 170 is moving while capturing images 152 of a stationary physical object. Image capture system 170 is described in more detail in FIG. 4 below.

[0041] Although FIG. 1 shows a particular arrangement of the network 110, the defect detector module 120, the interface 122, the memory 124, the processor 126, the image collection engine 130, the classification engine 132, the defect detector engine 134, the reporting engine 136, the labeling engine 138, the training engine 140, the database 150, and the image capture system 170, this disclosure contemplates any suitable arrangement of the network 110, the defect detector module 120, the interface 122, the memory 124, the processor 126, the image collection engine 130, the classification engine 132, the defect detector engine 134, the reporting engine 136, the labeling engine 138, the training engine 140, the database 150, and the image capture system 170. The network 110, the defect detector module 120, the interface 122, the memory 124, the processor 126, the image acquisition engine 130, the classification engine 132, the defect detector engine 134, the reporting engine 136, the labeling engine 138, the training engine 140, the database 150, and the image capture system 170 may be physically or logically co-located in whole or in part with one another.

[0042] 1 illustrates a particular number of networks 110, defect detector modules 120, interfaces 122, memory 124, processors 126, image collection engines 130, classification engines 132, defect detector engines 134, reporting engines 136, labeling engines 138, training engines 140, databases 150, and image capture systems 170, this disclosure contemplates any suitable arrangement of networks 110, defect detector modules 120, interfaces 122, memory 124, processors 126, image collection engines 130, classification engines 132, defect detector engines 134, reporting engines 136, labeling engines 138, training engines 140, databases 150, and image capture systems 170. One or more components of the defect detector modules 120 and / or image capture systems 170 may be implemented using one or more components of the computer system of FIG.

[0043] 1 illustrates a system 100 for determining defects 156 in a physical object, one or more components of the system 100 may be applied to other implementations. For example, one or more components of the defect detector module 120 and / or the image capture system 170 may be used for set identification and / or inventory of assets. For example, the classification engine 132 of the defect detector module 120 may be used to identify physical objects (e.g., rail joint bars, switches, crossings, frogs, etc.) and record an inventory of the identified physical objects.

[0044] In operation, the image acquisition engine 130 of the defect detector module 120 receives an image 152 of a physical object (e.g., a rail joint) from the image capture system 170. The classification engine 132 of the defect detector module 120 classifies features 154 (e.g., bolts, breaks, discontinuities, and holes) from the image 152 of the physical object into categories (e.g., bolts, breaks, discontinuities, and holes) using one or more machine learning algorithms (e.g., CNN 166). The defect detector engine 134 of the defect detector module 120 analyzes the categories 155 and determines that the physical object includes a defect 156 (e.g., a break) based on the analysis. The defect detector engine 134 uses the image 152 to determine a location 158 of the defect 156 relative to other features 154 of the physical object. The defect detector engine 134 determines a geographic location 158 of the defect 156 in the physical object based at least in part on the location 158 of the defect 156 relative to other features 154 of the physical object. The reporting engine 136 generates a report 160 indicating that the physical object includes a defect 156. The report 160 includes a geographic location 158 of the defect 156. The labeling engine 138 labels the one or more first features 154 of the image 152 with one or more labels 162. The labels 162 include a label 162 that indicates the defect 156 (e.g., breakage) in the physical object.

[0045] In this manner, the system 100 for reducing or eliminating the need for manual inspection of FIG. 1 determines defects 156 in the physical object by taking an image 152 of the physical object, analyzing the image 152, classifying features 154 from the image 152 using one or more machine learning algorithms, and determining the defects 156 based on the classification 155.

[0046] 2 illustrates an example method 200 for determining defects in a physical object. Method 200 begins at step 210. At step 220, a defect detector module (e.g., defect detector module 120 of FIG. 1 ) receives an image of the physical object (e.g., image 152 of FIG. 1 ) from an image capture system (e.g., image capture system 170 of FIG. 1 ). For example, an image acquisition engine (e.g., image acquisition engine 130 of FIG. 1 ) of the defect detector module may receive an image of a rail joint from the image capture system. The image capture system may be attached to a component (e.g., a locomotive), and the image capture system may capture images while the component moves relative to the physical object.

[0047] In step 230, the defect detector module classifies one or more features from the image (e.g., features 154 in FIG. 1 ) into one or more classifications (e.g., classification 155 in FIG. 1 ) using one or more machine learning algorithms (e.g., CNN 166 in FIG. 1 ). For example, the classification engine of the defect detector module (e.g., classification engine 132) may use one or more CNNs to classify features such as square nuts, bolt heads, breaks, holes, discontinuities, and bars into classifications that include each of square nuts, bolt heads, breaks, holes, discontinuities, and bars. Because the visual division between square nuts and bolt heads equates to a physical representation of a bolt, one or more algorithms may logically reduce the classifications of square nuts and bolt heads to a single bolt classification.

[0048] At step 240, the defect detector module analyzes the one or more classifications. A defect detector engine of the defect detector module (e.g., defect detector engine 134 of FIG. 1 ) may analyze the one or more classifications to identify the presence of a defect in the physical object (e.g., defect 156 of FIG. 1 ). At step 250, the defect detector module determines whether the physical object is defective based on the classifications. The defect detector engine may determine whether the physical object is defective if one or more of the classifications include a defect (e.g., breakage).

[0049] If the defect detector module determines that the physical object is not defective based on the classification, method 200 proceeds from step 250 to step 260, where a reporting engine (e.g., reporting engine 136 of FIG. 1) generates a report indicating that the physical object is not defective. Method 200 then proceeds to step 280, where method 200 ends. If the defect detector module determines that the physical object is defective based on the classification, method 200 proceeds from step 250 to step 270, where a reporting engine generates a report indicating that the physical object is defective. Method 200 then proceeds to step 280, where method 200 ends.

[0050] Modifications, additions, or omissions may be made to the method 200 shown in FIG. 2. Method 200 may include more, fewer, or other steps. For example, method 200 may include training with a training engine (e.g., training engine 140 of FIG. 1 ) of the defect detector module, one or more neural networks, and / or algorithms (e.g., CNNs) to accurately recognize and classify features in an image. As another example, method 200 may include generating, by a labeling engine (e.g., labeling engine 138 of FIG. 1 ) of the defect detector module, one or more labels (e.g., labels 162 of FIG. 1 ) indicative of the one or more features. Steps may be performed in parallel or in any suitable order. Although discussed as specific components completing steps of method 200, any suitable component may perform any step of method 200.

[0051] 3 illustrates an example method 300 for determining defects in a physical object. Method 300 begins at step 305. In step 310, an image acquisition engine (e.g., image acquisition engine 130 of FIG. 1) of a defect detector module (e.g., defect detector module 120 of FIG. 1) receives an image of the physical object (e.g., image 152 of FIG. 1) from an image capture system (e.g., image capture system 170 of FIG. 1). The image acquisition engine of the defect detector module may detect the presence of a rail joint in the image.

[0052] At step 315, the defect detector module classifies a first feature from the image (e.g., feature 154 in FIG. 1 ) into a first classification (e.g., classification 155 in FIG. 1 ). For example, one or more CNNs may be trained to detect classifications such as "bar," "discontinuity," and "end post," and a classification engine of the defect detector module (e.g., classification engine 132 in FIG. 1 ) may use one or more CNNs to classify the first feature of the bar into the classification of "bar." At step 320, the defect detector module crops the image to a region surrounding the first feature. For example, the classification engine may crop the image to a region surrounding the bar of a rail joint.

[0053] At step 325, the defect detector module classifies one or more second features from the image into one or more second classifications using one or more CNNs. For example, one or more CNNs may be trained to detect second classifications such as "bolt head," "square nut," "hex nut," "round nut," "hole," and "break," and the classification engine of the defect detector module may use one or more CNNs to classify the break second feature into the classification of "break."

[0054] At step 330, the defect detector module analyzes the one or more second classifications. At step 335, the defect detector module determines whether the image includes a defect (e.g., defect 156 of FIG. 1 ) based on analyzing the one or more second classifications. For example, a defect detector engine of the defect detector module (e.g., defect detector engine 134 of FIG. 1 ) may determine that the image of the physical object includes a defect (e.g., defect 156 of FIG. 1 ) if the one or more second classifications are a defect.

[0055] If the defect detector module determines that the image does not include defects based on the one or more second classifications, method 300 proceeds from step 335 to step 345, where a reporting engine (e.g., reporting engine 136 of FIG. 1) generates a report indicating that the physical object is free of defects. Method 300 then moves from step 345 to step 365, where method 300 ends.

[0056] If the defect detector module determines that the image includes a defect based on the one or more second classifications, method 300 proceeds from step 335 to step 340. At step 340, the defect detector module determines whether the defect is part of a physical object. For example, the defect detector engine may determine that the physical object includes a defect if the location of the defect is part of the physical object of the image. As another example, the defect detector engine may determine that the physical object does not include a defect if the location of the defect is outside of the physical object of the image (e.g., part of a different physical object in the same image). If the defect detector module determines that the defect is not part of the physical object, method 300 proceeds from step 340 to step 345, where the reporting engine generates a report indicating that the physical object is free of defects. Method 300 then moves from step 345 to step 365, where method 300 ends.

[0057] If the defect detector module determines that the defect is part of the physical object, method 300 proceeds from step 340 to step 350, where the defect detector module determines a location of the defect in the cropped image. In step 355, the defect detector module determines a geographic location (e.g., GPS location) of the physical object defect using the location of the defect in the image and sensor information obtained from an image capture system (e.g., image capture system 170 of FIG. 1). Method 300 then proceeds to step 360, where a reporting engine generates a report indicating that the physical object is defective. The report may also indicate the geographic location of the physical object defect. Method 300 then moves from step 360 to step 365, where method 300 ends.

[0058] Modifications, additions, or omissions may be made to the method 300 shown in FIG. 3. Method 300 may include more, fewer, or other steps. For example, method 300 may further include classifying a defect (e.g., a break) in a physical object (e.g., a rail joint bar) into a more specific defect (e.g., a center broken joint bar or a quarter broken joint bar) based on the location of the defect with respect to other features (e.g., bolts, holes, and discontinuities) of the physical object. Steps may be performed in parallel or in any suitable order. Although discussed as specific components completing steps of method 300, any suitable component may perform any step of method 300.

[0059] 4 illustrates an exemplary image capture system 170. The image capture system 170 captures an image of a physical object (e.g., image 152 of FIG. 1) and communicates the image to a defect detector module (e.g., defect detector module 120 of FIG. 1). The image capture system 170, or portions thereof, may include or be associated with any entity, such as a business, company (e.g., railroad company, transportation company, etc.), or government agency (e.g., transportation department, public safety department, etc.) that captures images. Elements of the system 100 may be implemented using any suitable combination of hardware, firmware, and software.

[0060] The image capture system 170 of FIG. 4 includes a vehicle 410, a rotational position encoder 420, a subframe 172, and a beam 174. The vehicle 410 is any machine to which the beam 174 may be connected. The vehicle 410 may have an engine and / or wheels. The vehicle 410 may be an automobile, a locomotive, a truck, a bus, an aircraft, or any other machine suitable for movement. The vehicle 410 may operate at any speed that allows one or more components of the beam 174 (e.g., sensors, cameras, etc.) to capture images. For example, the vehicle 410 may be a rail car traveling 70 miles per hour.

[0061] The rotational position encoder 420 of the image capture system 170 is a wheel encoder or other timing device used to measure axle rotations. The rotational position encoder 420 may measure the number of times an axle rotates. The rotational position encoder 420 may be attached to an axle of the vehicle 410. The rotational position encoder 420 may be physically and / or logically connected to one or more components of the image capture system 170. For example, the rotational position encoder 420 may be physically and / or logically connected to one or more cameras and / or sensors of the module 176. As another example, the rotational position encoder 420 may be physically and / or logically connected to the controller 184.

[0062] The rotational position encoder 420 may communicate with the cameras of the module 176 to ensure that the cameras capture images with the same perspective and proportions regardless of the speed at which the vehicle 410 is traveling. For example, the rotational position encoder 420 may be synchronized with multiple cameras of the beam 174 to ensure that all cameras capture images at the same time. As another example, the rotational position encoder 420 may be synchronized with the cameras of the beam 174 to ensure that a camera moving with the vehicle 410 at a first speed (e.g., 7 miles per hour) captures images with the same perspective and proportions as a camera moving with the vehicle 410 at a second speed (e.g., 70 miles per hour).

[0063] The subframe 172 of the image capture system 170 is an intermediate structure that connects the vehicle 410 to the beam 174. The subframe 172 engages the vehicle 410 at multiple locations. The subframe 172 may connect to the vehicle 410 and / or the beam 174 using one or more bolts 430, welds, and / or any other suitable connection. Slots 440 in the subframe 172 provide level and / or height adjustment of the beam 174. The slots 440 may be vertical and / or horizontal. The vertically oriented slots 440 in the subframe 172 provide height adjustment of the beam 174. The subframe 174 may be connected to the front end of the vehicle 410, the rear end of the vehicle 410, the side of the vehicle 410, or any other suitable location for connecting the vehicle 410 to the beam 174. The subframe 172 may be manufactured from metal (e.g., steel or aluminum), plastic, or any other material suitable for connecting the vehicle 410 and the beam 174. In certain embodiments, the subframe 172 may be omitted such that the beam 174 is directly attached to the vehicle 410.

[0064] The beam 174 of the image capture system 170 is a structure that contains and directs the components (e.g., cameras and sensors) used to capture images. In certain embodiments, the beam 174 operates similarly to a flatbed document scanner, except that the beam 174 is moving while capturing images 152 of a fixed physical object. The beam 174 engages with the subframe 172. For example, the beam 174 may be bolted to the subframe 172 with bolts 430. In the embodiment shown in FIG. 4, the beam 174 has three sections, including two end sections and a center section. The beam 174 has a gullwing configuration such that the center section bends inward toward the center of the beam 174. The gullwing configuration allows the image capture components (e.g., sensors, cameras, etc.) of the modules 176 within the beam 174 to be properly oriented with respect to the physical object being captured. In certain embodiments, the center section of the beam 174 may be omitted, and each end section is connected to the subframe 172. The beams 174 may be manufactured from metal (eg, steel or aluminum), plastic, or any other material suitable for housing the components of the beams 174 and for mounting the beams 174 to the subframe 172.

[0065] The beam 174 may include one or more openings 450. The openings 450 may provide for placement of the modules 176 within the beam 174. The openings may allow for installation, adjustment, and maintenance of the modules 176. Although the beam 174 is illustrated in FIG. 4 as having a particular size and shape, the beam 174 may have any size and shape suitable for housing and orienting its components (e.g., the modules 176). Other factors that may contribute to the design of the beam 174 include shock resistance, vibration resistance, weather resistance considerations, durability, ease of maintenance, calibration considerations, and ease of installation.

[0066] Beam 174 includes modules 176, lighting system 178, HVAC system 180, data system 182, and controller 184. Modules 176 house the components for capturing images. Each module 176 may include one or more sensors, cameras, etc. Modules 176 are positioned within beam 174 and support positioning and support of the image capture components within beam 174. Modules 176 are designed to allow for serviceability and adjustment.

[0067] In certain embodiments, each end section of the beam 174 houses one or more camera modules 176. For example, a first end section of the beam 174 may house a module 176 including two downward-facing cameras that take images of the sleeper and ballast areas of the rail. The first end section of the beam 174 may house two downward-facing cameras in a portion of the first end section that is substantially horizontal to the rail. A second end section of the beam 174 opposite the first end section may house two modules 176, each including two angled cameras that take images of both sides of the rail and rail fastening system. The second end section of the beam 174 may house four angled cameras in a portion of the second end section that is at an angle (e.g., a 45 degree angle) to the rail.

[0068] The modules 176 of the image capture system 170 may house various types of sensors depending on the detection and / or measurement requirements. The sensors housed by the modules 176 may include optical sensors (e.g., cameras for visible light (monochromatic and color), infrared, ultraviolet, and / or thermal), motion sensors (e.g., gyroscopes and accelerometers), light detection and ranging (LIDAR) sensors, hyperspectral sensors, GPS sensors, etc. Optical sensors and lasers may be used together to perform laser triangulation to measure deflection or profile. LIDAR sensors may be used to generate three-dimensional (3D) point cloud data. Hyperspectral sensors may be used for specific wavelength responses. An exemplary module 176 is described in FIG. 5 below.

[0069] The lighting system 178 of the image capture system 170 provides external lighting for capturing images. The lighting system 178 provides lighting for capturing images during the day and in the dark. The lighting system 178 may include mechanisms for providing sufficient lighting intensity and properly orienting the lighting to capture images of stationary physical objects while the lighting system 178 moves at a predetermined speed (e.g., 70 miles per hour). The lighting system 178 may include any type of lighting suitable for providing suitable lighting for image capture. The lighting system 178 may include light emitting diode (LED) lighting (e.g., white LED lighting, off-road racing LED lighting, etc.), a light bar (e.g., off-road racing LED light bar), supplemental lighting (e.g., LED supplemental lighting), infrared lighting, combinations thereof, or other suitable types of lighting. The lighting system 178 may include one or more components that provide laser lighting, infrared lighting, ultraviolet lighting, or any other suitable type of lighting required by one or more components of the module 176 (e.g., sensors, cameras, etc.).

[0070] The HYAC system 180 of the image capture system 170 provides heating, ventilation, and / or air conditioning to the beam 174 of the image capture system 170. The HYAC system 180 may regulate the environmental conditions (e.g., interior temperature, humidity, etc.) of the beam 174. The HYAC system 180 may monitor the environmental conditions of the beam 174 to ensure that the operational requirements of the modules 176 of the beam 174 are met. For example, the HYAC system 180 may provide cooling to the beam 174 to ensure an enhanced environment for the modules 176 (e.g., cameras and sensors) to operate in hot weather. The HYAC system 180 may be a conventional HYAC system including one or more of a condenser, an evaporator, a compressor, an expansion valve, a belt, a hose, a refrigerant, and the like.

[0071] The HYAC system 180 may provide cooling to the beam 174 via an air-powered vortex generator. Dry, filtered, and conditioned air may be applied to the vortex generator from a main reservoir system of the vehicle 410 (e.g., a locomotive). The vortex generator may apply cooled compressed air to the beam 174. The cooled air may be routed to each module 176 (e.g., one or more sensors) for direct application. The vortex generator may also function as a humidity regulator. For example, the vortex generator may regulate humidity to about 50%. The compressed air may provide a slight positive pressure to the beam 174, which may prevent external dust and / or debris from entering the beam 174 through small holes.

[0072] The data system 182 of the image capture system 170 connects to and directs all data (e.g., sensor data) received from one or more components of the image capture system 170 to one or more computers, one or more controllers (e.g., controller 184), and / or one or more storage devices. The data system 182 may include one or more data cables for communicating data to the controllers and / or storage devices. The data cables of the data system 182 may include internal cables that are present inside the beam 174. The data cables of the data system 182 may include external cables that are present outside the beam 174. The internal and external cables may be joined by weatherproof connectors located in the walls of the beam 174. The one or more external cables may be routed to one or more computers, one or more controllers, and / or one or more storage devices. The data cables of the data system 182 provide a path for data inflow and / or outflow. For example, the data cables of the data system 182 may trigger a signal inflow from the controller 184. As another example, the data system 182 may wirelessly communicate data to the controller 184 and / or storage device using any suitable radio or cellular communication protocol.

[0073] The controller 184 of the image capture system 170 represents any suitable computing component that may be used to process information for the image capture system 170. The controller 184 may coordinate one or more components of the image capture system 170. The controller 184 may receive data from the modules 176, the lighting system 179, the HVAC system 180, the data system 182, the vehicle 410, and / or the rotational position encoder 420. The controller 184 may monitor the inputs and / or outputs of the modules 176, the lighting system 178, the HVAC system 180, and the rotational position encoder 420. The controller 184 may include communications capabilities that allow a user (e.g., a technician) to directly couple to the image capture system 170. For example, the controller 184 may be part of a computer (e.g., a laptop), and the user may access the controller 184 through an interface of the computer (e.g., a screen, a graphic user interface (GUI), or a panel). Controller 184 may communicate with one or more components of image capture system 170 over a network (e.g., network 110 of FIG. 1). For example, controller 184 may communicate with one or more components of image capture system 170 over a LAN and / or from a remote terminal over a WAN, such as a cellular or wireless network. Controller 184 may be located within vehicle 410.

[0074] The controller 184 may initiate adjustments to one or more components of the image capture system 170. The adjustments may be automatically initiated by the controller 184 in response to one or more conditions. For example, the controller 184 may instruct the HVAC system 180 to provide cooling to the beam 174 when the temperature within the beam 174 exceeds a predetermined value. The adjustments may be initiated locally (e.g., in the vehicle 410) by a user of the controller 184 (e.g., a technician using a computer including the controller 184). The adjustments may be initiated remotely (e.g., via a cellular or wireless link) by the controller 184. The adjustments may include adjustments to the lighting system 178 (e.g., lighting direction, lighting intensity, etc.), adjustments to the HVAC system 180 (e.g., temperature, humidity, etc.), adjustments to camera orientation, etc.

[0075] The controller 184 of the image capture system 170 may monitor the power status of one or more components of the image capture system 170. The controller 184 may provide power to one or more components of the image capture system 170 (e.g., sensors of the module 176). The controller 184 may use non-data functions such as remote reset. The controller 184 may receive a trigger signal from the rotational position encoder 420 and distribute the signal to a controller of the module 176 (e.g., a sensor controller). The controller of the module 176 may use the control system signal to actuate one or more components of the module 176 (e.g., sensors) in real time. The image capture system 170 may include one or more motion detection sensors, such as a gyroscope and an accelerometer, that provide a trigger signal compensation coefficient to account for vehicle motion caused by external features (e.g., uneven tracks).

[0076] In certain embodiments, image capture system 170 may receive the speed of vehicle 410. Controller 184 may receive the speed (e.g., a GPS speed signal) from a GPS receiver, a radar speed measurement system, a laser speed measurement system, or any other suitable component operable to measure the speed of vehicle 410.

[0077] 4 illustrates a particular arrangement of the vehicle 410, rotational position encoder 420, subframe 172, beam 174, module 176, lighting system 178, HVAC system 180, data system 182, and controller 184, this disclosure contemplates any suitable arrangement of the vehicle 410, rotational position encoder 420, subframe 172, beam 174, module 176, lighting system 178, HVAC system 180, data system 182, and controller 184. For example, lighting system 170 may be located external to beam 174. The vehicle 410, rotational position encoder 420, subframe 172, beam 174, module 176, lighting system 178, HVAC system 180, data system 182, and controller 184 may be physically or logically co-located in whole or in part with one another.

[0078] 4 illustrates a particular number of vehicles 410, rotational position encoders 420, subframes 172, beams 174, modules 176, lighting systems 178, HVAC systems 180, data systems 182, and controllers 184, this disclosure contemplates any suitable number of vehicles 410, rotational position encoders 420, subframes 172, beams 174, modules 176, lighting systems 178, HVAC systems 180, data systems 182, and controllers 180. For example, image capture system 170 may include a first beam 174 at a front end of vehicle 410 and a second beam 174 at a rear end of vehicle 410. As another example, image capture system 178 may include multiple controllers 184 (e.g., controllers and sub-controllers). One or more components of image capture system 170 may be implemented using one or more components of the computer system of FIG.

[0079] 5 illustrates an exemplary module 176 that may be used by the image capture system 170 of FIG. 4. The module 176 includes a camera 510, a lens 520, a top plate 530, a base plate 540, a cover plate 550, a cleaning device 560, a bolt 570, an opening 580, and an air compressor 585. The camera 510 is any device that captures an image of a physical object (e.g., image 152 of FIG. 1). For example, the camera 510 may capture an image of a rail component (e.g., a rail joint, a switch, a frog, a fastener, ballast, a rail head, and / or a rail sleeper). In a particular embodiment, the camera 510 is a sensor.

[0080] One or more cameras 510 may take images of one or more physical objects from different angles. For example, one or more cameras 510 may take images of the rails on both sides of a rail system at any given location. Each beam (e.g., beam 174 of FIG. 4) may include multiple cameras 510. A beam may include a first camera 510 pointed straight down to take an overhead image of the physical object. A beam may include a second camera 510 pointed downward and outward to take an angled image of the physical object.

[0081] The camera 510 may be a line scan camera. A line scan camera includes one row of pixels. The camera 510 may be a dual line scan camera. A dual line scan camera includes two rows of pixels that can be captured and / or processed simultaneously. As the camera 510 moves over the physical object, the camera 510 may capture images such that a complete image of the physical object can be reconstructed row by row in software. The camera 510 may have a capture rate of up to 140 kilohertz. The camera 510 may have a resolution and optics to detect physical objects at least 1 / 16 inch in size. In an alternative embodiment, the camera 510 may be an area scan camera.

[0082] The camera 510 includes a lens 520 that focuses and directs incident light onto a sensor of the camera 510. The lens 520 may be a piece of glass or other transparent material. The lens 520 may be made of any suitable material (e.g., steel, aluminum, glass, plastic, or a combination thereof).

[0083] The top plate 530 and the base plate 540 are structural elements used to position, support, and / or stabilize one or more components of the module 176 (e.g., the camera 510 or the sensor). The top plate 530 and the base plate 540 may be manufactured of any suitable material (e.g., steel, aluminum, plastic, glass, etc.). The top plate 530 may be connected to the base plate 540 with one or more bolts 570. The bolts 570 (e.g., jack bolts) may be used to change the pitch and / or roll direction of the camera 510. For example, the bolts 570 may be used to change the effective height between the top plate 530 and the base plate 540. The top plate 530 and / or the base plate 540 may be adjusted to reduce vibration and / or shock of the module 176. The top plate 530 and / or the base plate 540 may include resistive heating elements to provide a warm environment for the camera 510 and the lens 520 to operate during cooler weather periods.

[0084] The cover plate 550 is a plate that covers the base plate 540. The cover plate 550 may be manufactured of any suitable material (e.g., glass, steel, aluminum, etc.). The cover plate 550 includes an aperture 580. The aperture 580 may act as an opening through which the lens of the camera 510 sees physical objects. The aperture 580 allows transmission of signals detected from the environment to reach the sensor of the camera 510. The aperture 580 may be of any size (e.g., oval, rectangular, etc.) suitable to accommodate the view of the camera 510. The lens 520 of the camera 510 and / or the air compressor 585 may be located directly above the aperture 580. The air compressor 585 may provide cooling to the beam 174 via an air-powered vortex generator as described in FIG. 4 above.

[0085] The cleaning device 560 is any device that protects the lens 520 and / or the sensor of the camera 510 from the environment. The cleaning device 560 removes dust, small debris, water, and / or other items that may obstruct the view through the opening 580. The cleaning device 560 minimizes or eliminates the disturbance of signals transmitted by the components of the module 176 (e.g., the camera or the sensor). The cleaning device 560 may be provided between the cover plate 550 and the base plate 540. In certain embodiments, the cleaning device 560 is physically connected to the cover plate 550 and / or the base plate 540. The cleaning device 560 may be manufactured of any suitable material (e.g., glass, steel, aluminum, etc.). The cleaning device 560 may be disposed on an exterior surface of the module 176 (e.g., the bottom surface of the module 176).

[0086] The cleaning device 560 may use any suitable method to clean the lens 520 of the camera 510. For example, the cleaning device 560 may include a cleaning agent that emits compressed air, compressed gas, or a cleaning liquid. The cleaning device 560 may include a wiper blade, a brush, or any other suitable method to clean the lens 520. In certain embodiments, the cleaning agent is a compressor (e.g., air compressor 585) that emits compressed air or compressed gas. The compressed air or compressed gas is discharged through an orifice designed to exploit the Coanda effect, which allows ambient air to enter the primary flow to amplify the amount of air displaced across the lens 520 (see notation 590 for air flow). In certain embodiments, the cleaning device 560 may be part of an HVAC system (e.g., HVAC system 180 of FIG. 4).

[0087] Although FIG. 5 shows a particular arrangement of the camera 510, lens 520, top plate 530, base plate 540, cover plate 550, cleaning device 560, bolt 570, opening 580, and air compressor 585, this disclosure contemplates any suitable arrangement of the camera 510, lens 520, top plate 530, base plate 540, cover plate 550, cleaning device 560, bolt 570, opening 580, and air compressor 585. 5 illustrates a particular number of cameras 510, lenses 520, top plate 530, base plate 540, cover plate 550, cleaning device 560, bolts 570, openings 580, and air compressor 585, but this disclosure contemplates any suitable number of cameras 510, lenses 520, top plate 530, base plate 540, cover plate 550, cleaning device 560, bolts 570, openings 580, and air compressor 585. For example, module 176 may include multiple cameras 510. One or more components of module 176 may be implemented using one or more components of the computer system of FIG.

[0088] FIG. 6 illustrates an exemplary method 600 for tagging features 154 with labels 162. FIG. 6 includes a diagram of a rail 630 and a rail joint bar 632. The method 600 begins at step 610, where a classification engine (e.g., classification engine 132 of FIG. 1) identifies features 154 of the rail joint 132. The features 154 include a first square nut 634, a first bolt head 636, a first hole 638, a second hole 640, a second square nut 642, a second bolt head 636, a break 646, and a discontinuity 648. The break 646 indicates a defect in the rail joint bar 632 (e.g., defect 156 of FIG. 1). The discontinuity 648 indicates a separation between the rails 630. The classification engine may use one or more CNNs to identify (e.g., classify) the features 154 of the rail joint 132.

[0089] At step 620 of method 600, a labeling engine (e.g., labeling engine 138 of FIG. 1 ) tags features 154 of rail joint 132 with labels 162. As shown, the labeling engine tags feature 634 as a “square nut (0.84)”, feature 636 as a “bolt head (0.77)”, feature 638 as a “hole (0.89)”, feature 640 as a “hole (0.89)”, feature 642 as a “square nut (0.84)”, feature 644 as a “bolt head (0.77)”, feature 646 as a “break (0.83)”, and feature 648 as a “discontinuity (0.87)”. Each label includes a classification (e.g., classification 155 in FIG. 1 ) for each feature 154 and a probability that each classification correctly identifies each feature. For example, feature 636 tagged as “Bolt Head (0.77)” has a classification of “Bolt Head” and a probability of 0.77 (e.g., 77%) that “Bolt Head” correctly identifies feature 636. As another example, feature 638 tagged as “Hole (0.89)” has a classification of “Hole” and a probability of 0.89 (e.g., 89%) that “Hole” correctly identifies feature 638. As another example, feature 646 tagged as “Fracture (0.83)” has a classification of “Fracture” and a probability of 0.83 (e.g., 83%) that “Fracture” correctly identifies feature 646. As another example, feature 648 tagged as “Discontinuity (0.87)” has a classification of “Discontinuity” and a probability of 0.87 (e.g., 87%) that “Discontinuity” correctly identifies feature 648.

[0090] The classification and / or probability may be determined by a classification engine using one or more machine learning networks (e.g., CNN) and / or algorithms. In the illustrated embodiment, the position of each bounding box indicates the position of each feature 154 relative to the other features 154 and relative to the rail joint bar 632. Each bounding box may be any suitable quadrilateral shape (i.e., square, rectangle, etc.). An object detection model may be used to output bounding boxes of the detected classifications and labels 162. Although the embodiment illustrated in FIG. 6 depicts the features 154 with bounding boxes, the features 154 may be represented by any suitable shapes and / or characters.

[0091] Method 600 may be used to further classify the defect. In FIG. 6, the defect is shown as a "break". One or more machine learning algorithms (e.g., CNN 166 in FIG. 1) may be trained to recognize the illustrated classifications of square nuts, bolt heads, breaks, holes, discontinuities, and bars. Although one or more CNNs are trained to visually distinguish between square nuts and bolt heads, the visual representation of the two classifications is equivalent to the physical representation of the presence of a bolt. A defect detector engine (e.g., defect detector engine 134 in FIG. 1) may use one or more algorithms to logically reduce the classifications of square nuts and bolt heads to a single classification of bolts. The defect detector engine may use one or more algorithms to confirm the relative positions (e.g., position 158 in FIG. 1) of each of the detected objects (i.e., bolts, breaks, holes, and discontinuities).

[0092] The defect detector engine may use one or more algorithms to determine whether the break detected in the joint bar is a center break or a quarter break defect using the relative location. In the illustrated embodiment, the bounding box detected for the break is located between the two middle bolts (e.g., "Bolt Head (0.77)" and "Square Nut (0.84)"). The location of the break between the two middle bolts allows the defect detector engine to determine that the break is a center broken joint bar defect, indicating that the break is very close to the discontinuity. In an alternative embodiment, the break may be located outside the two middle bolts, and the defect detector engine may determine that it is a quarter broken joint bar due to the break location outside the two middle bolts.

[0093] Modifications, additions, or omissions may be made to the method 600 illustrated in FIG. 6. Method 600 may include more, fewer, or other steps. For example, as discussed above, method 600 may further include classifying a defect (e.g., a break) in a physical object (e.g., a rail joint bar) into a more specific defect (e.g., a center broken joint bar or a quarter broken joint bar) based on the location of the defect with respect to other features of the physical object (e.g., bolts, holes, and discontinuities). Steps may be performed in parallel or in any suitable order. Although discussed as specific components completing steps of method 600, any suitable component may perform any step of method 600.

[0094] 7 illustrates an example computer system that may be used by the systems and methods described herein. For example, network 110, defect detector module 120, and image capture system 170 of FIG. 1 may include one or more interfaces 710, processing circuitry 720, memory 730, and / or other suitable elements. Interface 710 (e.g., interface 122 of FIG. 1) receives input, sends output, processes the input and / or output, and / or performs other suitable operations. Interface 710 may include hardware and / or software.

[0095] Processing circuitry 720 (e.g., processor 126 of FIG. 1) executes or manages the operation of the components. Processing circuitry 720 may include hardware and / or software. Examples of processing circuitry include one or more computers, one or more microprocessors, one or more applications, etc. In certain embodiments, processing circuitry 720 executes logic (e.g., instructions) to perform actions (e.g., operations) such as generating output from input. The logic executed by processing circuitry 720 may be encoded in one or more tangible, non-transitory computer-readable media (e.g., memory 730). For example, logic may include computer programs, software, computer-executable instructions, and / or instructions that may be executed by a computer. In certain embodiments, operations of the embodiments may be performed by one or more computer-readable media that store, embody, and / or encode a computer program, and / or have a computer program stored and / or encoded therein.

[0096] The memory 730 (or memory unit) stores information. The memory 730 (e.g., memory 124 of FIG. 1) may include one or more non-transitory, tangible, computer-readable, and / or computer-executable storage media. Examples of the memory 730 include computer memory (e.g., RAM or ROM), mass storage media (e.g., hard disks), removable storage media (e.g., compact discs (CDs) or digital video discs (DVDs)), databases, and / or network storage (e.g., servers), and / or other computer-readable media.

[0097] As used herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs) or application specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM drives, secure digital cards or drives, other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, as appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, as appropriate.

[0098] In this specification, "or" is inclusive and not exclusive, unless expressly indicated otherwise or the context indicates otherwise. Thus, in this specification, "A or B" means "A, B, or both" unless expressly indicated otherwise or the context indicates otherwise. Furthermore, "and" means jointly and severally, unless expressly indicated otherwise or the context indicates otherwise. Thus, in this specification, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or the context indicates otherwise.

[0099] The scope of the present disclosure includes all changes, substitutions, variations, modifications, and alterations to the exemplary embodiments described or illustrated herein that would be understood by a person skilled in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, although the present disclosure describes and illustrates each embodiment herein as including certain components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated herein that would be understood by a person skilled in the art. Furthermore, any reference in the appended claims to an apparatus, or system, or components thereof, that is suitably configured, arranged, and operative or operable to perform a particular function includes the system or components of the apparatus, regardless of whether the apparatus, system, or components are suitably configured, arranged, and operative or operable, or whether the particular function is activated or deactivated. Furthermore, although the present disclosure describes or illustrates certain embodiments as providing certain advantages, certain embodiments may provide none, some, or all of these advantages.

Claims

1. 1. A system for determining defects in a physical object, comprising: an image capture system including a subframe, a beam, one or more sensors, a lighting system, a data system, and one or more controllers; and a defect detector module including: an image acquisition engine configured to acquire one or more images of one or more physical objects; a classification engine configured to classify one or more features from the images into one or more classifications; a defect detector engine configured to analyze the one or more classifications to identify one or more defects; and a reporting engine configured to generate a report indicating whether the physical object is defective based on the defect detector engine analysis.

2. The system of claim 1 , wherein the classification engine analyzes the images via one or more machine learning algorithms.

3. The system of claim 2 , wherein each machine learning algorithm is trained on labeled image data to recognize the one or more features from the one or more images.

4. The system of claim 2 , wherein the one or more machine learning algorithms are trained to recognize particular physical objects viewed by one or more components of the image capture system.

5. The system of claim 2 , wherein the one or more machine learning algorithms output the one or more classifications that explain the one or more features of the one or more images within a level of certainty.

6. The system of claim 5 , wherein the level of certainty can vary depending on the number of categories, the desired accuracy, and the availability of sample training data for each category, etc.

7. The system of claim 5 , wherein the level of certainty can be expressed as a probability.

8. The system of claim 1 , wherein the first classification indicates a first characteristic of the rail joint.

9. The system of claim 1 , wherein a second classification indicates one or more second characteristics associated with the rail joint.

10. The system of claim 5 , wherein the one or more classifications include one or more rail defects.

11. The system of claim 1 , wherein the defect detector engine generates additional information about the defect based on a relative position of the defect.

12. 1. A defect detector module comprising: an interface configured to transmit and receive data; an image acquisition engine configured to acquire one or more images of one or more physical objects; a classification engine configured to classify one or more features from the images into one or more classifications; a defect detector engine configured to analyze the one or more classifications to identify one or more defects; a reporting engine configured to generate a report indicating whether the physical object is defective or not based on the defect detector engine analysis; and one or more processors configured to perform operations of the defect detector module.

13. The system of claim 12 , wherein the classification engine analyzes the images via one or more machine learning algorithms.

14. 14. The system of claim 13, wherein each machine learning algorithm is trained on labeled image data to recognize the one or more features from the one or more images.

15. The system of claim 13 , wherein the one or more machine learning algorithms are trained to recognize particular physical objects viewed by one or more components of the image capture system.

16. 14. The system of claim 13, wherein the one or more machine learning algorithms output the one or more classifications that explain the one or more features of the one or more images within a level of certainty.

17. 17. The system of claim 16, wherein the level of certainty can vary depending on the number of categories, the desired accuracy, and the availability of sample training data for each category, etc.

18. The system of claim 12 , wherein the first classification indicates a first characteristic of the rail joint.

19. The system of claim 12 , wherein a second classification indicates one or more second characteristics associated with the rail joint.

20. The system of claim 12 , wherein the defect detector engine generates additional information about the defect based on a relative position of the defect.

Images