System for determining defects of physical object and corresponding defect detector module

Abstract

A system for determining defects in a physical object and a corresponding defect detector module. The system may include an image capture system comprising a subframe, a light beam, one or more sensors, an illumination system, a data system, and one or more controllers; and a defect detector module comprising: an image collection engine configured to collect one or more images of the physical object from the image capture system; a category engine configured to classify one or more features from the image into one or more categories; a defect detector engine configured to analyze the one or more categories to identify one or more defects; and a reporting engine configured to generate a report indicating whether the physical object is defective based on the defect detector engine analysis; wherein the category engine analyzes the image via a machine learning algorithm; and wherein the machine learning algorithm outputs one or more categories that describe features of the image within a deterministic level.

Description

[0001] This application is a divisional application of an invention patent application with an application date of November 19, 2019, application number 201980072375X (PCT application number PCT / US2019 / 062145), and name “System and method for determining defects of physical objects”. Technical Field

[0002] The present invention relates generally to determining defects and, more particularly, to determining defects in physical objects. Background Art

[0003] Physical objects are used in various industries to achieve business goals. For example, railcars transport freight on tracks. Railcars and tracks comprise a combination of many physical objects. Over time, physical objects may wear, break, or otherwise become defective, and the defect may require repair to ensure the continued safe operation of the railcars and tracks. Typically, technicians manually inspect physical objects such as railroad components to identify defects. Summary of the invention

[0004] According to an embodiment, a method includes the steps of receiving an image of a physical object by a defect detector module, and classifying, by the defect detector module, one or more first features from the image of the physical object into one or more first classifications using one or more machine learning algorithms. The method also includes the steps of analyzing, by the defect detector module, the one or more first classifications, and determining, by the defect detector module, that the physical object includes a defect based on analyzing the one or more first classifications.

[0005] According to another embodiment, a system includes one or more processors and a memory storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations, the operations including the following steps: receiving an image of a physical object through a defect detector module, and classifying one or more first features from the image of the physical object into one or more first classifications using one or more machine learning algorithms through the defect detector module. The operations also include the following steps: analyzing the one or more first classifications through the defect detector module, and determining through the defect detector module that the physical object includes a defect based on analyzing the one or more first classifications.

[0006] According to another embodiment, one or more computer-readable storage media, which include instructions that, when executed by a processor, cause the processor to perform operations, the operations including the steps of: receiving, by a defect detector module, an image of a physical object, and classifying, by the defect detector module, one or more first features from the image of the physical object into one or more first classifications using one or more machine learning algorithms. The operations also include the steps of: analyzing, by the defect detector module, the one or more first classifications, and determining, by the defect detector module, that the physical object includes a defect based on analyzing the one or more first classifications.

[0007] Technical advantages of certain embodiments of the present invention may include one or more of the following. The disclosed image capture system automatically captures images of physical objects, which eliminates or reduces manual data collection and manpower, thereby saving time and money. The image capture system can capture images of physical objects such as rail components from angles that may not be identifiable through manual inspection, which can improve the accuracy of detecting defects in physical objects. The disclosed defect detector module can use machine learning algorithms to automatically detect defects in physical objects, which can eliminate the manual labor of scanning and marking images of potential defects. The systems and methods described in the present invention can be generalized to different transportation infrastructures, including railways, roads, and waterways.

[0008] Other technical advantages will be apparent to those skilled in the art from the following drawings, descriptions and claims. Moreover, although specific advantages have been listed above, various embodiments may include all, some or none of the listed advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to help understand the present invention, reference is now made to the following description in conjunction with the accompanying drawings, in which:

[0010] Figure 1 An exemplary system for determining defects in a physical object is shown;

[0011] Figure 2 An exemplary method for determining a defect of a physical object is shown;

[0012] Figure 3 Another exemplary method for determining a defect in a physical object is shown;

[0013] Figure 4 An exemplary image capture system is shown;

[0014] Figure 5 It shows that Figure 4 Example modules for use with an image capture system;

[0015] Figure 6 An exemplary method of marking a feature using a tag is shown; and

[0016] Figure 7 An exemplary computer system that may be used by the systems and methods described herein is shown. DETAILED DESCRIPTION

[0017] Physical objects such as railroad components are used in a variety of industries to achieve business goals. Over time, physical objects may break or develop defects that may require repair to ensure continued and safe operation of the railcar. Typically, technicians will manually inspect physical objects such as railroad components to identify defects.

[0018] Condition and defect analysis algorithms can utilize one or more traditional computer vision techniques (e.g., region of interest analysis, filtering, thresholding, blob techniques, edge analysis, contour extraction, histogram analysis, etc.) and common assets and / or defect heuristics to automatically detect various conditions. While these techniques may be effective in very specific and consistent situations, they are often often brittle in real-world situations and may require a significant amount of time to maintain, correct, and enhance.

[0019] The use of machine learning algorithms for image classification and segmentation tasks involves a radically different approach to image analysis than traditional computational methods. Machine learning algorithms (e.g., neural networks) use a series of trained network layers, images or parts of images, through which they predict the contents of the image, rather than using algorithms to filter, distort, group, segment, and compute on a matrix of pixels.

[0020] Embodiments of the present invention use deep machine learning to capture and analyze visual images. The disclosed systems and methods identify physical objects (e.g., track components) observed from a specific hardware configuration on a moving vehicle (e.g., a vehicle moving on a track, a vehicle moving along a track, etc.) and interpret the visual data based on the following information: the amount, type, and accuracy of labeled training data, which is continuously captured, merged, and weighted. The disclosed systems and methods perform interpretations of physical objects using trained models on captured image data to obtain higher accuracy and higher levels of analysis of various captured physical objects. Machine learning algorithms are custom trained for each hardware configuration to achieve optimal continuous capture and analysis of various physical objects. These algorithms are used to classify certain features based on images of physical objects. The classification is then used to determine defects in the physical objects.

[0021] Figures 1 to 7Exemplary systems and methods for determining defects in a physical object are shown. Figure 1 An exemplary system for determining defects in a physical object is shown. Figure 2 and Figure 3 An exemplary method for determining defects in a physical object is shown. Figure 4 An exemplary image capture system is shown, Figure 5 It shows that Figure 4 Example modules for use with an image capture system. Figure 6 An exemplary method of marking features using tags is shown. Figure 7 An exemplary computer system that may be used by the systems and methods described herein is shown.

[0022] Figure 1 An exemplary system 100 for determining defects in a physical object is shown. Figure 1 The system 100 includes a network 110, a defect detector module 120, and an image capturing system 170. The system 100 or a portion thereof may be associated with an entity, which may include any entity, such as a business, a company (e.g., a railroad company, a transportation company, etc.), or a government agency (e.g., a department of transportation, a public safety department, etc.) that may determine defects of a physical object. The elements of the system 100 may be implemented using any suitable combination of hardware, firmware, and software.

[0023] The network 110 may be any type of network that facilitates communication between components of the system 100. The network 110 may connect the defect detector module 120 and the image capture system 170 of the system 100. Although the present invention shows the network 110 as a particular type of network, the present invention contemplates any suitable network. One or more portions of the network 110 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a public switched telephone network (PSTN), a cellular telephone network, a 3G network, a 4G network, a 5G network, a long term evolution (LTE) cellular network, a combination of two or more thereof, or other suitable types of networks. One or more portions of the network 110 may include one or more access (e.g., mobile access), core and / or edge networks. The network 110 may be any communication network, such as a private network, a public network, a connection through the Internet, a mobile network, a WI-FI network, a Bluetooth network, etc. The network 110 may include one or more network nodes. A network node is a connection point that can receive, create, store, and / or transmit data over the network 110. The network 110 may include cloud computing capabilities. One or more components of the system 100 may communicate over the network 110. For example, the defect detector module 120 may communicate over the network 110, including receiving information from the image capture system 170.

[0024] The defect detector module 120 of the system 100 represents any suitable computing component that can be used to determine defects 156 of a physical object. Defects 156 are defects that have the potential to impair the utility of a physical object. A physical object is an identifiable collection of matter. Physical objects can include transportation infrastructure components such as roads, railways, airways, waterways, canals, pipelines, and terminal components. Examples of railway components include joints, switches, frogs, rail heads, anchors, fasteners, gauge plates, ballasts, ties (e.g., concrete ties and wooden ties), etc.

[0025] The defect detector module 120 includes an interface 122, a memory 124, and a processor 126. The interface 122 of the defect detector module 120 represents any suitable computer component that can receive information from the network 110, send information through the network 110, perform appropriate information processing, and communicate with the network 110. Figure 1 The interface 122 represents any port or connection, real or virtual, including any suitable combination of hardware, firmware, and software, including protocol conversion and data processing capabilities, to allow communication over a LAN, WAN, or Figure 1The system 100 communicates with other communication systems to exchange information between components of the system 100.

[0026] The memory 124 of the defect detector module 120 permanently and / or temporarily stores received and transmitted information, as well as system software, control software, other software for the defect detector module 120, and various other information. The memory 124 may store information for execution by the processor 126. The memory 124 includes any one or combination of volatile or non-volatile local or remote devices suitable for storing information. The memory 124 may include random access memory (RAM), read-only memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage device or combination of these devices. The memory 124 may include any appropriate information used in the operation of the defect detector module 120. In addition, the memory 124 may be a component external to the defect detector module 120 (or may be partially external to the defect detector module 120). The memory 124 may be located in any location suitable for the memory 124 to communicate with the defect detector module 120. The memory 124 of the defect detector module 120 may store the image collection engine 130, the classification engine 132, the defect detector engine 134, the reporting engine 136, the labeling engine 138, and the training engine 140. As another example, the image collection engine 130, the classification engine 132, the defect detector engine 134, the reporting engine 136, the labeling engine 138, and the training engine 140 may be external to the memory 124 and / or the defect detector module 120.

[0027] The image collection engine 130 of the defect detector module 120 is an application for collecting images 152 of physical objects. The image collection engine 130 may receive one or more images 152 of one or more physical objects from the image capture system 170 via the network 110. The image collection engine 130 may receive the images 152 of the physical objects in real time or near real time as the physical objects are captured by the image capture system 170. The image collection engine 130 may receive the images 152 of the physical objects according to a schedule (e.g., every minute, every hour, every week, etc.). The image collection engine 130 may combine (e.g., stitch together) one or more images 152 to create a combined image 152. The image collection engine 130 may group the images 152 according to any suitable combination, such as by physical object, when the image 152 is captured by the image capture system 170, when the image 152 is received by the image collection engine 130, and / or when the image 152 is captured by a location 158 (e.g., a geographic location 158).

[0028] The classification engine 132 of the defect detector module 120 is an application that classifies features 154 from one or more images 152 of one or more physical objects into one or more classifications 155. Each feature 154 may be a feature of an image 152. For example, a feature 154 may include a railroad component such as a bolt head. A feature 154 may represent the physical object itself or one or more components of the physical object. For example, a first feature 154 may represent a physical object (e.g., a railroad joint), and one or more second features 154 may represent components of the railroad joint (e.g., a bolt head, a square nut, a hex nut, a round nut, a hole, etc.).

[0029] The classification engine 132 analyzes data from the images 152 by applying one or more machine learning algorithms. The machine learning algorithms may be associated with one or more neural networks (e.g., deep neural networks), one or more deep learning algorithms, one or more convolutional neural networks (CNNs), artificial intelligence (AI), any other suitable application, or any suitable combination of the foregoing. Each machine learning algorithm may be trained on labeled image data to identify one or more features 154 from one or more images 152. The one or more machine learning algorithms used by the classification engine 132 are trained to identify specific physical objects (e.g., rail components) observed by one or more components (e.g., one or more cameras) of the image capture system 170. For example, one or more CNNs 166 may utilize a series of convolutions, pooling, reduction, and fully connected layers through which an image 152 or a portion of an image 152 passes to determine a prediction about the content of the image 152. The one or more machine learning algorithms output one or more classifications 155 that describe one or more features 154 of the one or more images 152 within a certainty level. The certainty level may depend on the number of classifications 155, the desired accuracy, the availability of sample training data for each classification 155, etc. The certainty level may be expressed as a probability.

[0030] One or more classifications 155 are used to identify one or more features 154. For example, a first classification 155 may represent a first feature 154 of a railroad joint, and one or more second classifications 155 may represent one or more second features 154 associated with a railroad joint, such as a bolt head, a square nut, a hexagonal nut, a round nut, a hole, etc. The classifications 155 may include the location of the feature 154 within the image 152. For example, a classification 155 of "bolt head" may include the location of a bolt head in the image 152, which may be part of a physical object (e.g., a railroad joint).

[0031] One or more classifications 155 may include one or more defects 156. A defect includes any incomplete or defective attribute of a physical object. Defects 155 may include defects such as cracks, breaks, holes, mismatches, chips, wear, and the like. For example, defects 155 for railroad joints may include bolt missing, breaks, cracks, connection status (e.g., deteriorated connection), rail mismatch, rail end gap, rail end tilt, ballast status (e.g., deteriorated ballast), properly sized joint bars, and the like. As another example, defects 156 for railroad turnouts may include notches, breakpoints, missing fasteners (e.g., stock rails), tilted turnout poles, track running at turnout points, insufficient tie spacing, insufficient pole spacing, insufficient flange groove width, and the like. Defects 156 for railroad frogs may include notches, breakpoints, tread wear, broken gauge plates, tilted gauge plates, missing fasteners (e.g., missing frog bolts), missing plates, broken guardrails, and the like.

[0032] Defects 156 of rail heads may include rail corrugation, rail spalling, rail shelling, rail breakage, etc. Defects 156 of rail anchors and / or fasteners may include missing anchors and / or fasteners, inappropriate anchor patterns, missing clips, missing spikes, etc. Defects 156 of rail gauge plates may include tilted gauge plates, broken gauge plates, etc. Defects of rail ballast may include insufficient ballast, dirty ballast, dirty ballast, etc. Defects 156 of rail concrete ties may include broken ties, cracked ties, deteriorated ties, insufficient distance between ties, etc. Defects 156 of rail wood ties may include cut plates, cut wheels, insufficient distance between ties, rotten ties, hollow ties, etc. Other defects 156 may be identified or defined based on the captured physical objects.

[0033] The defect detector engine 134 of the defect detector module 120 is an application that determines whether the physical object includes one or more defects 156 based on analyzing the output of one or more machine learning algorithms (e.g., CNNs 166). For example, the defect detector engine 134 can analyze one or more classifications 155 output by the one or more machine learning algorithms. The defect detector engine 134 can apply one or more algorithms to the detection results (e.g., classifications 155) to determine (e.g., optically identify) one or more defects 156.

[0034] The defect detector engine 134 may analyze one or more classifications 155 to identify one or more defects 156. For example, if the classification 155 associated with the physical object is a defect 156 (e.g., a break), the defect detector engine 134 may determine that the physical object (e.g., a railroad joint) includes a defect 156. The defect detector engine 134 may determine a location 158 of the defect 156 on the physical object relative to other features 154 of the physical object to generate additional information about the defect 156. For example, the defect detector engine 134 may identify a break in a rebar based on the results of one or more machine learning algorithms, the algorithm including classifications 155 for "break" and "rebar." The defect detector engine 134 may determine that the rebar includes a bolt based on the classification 155 for "bolt." The defect detector engine 134 may determine the location of the break relative to the bolt. The defect detector engine 134 may then generate additional information about the break based on its relative location 158. For example, if the break is between two middle bolts, the defect detector engine 134 may classify the break as a "center break defect." As another example, if the fracture is outside of the two middle bolts, the defect detector engine 134 may classify the fracture as a "quarter fracture defect."

[0035] The defect detector engine 134 may determine a geographic location 158 of a defect 156 of a physical object. The defect detector engine 134 may determine the geographic location 158 using information received from the image collection engine 130. The image collection engine 130 may capture information (e.g., latitude and longitude) representing one or more geographic locations 158 associated with one or more images 152 and send the information to the defect detector engine 134. The defect detector engine 134 may convert the information received from the image collection engine 130 into any suitable format. For example, the defect detector engine 134 may convert the latitude and longitude received from the image collection engine 130 into a track type, a track number, a line segment, milestone information, etc. The defect detector engine 134 may use one or more algorithms (e.g., a closest point algorithm) to locate the geographic location 158 on a map (e.g., an existing map of a railroad track).

[0036] In certain embodiments, the defect detector engine 134 may detect welds (e.g., thermite welds) in the track images 152 and use the weld locations to identify milestones. Each image 152 may be "n" feet wide (e.g., 5 feet), representing "n" feet (e.g., 5 feet) of track, where n represents any suitable number. The defect detector engine 134 may determine the geographic location 158 of the physical object by determining a segment of the image 152 from the physical object relative to the milestone. The defect detector engine 134 may determine the geographic location 158 of the physical object by taking a shot of the physical object in the direction of ascending and / or descending milestones. For example, the defect detector engine 134 may determine that the physical object is a certain number of feet (e.g., 1000 feet) past the milepost 100, which may be represented (e.g., output) as "MP100.0+1000 feet." As another example, the defect detector engine 134 may determine that the physical object is a certain number of feet (e.g., 1000 feet) before the milepost 137, which may be represented (e.g., output) as "MP100.0-1000 feet." In some embodiments, the defect detector engine 134 may represent the geographic location 158 as a global positioning system (GPS) coordinate.

[0037] The defect detector engine 134 can determine the geographic location 158 based at least in part on the location 158 of the defect 156 relative to the physical object. For example, the defect detector engine 134 can determine the geographic location 158 of a rail joint break based in part on the break location 158 relative to the rail joint. The defect detector engine 134 can define the geographic location 158 based on coordinates (e.g., longitude and latitude) based at least in part on one or more sensors (e.g., position sensors) located within the system 100 (e.g., the image capture system 170).

[0038] The defect detector engine 134 can also determine an identification mark of the physical object. The identification mark of the physical object can include any characters suitable for identifying the physical object (e.g., numbers, letters, etc.). The identification mark of the physical object can identify the owner and / or manufacturer (e.g., a railroad manufacturer) of the physical object. The identification mark of the physical object can be used to track the source of the physical object. For example, the identification mark can include a batch code that allows the physical object to be traced back to a specific manufacturing batch. The defect detector engine 134 can determine the identification mark of the physical object based on the geographic location 158 of the physical object and / or by analyzing one or more images 152 of the physical object.

[0039] The reporting engine 136 of the defect detector module 120 is an application that generates one or more reports 160. If the defect detector engine 134 determines that the physical object does not include the defect 156, the reporting engine 136 can generate a report 160 indicating that the physical object does not have a defect. If the defect detector engine 134 determines that the physical object includes the defect 156, the reporting engine 136 can generate a report 160 indicating that the physical object has a defect. The report 160 can take any suitable form (e.g., written and / or verbal) to communicate the defect 156. The report 160 can include information associated with the physical object, features 154, classifications 155, locations 158, labels 162, etc. For example, the report 160 can include Figure 6 The report 160 may include one or more charts, tables, lists, graphs, and / or any other suitable format for conveying information. The report engine 136 may transmit the report 160 to one or more components of the system 100 (e.g., the image capture system 170) and / or a user of the system 100 (e.g., an administrator, a technician, etc.).

[0040] The tagging engine 138 of the defect detector module 120 is an application that generates one or more tags 162. The tags 162 provide visual information associated with features 154, classifications 155, defects 156, and / or locations 158. The tagging engine 138 tags the features 154 of one or more physical objects with the tags 162. Each tag 162 may include information associated with the physical object. Each tag 162 may include the feature 154, a probability, a size (e.g., diameter, area, etc.) associated with the feature 154, a classification 155, a location 158, etc. For example, the tag 162 may represent the feature 154 as "bolt head (0.77)" to describe the classification 155 of "bolt head" and the probability that the feature 154 is accurately identified as a bolt head is 0.77 (i.e., 77%).

[0041] The marking engine 138 may insert one or more portions of a marking 162 that represents a feature 154 of a physical object (e.g., a rail joint) on a diagram of the physical object. The marking engine 138 may insert one or more portions of the marking 162 into the diagram at a location 158 where the relevant feature 154 appears on the physical object. For example, for a feature 154 of a bolt head, the marking engine 138 may insert one or more portions of the marking 162 at a location 158 on the diagram of the physical object where the bolt head is located on the physical object. The marking 162 may represent the feature 154 having any suitable shape or character. For example, the marking 162 that represents the feature 154 of a bolt head may be a bounding box having a square or rectangular shape. As shown below Figure 6 As described above, defect detector engine 134 may use tags 162 to further categorize defects 156 .

[0042] The training engine 140 of the defect detector module 120 is an application that trains one or more machine learning algorithms (e.g., CNNs 166). The training engine 140 uses training data (e.g., training images 164) to train the machine learning algorithms, through which the weights can be adjusted to accurately identify physical objects. The machine learning algorithms can be trained on each specific camera angle of the image capture system 170 to provide the highest accuracy for each classification 155. The training engine 140 can train the one or more machine learning algorithms by collecting sample data (e.g., training images 164) representing each classification 155. The training engine 140 can label the sample data with labels 162 and use the labeled data to train one or more machine learning algorithms to identify each classification 155. The training engine 140 can use a subset of the labeled images to check the accuracy of each machine learning algorithm.

[0043] Training engine 140 may receive initial training data from an administrator of system 100. The amount and variety of training data (e.g., training images 164) used by one or more machine learning algorithms depends on the number of classes 155, the accuracy required, and the availability of sample data for each class 155.

[0044] The memory 124 may store a database 150. The database 150 may store certain types of information for the defect detector module 120. For example, the database 150 may store images 152, features 154, classifications 155, defects 156, locations 158, reports 160, tags 162, training images 164, and machine learning algorithms (e.g., CNNs 166). The database 150 may be any one or combination of volatile or non-volatile local or remote devices suitable for storing information. The database 150 may include RAM, ROM, magnetic storage devices, optical storage devices, or any other suitable information storage device or combination of these devices. The database 150 may be a component external to the defect detector module 120. The database 150 may be located in any location suitable for the database 150 to store information for the defect detector module 120. For example, the database 150 may be located in a cloud environment.

[0045] The processor 126 of the defect detector module 120 controls certain operations of the defect detector module 120 by processing information received from the interface 122 and the memory 124 or information otherwise accessed by the processor 126. The processor 126 is communicatively coupled to the interface 122 and the memory 124. The processor 126 may include any hardware and / or software for controlling and processing information. The processor 126 may be a programmable logic device, a microcontroller, a microprocessor, any suitable processing device, or any suitable combination of the above. In addition, the processor 126 may be a component external to the defect detector module 120. The processor 126 may be located at any location suitable for the processor 126 to communicate with the defect detector module 120. The processor 126 of the defect detector module 120 controls the operation of the image collection engine 130, the classification engine 132, the defect detector engine 134, the reporting engine 136, the labeling engine 138, and the training engine 140.

[0046] One or more components of the defect detector module 120 may operate in the cloud. The cloud may provide different services (e.g., applications, servers, and storage) to the defect detector module 120 via the network 110. The cloud may be implemented using any suitable combination of hardware, firmware, and software. For example, Figure 7 One or more components of a computer system to implement the cloud.

[0047] The image capture system 170 of system 100 represents a system for capturing images 152 of physical objects. The image capture system includes a subframe 172, a beam 174, one or more modules 176, a lighting system 178, a heating, ventilation, and air conditioning (HVAC) system 180, a data system 182, and one or more controllers 184. The components of the image capture system 180 may be connected (e.g., physically connected) to a vehicle (e.g., a locomotive). The image capture system 170 may capture one or more images 152 of a physical object while the vehicle is in motion relative to the physical object. For example, the image capture system 170 may be connected to a locomotive, and the image capture system 170 may capture images 152 of a rail joint while the locomotive is traveling along a railroad track. The operation of the image capture system 170 may be similar to a flatbed document scanner, except that the image capture system 170 captures images 152 of a stationary physical object while in motion. Below, in Figure 4 Image capture system 170 is described in more detail in .

[0048] although Figure 1A specific arrangement of the network 110, defect detector module 120, interface 122, memory 124, processor 126, image collection engine 130, classification engine 132, defect detector engine 134, reporting engine 136, labeling engine 138, training engine 140, database 150, and image capture system 170 is shown, and the present invention contemplates any suitable arrangement of the network 110, defect detector module 120, interface 122, memory 124, processor 126, image collection engine 130, classification engine 132, defect detector engine 134, reporting engine 136, labeling engine 138, training engine 140, database 150, and image capture system 170. The network 110, defect detector module 120, interface 122, memory 124, processor 126, image collection engine 130, classification engine 132, defect detector engine 134, reporting engine 136, labeling engine 138, training engine 140, database 150 and image capture system 170 may all or partly physically or logically co-exist with each other.

[0049] although Figure 1 While a particular number of networks 110, defect detector modules 120, interfaces 122, memories 124, processors 126, image collection engines 130, classification engines 132, defect detector engines 134, reporting engines 136, labeling engines 138, training engines 140, databases 150, and image capture systems 170 are shown, any suitable number of networks 110, defect detector modules 120, interfaces 122, memories 124, processors 126, image collection engines 130, classification engines 132, defect detector engines 134, reporting engines 136, labeling engines 138, training engines 140, databases 150, and image capture systems 170 are contemplated by the present invention. Figure 7 One or more components of a computer system of the invention may be used to implement one or more components of the defect detector module 120 and / or the image capture system 170.

[0050] although Figure 1 The system 100 is described as being used to determine defects 156 of physical objects, but one or more components of the system 100 may be applied to other embodiments. For example, one or more components of the defect detector module 120 and / or the image capture system 170 may be used for asset identification and / or inventory. For example, the classification engine 132 of the defect detector module 120 may be used to identify physical objects (e.g., railroad bars, switches, crossings, frogs, etc.) and record an inventory of the identified physical objects.

[0051] In operation, the image collection engine 130 of the defect detector module 120 receives an image 152 of a physical object (e.g., a rail joint) from an image capture system 170. The classification engine 132 of the defect detector module 120 classifies features 154 (e.g., bolts, breaks, discontinuities, and holes) from the image 152 of the physical object into classifications 155 (e.g., bolts, breaks, discontinuities, and holes) using one or more machine learning algorithms (e.g., CNNs 166). The defect detector engine 134 of the defect detector module 120 analyzes the classifications 155 and determines, based on the analysis, that the physical object includes a defect 156 (e.g., a break). The defect detector engine 134 uses the image 152 to determine a location 158 of the defect 156 relative to other features 154 of the physical object. The defect detector engine 134 determines a geographic location 158 of the defect 156 of the physical object based, at least in part, on the location 158 of the defect 156 relative to other features 154 of the physical object. The reporting engine 136 generates a report 160 indicating that the physical object includes the defect 156. The report 160 includes a geographic location 158 of the defect 156. The tagging engine 138 tags the one or more first features 154 of the image 152 with one or more tags 162. The tags 162 include a tag 162 that represents a defect 156 (eg, a break) of the physical object.

[0052] therefore, Figure 1 The system 100 reduces or eliminates the need for manual inspection by capturing an image 152 of the physical object, analyzing the image 152, classifying features 154 from the image 152 using one or more machine learning algorithms, and determining the defects 156 based on the classification 155.

[0053] Figure 2 An exemplary method 200 for determining defects of a physical object is shown. The method 200 begins at step 210. At step 220, a defect detector module (e.g., Figure 1 The defect detector module 120) from the image capture system (e.g., Figure 1 The image capture system 170 receives an image of a physical object (e.g., Figure 1 For example, the image collection engine of the defect detector module (e.g., Figure 1 The image collection engine 130 of the embodiment of the present invention can receive images of the rail joint from an image capture system. The image capture system can be connected to a component (e.g., a locomotive), and the image capture system can capture images when the component moves relative to a physical object.

[0054] At step 230, the defect detector module uses one or more machine learning algorithms (e.g., Figure 1 CNNs166) combine one or more features in an image (e.g., Figure 1 154) into one or more categories (e.g., Figure 1 For example, a classification engine (e.g., classification engine 132) of the defect detector module can use one or more CNNs to classify features such as square nuts, bolt heads, fractures, holes, discontinuities, and rebars into classifications that include square nuts, bolt heads, fractures, holes, discontinuities, and rebars, respectively. One or more algorithms can logically reduce the square nut and bolt head classifications to a single bolt classification, because the visual distinction between the square nut and the bolt head is equivalent to the physical representation of the bolt.

[0055] At step 240, the defect detector module analyzes one or more of the classifications. The defect detector engine (e.g., Figure 1 The defect detector engine 134 of the embodiment may analyze one or more classifications to identify defects in the physical object (e.g., Figure 1 The defect detector module determines whether the physical object is defective based on the classification at step 250. If one or more classifications include defects (e.g., fractures), the defect detector engine can determine whether the physical object is defective.

[0056] If the defect detector module determines based on the classification that the physical object is not defective, then the method 200 proceeds from step 250 to step 260, where a reporting engine (e.g., Figure 1 The method 200 then moves to step 280, where the method 200 ends. If the defect detector module determines that the physical object is defective based on the classification, the method 200 proceeds from step 250 to step 270, where the reporting engine generates a report indicating that the physical object is defective. The method 200 then moves to step 280, where the method 200 ends.

[0057] Can Figure 2 The method 200 described in the specification may be modified, added, or omitted. The method 200 may include more, fewer, or other steps. For example, the method 200 may include a training engine (e.g., Figure 1 The method 200 may include training one or more neural networks and / or algorithms (e.g., CNN) to correctly identify and classify features in an image. As another example, the method 200 may include training one or more neural networks and / or algorithms (e.g., CNN) to correctly identify and classify features in an image via a training engine 140 of a defect detector module (e.g., Figure 1 The tagging engine 138 of the embodiment generates one or more tags representing one or more features (e.g., Figure 1162). The steps may be performed in parallel or in any suitable order. Although discussed as specific components that perform the steps of method 200, any suitable component may perform any step of method 200.

[0058] Figure 3 An exemplary method 300 for determining defects of a physical object is shown. The method 300 begins at step 305. At step 310, a defect detector module (e.g., Figure 1 The image collection engine (e.g., Figure 1 The image collection engine 130) collects information from an image capture system (e.g., Figure 1 The image capture system 170 receives an image of a physical object (e.g., Figure 1 The image collection engine of the defect detector module can detect whether there is a rail joint in the image.

[0059] At step 315, the defect detector module extracts the first feature (e.g., Figure 1 154) is classified as the first category (e.g., Figure 1 155). For example, one or more CNNs may be trained to detect categories such as "bar", "discontinuity", and "end post", and the classification engine of the defect detector module (e.g., Figure 1 The classification engine 132) can use one or more CNNs to classify the first feature of the steel bar as a "steel bar" classification. At step 320, the defect detector module crops the image to the area around the first feature. For example, the classification engine can crop the image to the area surrounding the steel bar of the rail joint.

[0060] At step 325, the defect detector module classifies the one or more second features from the image into one or more second classifications using the one or more CNNs. For example, the one or more CNNs may be trained to detect second classifications such as "bolt head", "square nut", "hex nut", "round nut", "hole", and "broken", and the classification engine of the defect detector module may classify the second feature of the fracture into the "broken" classification using the one or more CNNs.

[0061] At step 330, the defect detector module analyzes the one or more second classifications. At step 335, the defect detector module determines whether the image includes a defect (e.g., Figure 1 For example, if one or more of the second classifications are defects, then the defect detector engine (e.g., Figure 1The defect detector engine 134 may determine that the image of the physical object includes a defect (e.g., Figure 1 Defects 156).

[0062] If the defect detector module determines based on the one or more second classifications that the image does not include a defect, the method 300 proceeds from step 335 to step 345, where a reporting engine (e.g., Figure 1 The reporting engine 136 of the method 300 generates a report indicating that the physical object has no defects. Subsequently, the method 300 moves from step 345 to step 365, wherein the method 300 ends.

[0063] If the defect detector module determines that the image includes a defect based on one or more second classifications, the method 300 proceeds from step 335 to step 340. At step 340, the defect detector module determines whether the defect is part of a physical object. For example, if the location of the defect is part of a physical object of the image, the defect detector engine can then determine that the physical object includes a defect. As another example, if the location of the defect is outside of the physical object of the image (e.g., part of a different physical object of the same image), the defect detector engine can determine that the physical object does not include a defect. If the defect detector module determines that the defect is not part of the physical object, the method 300 proceeds from step 340 to step 345, where a reporting engine generates a report indicating that the physical object does not have a defect. Subsequently, the method 300 moves from step 345 to step 365, where the method 300 ends.

[0064] If the defect detector module determines that the defect is part of the physical object, the method 300 proceeds from step 340 to step 350, where the defect detector module determines the location of the defect in the cropped image. In step 355, the defect detector module uses the image captured by the image capture system (e.g., Figure 1 The method 300 then proceeds to step 360, where a reporting engine generates a report indicating that the physical object has a defect. The report may also indicate the geographic location of the defect on the physical object. The method 300 then moves from step 360 to step 365, where the method 300 ends.

[0065] Can Figure 3The method 300 described in the foregoing may be modified, added to, or omitted. The method 300 may include more, fewer, or other steps. For example, the method 300 may include the following steps: further classifying a defect (e.g., a break) of a physical object (e.g., a rail dowel) into more specific defect objects (e.g., a center-broken dowel or a quarter-broken dowel) based on the location of the defect relative to other features of the physical object (e.g., bolts, holes, and discontinuities). The steps may be performed in parallel or in any suitable order. Although discussed as specific components that perform the steps of the method 300, any suitable component may perform any of the steps of the method 300.

[0066] Figure 4 An exemplary image capture system 170 is shown. The image capture system 170 captures images of physical objects (e.g., Figure 1 152), and transmits the image to a defect detector module (e.g., Figure 1 The image capture system 170 or a portion thereof may be associated with an entity, which may include any entity, such as a business, a company (e.g., a railroad company, a transportation company, etc.), or a government agency (e.g., a department of transportation, a department of public safety, etc.) that captures the image. The elements of the system 100 may be implemented using any suitable combination of hardware, firmware, and software.

[0067] Figure 4 The image capture system 170 includes a vehicle 410, a rotary position encoder 420, a subframe 172, and a beam 174. The vehicle 410 is any machine to which the beam 174 can be connected. The vehicle 410 can have an engine and / or wheels. The vehicle 410 can be a car, a locomotive, a truck, a bus, an airplane, or any other machine suitable for movement. The vehicle 410 can be operated at any speed that allows one or more components (e.g., sensors, cameras, etc.) of the beam 174 to capture images. For example, the vehicle 410 can be a rail vehicle that travels 70 miles per hour.

[0068] The rotary position encoder 420 of the image capture system 170 is a wheel encoder or other timing device for measuring the rotation of an axle. The rotary position encoder 420 can measure the number of times the axle rotates. The rotary position encoder 420 can be connected to an axle of the vehicle 410. The rotary position encoder 420 can be physically and / or logically connected to one or more components of the image capture system 170. For example, the rotary position encoder 420 can be physically and / or logically connected to one or more cameras and / or sensors of the module 176. As another example, the rotary position encoder 420 can be physically and / or logically connected to the controller 184.

[0069] The rotary position encoder 420 can communicate with the camera of the module 176 to ensure that the camera captures images of the same perspective and scale regardless of the speed at which the vehicle 410 is traveling. For example, the rotary position encoder 420 can be synchronized with multiple cameras of the beam 174 to ensure that all cameras capture images at the same time. As another example, the rotary position encoder 420 can be synchronized with the cameras of the beam 174 to ensure that a camera traveling with the vehicle 410 at a first speed (e.g., 7 mph) captures images with the same perspective and scale as a camera traveling with the vehicle 410 at a second speed (e.g., 70 mph).

[0070] The subframe 172 of the image capture system 170 is an intermediate structure that connects the vehicle 410 to the beam 174. The subframe 172 engages the vehicle 410 at multiple locations. The subframe 172 can be connected to the vehicle 410 and / or the beam 174 by one or more bolts 430, welds, and / or any other suitable couplings. The slots 440 of the subframe 172 provide horizontal and / or height adjustment for the beam 174. The slots 440 can be vertical and / or horizontal. The vertically oriented slots 440 of the subframe 172 provide height adjustment for the beam 174. The subframe 172 can be connected to the front end of the vehicle 410, the rear end of the vehicle 410, the side of the vehicle 410, or any other suitable location to connect the vehicle 410 to the beam 174. The subframe 172 can be made of metal (e.g., steel or aluminum), plastic, or any other material suitable for connecting the vehicle 410 to the beam 174. In some embodiments, the subframe 172 can be omitted so that the beam 174 is directly connected to the vehicle 410.

[0071] The beam 174 of the image capture system 170 is a structure that contains components (e.g., a camera and a sensor) for capturing an image and orienting it. In some embodiments, the operation of the beam 174 is similar to that of a flatbed document scanner, except that the beam 174 is in motion when capturing the image 152 of a stationary physical object. The beam 174 is coupled to the subframe 172. For example, the beam 174 can be bolted to the subframe 172 using bolts 430. Figure 4 As shown, the beam 174 has three parts, including two end portions and a center portion. The beam 174 has a gull-wing structure, so that the center portion bends inward toward the center of the beam 174. The gull-wing configuration allows the image capture components (e.g., sensors, cameras, etc.) of the module 176 within the beam 174 to be correctly oriented within the beam 174 relative to the physical object being captured. In some embodiments, the center portion of the beam 174 can be omitted and each end portion is connected to the sub-frame 172. The beam 174 can be made of metal (e.g., steel or aluminum), plastic, or any other material suitable for the housing components of the beam 174 and the material that secures the beam 174 to the sub-frame 172.

[0072] The beam 174 may include one or more openings 450. The openings 450 may be used to place the module 176 within the beam 174. The openings may be used for installation, adjustment, and maintenance of the module 176. Figure 4 The beam 174 is shown as having a particular size and shape, but the beam 174 can have any size and shape suitable for accommodating and positioning its components (e.g., module 176). Other factors that may contribute to the design of the beam 174 include impact resistance, vibration resistance, weather resistance, durability, ease of maintenance, calibration considerations, and ease of installation.

[0073] The beam 174 includes modules 176, a lighting system 178, an HVAC system 180, a data system 182, and a controller 184. The modules 176 house components for capturing images. Each module 176 may include one or more sensors, cameras, etc. The modules 176 are located within the beam 174 and help position and support the image capture components within the beam 174. The modules 176 are designed to allow for maintenance and adjustment.

[0074] In some embodiments, each end of the beam 174 accommodates one or more camera modules 176. For example, the first end of the beam 174 can accommodate a module 176 that includes two downward cameras that capture images of the tie-down area and the ballast area of ​​the track. The first end of the beam 174 can accommodate two downward cameras in a portion of the first end that is substantially horizontal to the track. The second end of the beam 174 opposite the first end can accommodate two modules 176, each module 176 including two angled cameras for capturing images of both sides of the track and the track fastening system. The second end of the beam 174 can accommodate four angled cameras in a portion of the second end that is at an angle (e.g., a 45 degree angle) to the track.

[0075] Module 176 of image capture system 170 may accommodate various types of sensors, depending on the sensing and / or measurement requirements. Sensors accommodated by module 176 may include optical sensors (e.g., cameras for visible light (monochrome and color), infrared, ultraviolet, and / or thermal), motion sensors (e.g., gyroscopes and accelerometers), light detection and ranging (LIDAR) sensors, hyperspectral sensors, GPS sensors, etc.). Optical sensors and lasers may be used together for laser triangulation to measure deflection or profile. LIDAR sensors may be used to generate three-dimensional (3D) point cloud data. Hyperspectral sensors may be used for specific wavelength responses. Below, in Figure 5 An exemplary module 176 is described in .

[0076] The lighting system 178 of the image capture system 170 provides external lighting for capturing images. The lighting system 178 provides lighting for capturing images during daytime and darkness. When the lighting system 178 travels at a predetermined speed (e.g., 70 miles per hour), the lighting system 178 can provide an illumination intensity sufficient to capture images of stationary physical objects. The lighting system 178 may include a mechanism for properly directing the lighting. The lighting system 178 may include any type of light suitable for providing sufficient illumination to capture images. The lighting system 178 may include a light emitting diode (LED) light (e.g., a white LED light, an off-road racing LED light, etc.), a light bar (e.g., an off-road racing LED light bar), an auxiliary light (e.g., an LED auxiliary light), infrared lighting, a combination of these, or any other appropriate type of light. The lighting system 178 may include one or more components that provide laser lighting, infrared lighting, ultraviolet lighting, or any other appropriate type of lighting required by one or more components of the module 176 (e.g., sensors, cameras, etc.).

[0077] The HVAC system 180 of the image capture system 170 provides heating, ventilation, and / or air conditioning for the beam 174 of the image capture system 170. The HVAC system 180 can adjust the environmental conditions (e.g., internal temperature, humidity, etc.) of the beam 174. The HVAC system 180 can monitor the environmental conditions of the beam 174 to ensure that the operating requirements of the modules 176 of the beam 174 are met. For example, the HVAC system 180 can provide cooling to the beam 174 to ensure a tempered environment for the modules 176 (e.g., cameras and sensors) to operate during hot weather. The HVAC system 180 can be a conventional HVAC system that includes one or more of a condenser, an evaporator, a compressor, an expansion valve, belts, hoses, a refrigerant, etc.

[0078] The HVAC system 180 can provide cooling to the beam 174 through aerodynamic vortex generators. Dry, filtered, and conditioned air from the main storage system of the vehicle 410 (e.g., a locomotive) can be applied to the vortex generators. The vortex generators can apply cooled compressed air to the beam 174. The cooled air can be delivered to each module 176 (e.g., one or more sensors) for direct application. The vortex generators can also be used as humidity regulators. For example, the vortex generators can adjust the humidity to approximately 50%. The compressed air can provide a slight positive pressure on the beam 174, which can prevent external dust and / or debris from entering the beam 174 through the small holes.

[0079] The data system 182 of the image capture system 170 connects and directs all data (e.g., sensor data) received from one or more components of the image capture system 170 to one or more computers, one or more controllers (e.g., controller 184), and / or one or more storage devices. The data system 182 may include one or more data cables for transmitting data to the controller and / or storage device. The data cable of the data system 182 may include an internal cable located within the beam 174. The data cable of the data system 182 may include an external cable located outside the beam 174. The internal and external cables may be connected by a weatherproof connector located on the wall of the beam 174. The one or more external cables may be routed to one or more computers, one or more controllers, and / or one or more storage devices. The data cable of the data system 182 provides a path for data to enter and / or leave. For example, the data cable of the data system 182 may trigger a signal from the controller 184 to enter. As another example, the data system 182 may wirelessly transmit data to the controller 184 and / or storage device using any suitable wireless or cellular communication protocol.

[0080] The controller 184 of the image capture system 170 represents any suitable computing component that can be used to process information for the image capture system 170. The controller 184 can coordinate one or more components of the image capture system 170. The controller 184 can receive data from the module 176, the lighting system 179, the HVAC system 180, the data system 182, the vehicle 410, and / or the rotary position encoder 420. The controller 184 can monitor the inputs and / or outputs of the module 176, the lighting system 178, the HVAC system 180, and the rotary position encoder 420. The controller 184 may include communication functionality that allows a user (e.g., a technician) to directly engage with the image capture system 170. For example, the controller 184 can be part of a computer (e.g., a laptop computer), and the user can access the controller 184 through an interface (e.g., a screen, a graphical user interface (GUI), or a panel) of the computer. The controller 184 can communicate with the computer via a network (e.g., Figure 1 The controller 184 may communicate with one or more components of the image capture system 170 via a network 110 of the vehicle 410. For example, the controller 184 may communicate with one or more components of the image capture system 170 via a LAN and / or from a remote terminal on a WAN such as a cellular or wireless network. The controller 184 may be located within the vehicle 410.

[0081] The controller 184 may initiate adjustments to one or more components of the image capture system 170. These adjustments may be automatically initiated by the controller 184 in response to one or more conditions. For example, when the temperature within the beam 174 exceeds a predetermined value, the controller 184 may instruct the HVAC system 180 to provide cooling to the beam 174. The adjustments may be initiated locally (e.g., within the vehicle 410) by a user of the controller 184 (e.g., a technician using a computer that includes the controller 184). These adjustments may be initiated remotely by the controller 184 (e.g., via a cellular or wireless link). The adjustments may include adjusting the lighting system 178 (e.g., lighting direction, lighting intensity, etc.), adjusting the HVAC system 180 (e.g., temperature, humidity, etc.), adjusting the camera direction, etc.

[0082] The controller 184 of the image capture system 170 can monitor the power status of one or more components of the image capture system 170. The controller 184 can provide power to one or more components of the image capture system 170 (e.g., sensors of module 176). The controller 184 can engage non-data features such as remote reset. The controller 184 can receive a trigger signal from the rotary position encoder 420 and distribute the signal to the controller of the module 176 (e.g., a sensor controller). The controller of the module 176 can then drive one or more components of the module 176 (e.g., sensors) in real time using the control system signal. The image capture system 170 may include one or more motion sensing sensors, such as gyroscopes and accelerometers, which provide a trigger signal compensation factor to account for vehicle motion caused by external features (e.g., rough track).

[0083] In certain embodiments, the image capture system 170 may receive the speed of the vehicle 410. The controller 184 may receive the speed of the vehicle 410 (e.g., a GPS speed signal) from a GPS receiver, a radar speed measurement system, a laser speed measurement system, or any other suitable component operable to measure the speed of the vehicle 410.

[0084] although Figure 4A particular arrangement of vehicle 410, rotary position encoder 420, subframe 172, beam 174, module 176, lighting system 178, HVAC system 180, data system 182, and controller 184 is shown, but the present disclosure contemplates any suitable arrangement of vehicle 410, rotary position encoder 420, subframe 172, beam 174, module 176, lighting system 178, HVAC system 180, data system 182, and controller 184. For example, lighting system 170 may be located external to beam 174. Vehicle 410, rotary position encoder 420, subframe 172, beam 174, module 176, lighting system 178, HVAC system 180, data system 182, and controller 184 may physically or logically coexist in whole or in part with one another.

[0085] although Figure 4 A particular number of vehicles 410, rotary position encoders 420, subframes 172, beams 174, modules 176, lighting systems 178, HVAC systems 180, data systems 182, and controllers 184 are shown, but the present invention contemplates any suitable number of vehicles 410, rotary position encoders 420, subframes 172, beams 174, modules 176, lighting systems 178, HVAC systems 180, data systems 182, and controllers 180. For example, image capture system 170 may include a first beam 174 at a front end of vehicle 410 and a second beam 174 at a rear end of vehicle 410. As another example, image capture system 178 may include multiple controllers 184 (e.g., controllers and sub-controllers). Figure 7 One or more components of the image capture system 170 may be implemented using one or more components of a computer system.

[0086] Figure 5 It shows that Figure 4 Module 176 for use with image capture system 170. Module 176 includes camera 510, lens 520, top plate 530, bottom plate 540, cover plate 550, cleaning device 560, bolts 570, opening 580, and air compressor 585. Camera 510 is a device for capturing images of physical objects (e.g., Figure 1 The camera 510 may be any device that can capture images of track components (e.g., rail joints, switches, frogs, fasteners, ballast, rail heads, and / or rail ties). In some embodiments, the camera 510 is a sensor.

[0087] One or more cameras 510 can capture images of one or more physical objects from different angles. For example, one or more cameras 510 can capture images of two tracks of a railway system at any given location. Each beam (e.g., Figure 4The beam 174 of the embodiment of the present invention may include multiple cameras 510. The beam may include a first camera 510 aimed directly downward to capture a top-down image of the physical object. The beam may include a second camera 510 aimed downward and outward to capture an oblique image of the physical object.

[0088] Camera 510 may be a line scan camera. A line scan camera includes a single row of pixels. Camera 510 may be a dual line scan camera. A dual line scan camera includes two rows of pixels that may be captured and / or processed simultaneously. Camera 510 may capture images as it moves over a physical object so that a complete image of the physical object may be reconstructed line by line in software. Camera 510 may have a capture rate of up to 140 kilohertz. Camera 510 may have a resolution and optics for detecting physical objects of at least 1 / 16 inch in size. In an alternative embodiment, camera 510 may be an area scan camera.

[0089] The camera 510 includes a lens 520 that focuses incident light and directs it to a sensor of the camera 510. The lens 520 may be a sheet of glass or other transparent material. The lens 520 may be made of any suitable material (e.g., steel, aluminum, glass, plastic, or a combination thereof).

[0090] The top plate 530 and the bottom plate 540 are structural elements for positioning, supporting and / or stabilizing one or more components (e.g., camera 510 or sensor) of the module 176. The top plate 530 and the bottom plate 540 can be made of any suitable material (e.g., steel, aluminum, plastic, glass, etc.). The top plate 530 can be connected to the bottom plate 540 with one or more bolts 570. The bolts 570 (e.g., positioning bolts) can be used to change the pitch and / or roll direction of the camera 510. For example, the bolts 570 can be used to change the effective height between the top plate 530 and the bottom plate 540. The top plate 530 and / or the bottom plate 540 can be adjusted to reduce vibration and / or shock of the module 176. The top plate 530 and / or the bottom plate 540 can include a resistive heating element to provide a warm environment for the operation of the camera 510 and the lens 520 during colder weather.

[0091] The cover plate 550 is a plate covering the base plate 540. The cover plate 550 can be made of any suitable material (e.g., glass, steel, aluminum, etc.). The cover plate 550 includes an opening 580. The opening 580 can be used as an aperture through which the lens of the camera 510 views the physical object. The opening 580 allows the sensing signal to be transmitted from the environment to reach the sensor of the camera 510. The opening 580 can be any suitable size (e.g., oval, rectangular, etc.) to accommodate the view of the camera 510. The lens 520 of the camera 510 and / or the air compressor 585 can be positioned directly above the opening 580. The air compressor 585 can be provided as described above. Figure 4 The aerodynamic vortex generators described provide cooling to the beam 174 .

[0092] The cleaning device 560 is any device that protects the lens 520 and / or the sensor of the camera 510 from environmental influences. The cleaning device 560 removes dust, small debris, water and / or other items that may obstruct the view through the opening 580. The cleaning device 560 provides minimal or no obstruction to the signals sent by the components (e.g., cameras or sensors) of the module 176. The cleaning device 560 can be located between the cover plate 550 and the base plate 540. In some embodiments, the cleaning device 560 is physically connected to the cover plate 550 and / or the base plate 540. The cleaning device 560 can be made of any suitable material (e.g., glass, steel, aluminum, etc.). The cleaning device 560 can be located on the outer surface of the module 176 (e.g., the bottom surface of the module 176).

[0093] The cleaning device 560 can use any suitable method to clean the lens 520 of the camera 510. For example, the cleaning device 560 can include a cleaning agent that discharges compressed air, compressed gas, or a cleaning fluid. The cleaning device 560 can include wiper blades, a brush, or any other suitable method to clean the lens 520. In some embodiments, the cleaning agent is a compressor (e.g., air compressor 585) that discharges compressed air or compressed gas. The compressed air or compressed gas is discharged through an orifice that is designed to utilize the Coanda effect, which draws nearby air into the mainstream to amplify the amount of air discharged through the lens 520 (see symbol 590 for air flow). In some embodiments, the cleaning device 560 can be part of an HVAC system (e.g., Figure 4 HVAC system 180).

[0094] although Figure 5 A particular arrangement of camera 510, lens 520, top plate 530, bottom plate 540, cover plate 550, cleaning device 560, bolts 570, opening 580, and air compressor 585 is shown, but the present invention contemplates any suitable arrangement of camera 510, lens 520, top plate 530, bottom plate 540, cover plate 550, cleaning device 560, bolts 570, opening 580, and air compressor 585. Although Figure 5 A particular number of cameras 510, lenses 520, top plates 530, bottom plates 540, covers 550, cleaning devices 560, bolts 570, openings 580, and air compressors 585 are shown, but the present invention contemplates any suitable number of cameras 510, lenses 520, top plates 530, bottom plates 540, covers 550, cleaning devices 560, bolts 570, openings 580, and air compressors 585. For example, module 176 may include multiple cameras 510. Figure 7One or more components of the computer system of the computer system are used to implement one or more components of module 176.

[0095] Figure 6 An exemplary method 600 of marking a feature 154 with a marker 162 is shown. Figure 6 The method 600 begins at step 610, where a classification engine (e.g., Figure 1 The classification engine 132 of the embodiment of the present invention identifies features 154 of the rail joint 132. The features 154 include a first square nut 634, a first bolt head 636, a first hole 638, a second hole 640, a second square nut 642, a second bolt head 636, a fracture 646, and a discontinuity 648. The fracture 646 indicates a defect of the rail dowel 632 (e.g., Figure 1 The discontinuity 648 represents a gap between the rails 630. The classification engine can use one or more CNNs to identify (e.g., classify) the features 154 of the rail joint 132.

[0096] At step 620 of method 600, a tagging engine (e.g., Figure 1 The tagging engine 138 of the rail joint 132 uses tags 162 to tag features 154 of the rail joint 132. As shown, the tagging engine tags feature 634 as "square nut (0.84)", feature 636 as "bolt head (0.77)", feature 638 as "hole (0.89)", feature 640 as "hole (0.89)", feature 642 as "square nut (0.84)", feature 644 as "bolt head (0.77)", feature 646 as "break (0.83)", and feature 648 as "discontinuity (0.87)". Each tag includes a classification of each feature 154 (e.g., Figure 1 155) and the probability that each classification accurately identifies each feature. For example, feature 636 labeled "bolt head (0.77)" represents the classification of "bolt head", and the probability that "bolt head" accurately identifies feature 636 is 0.77 (e.g., 77%). As another example, feature 638 labeled "hole (0.89)" represents the classification of "hole", and the probability that "hole" accurately identifies feature 638 is 0.89 (e.g., 89%). As another example, feature 646 labeled "break (0.83)" represents the classification of "break", and the probability that "break" accurately identifies feature 646 is 0.83 (e.g., 83%). As yet another example, feature 648 labeled "discontinuity (0.87)" represents the classification of "discontinuity", and the probability that "discontinuity" accurately identifies feature 648 is 0.87 (e.g., 87%).

[0097] The classification and / or probability may be determined by a classification engine using one or more machine learning networks (e.g., CNNs) and / or algorithms. In the illustrated embodiment, the location of each bounding box represents the location of each feature 154 relative to other features 154 and relative to the rail rebar 632. Each bounding box may be any suitable quadrilateral (i.e., square, rectangular, etc.). An object detection model may be used to output a detected, classified bounding box and label 162. Although Figure 6 The illustrated embodiment of shows feature 154 having a bounding box, but feature 154 may be represented by any suitable shape and / or character.

[0098] Method 600 can be used to further classify defects. Figure 6 In the example, defects are shown as “breaks”. One or more machine learning algorithms (e.g., Figure 6 CNNs 166) can be trained to recognize the illustrated categories of square nuts, bolt heads, fractures, holes, discontinuities, and rebars. When one or more CNNs are trained to visually distinguish between square nuts and bolt heads, the visual representation of the two categories is equivalent to the physical representation of the presence of a bolt. Defect detector engines (e.g., Figure 1 The defect detector engine 134 may use one or more algorithms to logically reduce the square nut and bolt head classifications to a single bolt classification. The defect detector engine may use one or more algorithms to examine the relative position of each inspected object (i.e., bolts, breaks, holes, and discontinuities) (e.g., Figure 1 position 158).

[0099] The defect detector engine may use one or more algorithms to use the relative location to determine whether a detected break in the dowel is a center break or a quarter break defect. In the illustrated embodiment, the detected bounding box for the break is located between the two middle bolts (i.e., "Bolt Head (0.77)" and "Square Nut (0.84)"). The defect detector engine may determine that the break is a center break dowel defect due to the location of the break between the two middle bolts, indicating that the break is very close to the discontinuity. In an alternative embodiment, the break may be located outside of the two middle bolts, and the defect detector engine may determine that the break is a quarter break dowel due to the location of the break outside of the two middle bolts.

[0100] Can Figure 6The method 600 described in the foregoing may be modified, added to, or omitted. The method 600 may include more, fewer, or other steps. For example, as described above, the method 600 may include further classifying a defect (e.g., a break) of a physical object (e.g., a rail dowel) into a more specific defect (e.g., a center-broken dowel or a quarter-broken dowel) based on the location of the defect relative to other features of the physical object (e.g., bolts, holes, and discontinuities). The steps may be performed in parallel or in any suitable order. Although discussed as specific components that perform the steps of the method 600, any suitable component may perform any of the steps of the method 600.

[0101] Figure 7 An exemplary computer system that can be used by the systems and methods described herein is shown. For example, Figure 1 The network 110, the defect detector module 120, and the image capture system 170 may include one or more interfaces 710, processing circuits 720, memory 730, and / or other suitable components. The interface 710 (e.g., Figure 1 Interface 710 may include hardware and / or software.

[0102] Processing circuit 720 (eg, Figure 1 The processor 126 of the processor 720 performs or manages the operation of the component. The processing circuit 720 may include hardware and / or software. Examples of processing circuits include one or more computers, one or more microprocessors, one or more applications, etc. In some embodiments, the processing circuit 720 executes logic (e.g., instructions) to perform actions (e.g., operations), such as generating outputs from inputs. The logic performed by the processing circuit 720 may be encoded in one or more tangible, non-transitory computer-readable media (e.g., memory 730). For example, the logic may include a computer program, software, computer-executable instructions, and / or instructions that can be executed by a computer. In a particular embodiment, the operation of the embodiment may be performed by one or more computer-readable media, which store, implement and / or encode a computer program, and / or have a stored and / or encoded computer program.

[0103] The memory 730 (or memory unit) stores information. The memory 730 (e.g., Figure 1The memory 124 may include one or more non-transitory, tangible, computer-readable and / or computer-executable storage media. Examples of memory 730 include computer memory (e.g., RAM or ROM), mass storage media (e.g., hard disk), removable storage media (e.g., compact disk (CD) or digital video disk (DVD)), database and / or network storage (e.g., server) and / or other computer-readable media.

[0104] Here, the computer-readable non-transitory storage medium may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODSs), magneto-optical disks, magneto-optical disk drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more thereof. Where appropriate, the computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0105] Unless expressly stated otherwise or the context indicates otherwise, "or" herein is inclusive and not exclusive. Thus, as used herein, "A or B" means "A, B, or both," unless expressly stated otherwise or the context indicates otherwise. Additionally, "and" is conjunctive, unless expressly stated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly stated otherwise or the context indicates otherwise.

[0106] The scope of the present invention encompasses all changes, substitutions, variations, alterations and modifications to the exemplary embodiments described or illustrated herein that will be understood by a person skilled in the art. The scope of the present invention is not limited to the exemplary embodiments described herein or shown in the drawings. In addition, although the present disclosure describes and illustrates various embodiments herein, including specific components, elements, features, functions, operations or steps, any of these embodiments may include any combination or arrangement of any components, elements, features, functions, operations, or steps described or illustrated anywhere herein that will be understood by a person skilled in the art. In addition, in the appended claims, references to devices or systems or parts of devices or systems, which are suitable for being arranged, configured, operable or operable to perform specific functions, include the devices, systems, components, whether or not they or the specific functions are activated, turned on or unlocked, as long as the devices, systems or components are so adjusted, arranged, enabled, configured, enabled, operable or operable. In addition, although the present invention describes or illustrates specific embodiments as providing specific advantages, specific embodiments may not provide these advantages, some advantages or all advantages.

Claims

1. A system for determining a defect in a physical object, comprising: an image capture system including a subframe, a light beam, one or more sensors, an illumination system, a data system, and one or more controllers; as well as A defect detector module comprising: an image collection engine configured to collect one or more images of one or more physical objects from the image capture system; a classification engine configured to classify one or more features from the image into one or more categories; a defect detector engine configured to analyze the one or more categories to identify one or more defects; and a reporting engine configured to generate a report indicating whether the physical object is defective based on the defect detector engine analysis; wherein the category engine analyzes the image via one or more machine learning algorithms; and Wherein the one or more machine learning algorithms output the one or more categories that describe the one or more features of the one or more images within a level of certainty.

2. The system of claim 1 , wherein each machine learning algorithm is trained on labeled image data to discern the one or more features from the one or more images.

3. The system according to claim 1, wherein: The one or more machine learning algorithms are trained to recognize specific physical objects as viewed through one or more components of the image capture system.

4. The system of claim 1, wherein the level of certainty depends on the number of classes, the desired accuracy, and the availability of sample training data for each classification. The system of claim 1 , wherein the level of certainty can be expressed as a probability.

6. The system of claim 1, wherein the first category represents a first characteristic of the railroad joint.

7. The system of claim 1, wherein a second category represents one or more second characteristics associated with the rail joint.

8. The system of claim 1, wherein the one or more categories include one or more railroad defects.

9. The system of claim 1, wherein the defect detector engine generates additional information about the defect based on the relative location of the defect.

10. A defect detector module, comprising: an image collection engine configured to collect one or more images of one or more physical objects; a classification engine configured to classify one or more features from the image into one or more categories; a defect detector engine configured to analyze the one or more categories to identify one or more defects; as well as a reporting engine configured to generate a report indicating whether the physical object is defective based on the defect detector engine analysis; as well as an interface configured to receive and transmit data from the category engine, the defect detector engine, or the reporting engine; wherein the category engine analyzes the image via one or more machine learning algorithms; and Wherein the one or more machine learning algorithms output the one or more categories that describe the one or more features of the one or more images within a level of certainty.

11. The module of claim 10, wherein each machine learning algorithm is trained on labeled image data to discern the one or more features from the one or more images.

12. The module according to claim 10, wherein: The one or more machine learning algorithms are trained to recognize specific physical objects as viewed through one or more components of the image capture system.

13. The module of claim 10, wherein the level of certainty depends on the number of classes, the desired accuracy, and the availability of sample training data for each classification.

14. The module of claim 10, wherein the first category represents a first characteristic of a railroad joint.

15. The module of claim 10, wherein a second category represents one or more second characteristics associated with the rail joint.

16. The module of claim 10, wherein the defect detector engine generates additional information about the defect based on the relative location of the defect.

Images