System and method for reconstructing objects using transition images

Abstract

To reconstruct an image from consecutively captured images of objects such as railroad tracks taken from a railway vehicle, the image achieving both identification speed and identification accuracy.SOLUTION: A method includes, by an object reconstruction module, a step of receiving a first image and a second image. The first image includes a first region of an object, and the second image includes a second region of the object. The method also includes, by the object reconstruction module, a step of identifying a transitional image. The transitional image includes the first region of the object and the second region of the object. The method further includes, by the object reconstruction module, a step of determining that the first region of the object in the transitional image and the first region of the object in the first image are equivalent regions, and a step of generating a reconstruction of the object by using the first image and the transitional image. The reconstruction of the object includes the first region of the object and the second region of the object and excludes the equivalent regions.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure generally relates to reconstructing an object, and particularly to a system and method for reconstructing an object using transitional images.

Background Art

[0002] An object detection model processes a large amount of image data and analogizes each image independently. There is an inherent tradeoff between the speed and accuracy of a deep learning model. This tradeoff is amplified when training the model to detect an object with a dramatically changing size and / or a part of an object with a very small ratio of the overall pixel data. An object detection model can be trained to accurately identify objects in multiple images, but in a specific system, it may not be able to tolerate the latency penalty associated with accurately detecting a very small part of an object.

Summary of the Invention

Means for Solving the Problems

[0003] According to one embodiment, the method includes receiving a first image and a second image by an object reconstruction module. The first image includes a first region of the object, and the second image includes a second region of the object. The method also includes identifying a transition image by the object reconstruction module. The transition image includes a first region of the object and a second region of the object. The method also includes determining by the object reconstruction module that the first region of the object in the transition image and the first region of the object in the first image are a first equivalent region, and generating an object reconstruction by the object reconstruction module using the first image and the transition image. The object reconstruction includes the first region of the object and the second region of the object, but excludes the first equivalent region.

[0004] In certain embodiments, the first image further includes the third, fourth, and fifth regions of the object, and the transition image includes the third and fourth regions of the object. In some embodiments, the method includes the step of determining the third region of the object in the transition image and the third region of the object in the first image by the object reconstruction module. The image is the third equivalent region, and the object reconstruction module determines that the fourth region of the object in the transition image and the fourth region of the object in the first image are the fourth equivalent region. In certain embodiments, the object reconstruction excludes the third and fourth regions. In some embodiments, the method includes the step of concatenating the first, second, third, fourth, and fifth regions of the object by the object reconstruction module to generate an object reconstruction, where the object reconstruction represents the entire object.

[0005] In a particular embodiment, the method includes the steps of: identifying a first portion of an object in a first image using an object reconstruction module; and constructing a first bounding box around the first portion of the object in the first image using the object reconstruction module. The first bounding box may include a first region of the object. In some embodiments, the method includes the steps of: identifying a second portion of the object in the transition image by an object reconstruction module; and constructing a second bounding box around the second portion of the object in the transition image by an object reconstruction module. The second bounding box may include a first region and a second region of the object. In certain embodiments, the method includes the process of using an Intersection over Union (IoU) to determine that the first region of the object in the transition image and the first region of the object in the first image are equivalent regions. In some embodiments, the transition image includes half of the first image and half of the second image.

[0006] In other embodiments, the apparatus includes one or more processors and one or more computer-readable non-transitory storage media coupled to the one or more processors. The one or more computer-readable non-transitory storage media, when executed by one or more processors, include instructions causing the apparatus to perform processing including receiving a first image and a second image. The first image includes a first region of an object, and the second image includes a second region of an object. The processing also includes processing to identify a transition image. The transition image includes the first region and the second region of an object. The processing also includes processing to determine that the first region of an object in the transition image and the first region of an object in the first image are a first equivalent region, and processing to generate a reconstruction of the object using the first image and the transition image. The reconstruction of the object includes the first region and the second region of an object, but excludes the first equivalent region.

[0007] In another embodiment, one or more computer-readable storage media, when executed by a processor, embody instructions that cause the processor to perform a process including receiving a first image and a second image by an object reconstruction module. The first image includes a first region of the object, and the second image includes a second region of the object. The process also includes the object reconstruction module identifying a transition image. The transition image includes the first region and the second region of the object. The process also includes the object reconstruction module determining that the first region of the object in the transition image and the first region of the object in the first image are a first equivalent region, and the object reconstruction module generating a reconstruction of the object using the first image and the transition image. The reconstruction of the object includes the first region and the second region of the object, but excludes the first equivalent region.

[0008] The technical advantages of certain embodiments of this disclosure may include one or more of the following: This disclosure describes systems and methods for reconstructing objects using transition images. In certain embodiments, the bounding box reduction algorithm used to reconstruct objects may be used in combination with standard non-maximum suppression algorithms that operate on an image-by-image basis to reduce the bounding boxes contained within a single image and a single analogy result. In certain embodiments, the transition image is generated by superimposing two images at a predetermined percentage. In certain embodiments of this disclosure, a predetermined percentage of 50% is used, but this disclosure intends any appropriate predetermined percentage (e.g., 25 percent, 75 percent, etc.). In certain embodiments, the systems and methods disclosed herein may use reduction criteria such as IoU scores between two or more boxes, equivalence of detection labels, and / or confidence score thresholds.

[0009] In certain embodiments of this disclosure, the systems and methods disclosed herein may be used across several object classes within the same image. For example, in some embodiments, bounding boxes belonging to the same class can be logically reduced, and As a result, several fully configured objects may be generated. In certain embodiments, the equivalent regions of bounding boxes and / or other regions of interest may be calculated using IoU, similar to how standard non-maximal suppression algorithms are used to calculate equivalence. The systems and methods described herein can be generalized to various transport infrastructures, including railroads, roads, and waterways.

[0010] Other technical advantages will be readily apparent to those skilled in the art from the following drawings, description, and claims. Furthermore, while certain advantages are listed above, various embodiments may include all or some of the listed advantages, or none at all.

[0011] To aid in understanding the contents of this disclosure, please refer to the following explanation in conjunction with the attached drawings. [Brief explanation of the drawing]

[0012] [Figure 1] This diagram illustrates an exemplary system for reconstructing an object using transition images. [Figure 2] This figure shows an exemplary image that may be used by the system in Figure 1. [Figure 3] This figure shows an exemplary transition image that may be used by the system in Figure 1. [Figure 4] This figure shows an example of an equivalent region that can be determined by the system in Figure 1. [Figure 5] This figure shows an example of object reconstruction that can be generated by the system in Figure 1. [Figure 6] This figure illustrates an exemplary method for reconstructing an object using transition images. [Figure 7] This figure shows an exemplary computer system that may be used by the systems and methods described herein. [Modes for carrying out the invention]

[0013] Figures 1 to 7 illustrate exemplary systems and methods for reconstructing objects using transition images. Figure 1 shows an exemplary system for reconstructing objects using transition images, and Figure 2 shows an exemplary image that may be used by the system in Figure 1. Figure 3 shows an exemplary transition image that may be used by the system in Figure 1, and Figure 4 shows an exemplary equivalent region that may be determined by the system in Figure 1. Figure 5 shows an example of object reconstruction that may be generated by the system in Figure 1. Figure 6 shows an exemplary method for reconstructing objects using transition images. Figure 7 shows an exemplary computer system that may be used by the systems and methods described herein.

[0014] Figure 1 shows an exemplary system 100 for reconstructing an object using transition images. System 100 or any part thereof may be associated with an entity that reconstructs an object using transition images, and may include any organization such as a corporation, company (e.g., a railway company, a transport company, etc.) or government agency (e.g., a transport bureau, a public security bureau, etc.). The elements of system 100 can be implemented using any suitable combination of hardware, firmware, and software. For example, the elements of system 100 may be implemented using one or more components of the computer system shown in Figure 7. System 100 includes a network 110, a railway environment 120, railway tracks 130, railway vehicles 140, an image capture module 150, an object 160, an object reconstruction module 170, an image 172, a transition image 174, a bounding box 176, an equivalent region, and an object reconstruction 180.

[0015] Network 110 can be any type of network that facilitates communication between components of system 100. One or more parts of network 110 are ad hoc networks. Network 110 may include work networks, intranets, extranets, virtual private networks (VPNs), local area networks (LANs), wireless LANs (WLANs), wide area networks (WANs), wireless WANs (WWANs), metropolitan area networks (MANs), parts of the Internet, parts of public switched telephone networks (PSTNs), mobile phone networks, 3G networks, 4G networks, 5G networks, LTE (Long Term Evolution) cellular networks, two or more combinations of these, or other appropriate types of networks. One or more parts of network 110 may include one or more accesses (e.g., mobile access), core and / or edge networks. Network 110 can be any communication network, such as a private network, a public network, a connection via the Internet, a mobile network, a Wi-Fi network, or a Bluetooth® network. One or more components of system 100 may transmit information via network 110. For example, object reconstruction module 170 may transmit information via network 110, including receiving information from image capture module 150.

[0016] The railway environment 120 of system 100 is an area that includes one or more railway tracks 130. The railway environment 120 may be related to divisions and / or subdivisions. A division is a part of the railway under the supervision of a supervisor. A subdivision is an even smaller part of a division. Subdivisions may be crew districts and / or branch lines. In the embodiment shown in Figure 1, the railway environment 120 includes railway tracks 130, railway vehicles 140, an image capture module 150, and objects 160.

[0017] The railway track 130 of system 100 is a structure that allows the railway vehicle 140 to move by providing a surface on which the wheels of the railway vehicle 140 can roll. In certain embodiments, the railway track 130 includes rails, fasteners, sleepers, ballast, etc. The railway vehicle 140 of system 100 is a vehicle that can move along the railway track 130. The railway vehicle 140 may be a geometric vehicle, locomotive, passenger car, freight car, covered wagon, flatcar, tank car, etc. In certain embodiments, the railway vehicle 140 may be associated with one or more image capture modules 150.

[0018] The image capture module 150 of system 100 is a component that captures images 172 of object 160. Object 160 of system 100 is a transportation infrastructure component such as roads, railways, air routes, waterways, canals, pipelines, and terminal components. Object 160 may include components within the railway environment 120 such as railway tracks 130, debris (e.g., rubble, wreckage, ruins, litter, trash, brush, etc.), pedestrians (e.g., trespassers), animals, vegetation, and ballast. Object 160 may include components of the railway track 130, such as joints, switches, frogs, rail heads, anchors, fasteners, gauge plates, ballast, and sleepers (e.g., concrete sleepers and wooden sleepers). Image 172 of system 100 shows the physical form of one or more objects 160. Image 172 may include digital images, photographs, etc. Image 172 includes image 172a, image 172b, etc., and continues up to image 172n, where 172n represents any appropriate number.

[0019] The image capture module 150 may include one or more cameras, lenses, sensors, optical devices, lighting elements, etc. For example, the image capture module 150 may include Object 1 The system may include one or more cameras (e.g., high-resolution cameras, line-scan cameras, stereo cameras, etc.) that automatically capture 60 images 172. In certain embodiments, the image capture module 150 is mounted on a railway vehicle 140. The image capture module 150 can be mounted on the railway vehicle 140 at any suitable location that provides a clear view of the railway environment 120. For example, the image capture module 150 may be mounted on the front end of the railway vehicle 140 (e.g., the front bumper) to provide a downward view of the railway track 130. As another example, the image capture module 150 may be mounted on the front end of the railway vehicle 140 (e.g., the windshield) to provide a forward view of the railway track 130. As yet another example, the image capture module 150 may be mounted on the rear end of the railway vehicle 140 (e.g., the rear windshield) to provide a rear view of the railway track 130.

[0020] The image capture module 150 can automatically capture still images or videos 172 while the railway vehicle 140 moves along the railway line 130. The image capture module 150 can automatically capture any suitable number of still images or videos 172. For example, the image capture module 150 may use an encoder (e.g., a position encoder) to automatically capture an image 172 as a function of distance in order to trigger a camera (e.g., a line scan camera). As another example, the image capture module 150 may automatically capture a predetermined number of still images 172 per second, per minute, per hour, etc. In some embodiments, the image capture module 150 can inspect an image 172 for an object 160. For example, the image capture module 150 may use one or more models to detect an object 160 within the image 172. The models can include one or more object detection models, machine learning models, machine vision models, deep learning models, etc. The image capture module 150 can transmit one or more images 172 of one or more objects 160 to the object reconstruction module 170.

[0021] The object reconstruction module 170 of the system 100 is a component that regenerates an object 160 from one or more images 172 and / or one or more transition images 174. In certain embodiments, the object reconstruction module 170 receives an image 172 from one or more components of the system 100. For example, the object reconstruction module 170 may receive image 172a and image 172b from the image capture module 150. The plurality of images 172 can include one or more portions of a single object 160. For example, image 172a may include a first portion of object 160, and image 172b may include a second portion of object 160.

[0022] Each object 160 can be represented as a region. For example, the object 160 (e.g., the joint plate of the rail) may have five regions such that the image 172a includes four consecutive regions of the object 160 and the image 172b includes the remaining fifth region of the object 160. As another example, the object 160 may have five regions such that the image 172a includes three consecutive regions of the object 160 and the image 172b includes the remaining fourth and fifth regions of the object 160.

[0023] In certain embodiments, the object reconstruction module 170 performs an analogy for one or more images 172 and / or transition images 174. The object reconstruction module 170 may perform the analogy by executing an object detection algorithm to detect the object 160 within the image 172 and / or transition image 174. In response to performing the analogy, the object reconstruction module 170 may generate one or more bounding boxes 176 around one or more objects 160 (or a portion thereof). The bounding box 176 is a contour line that identifies the object 160. In certain embodiments, each bounding box 176 has x and y coordinates that identify the four corners of a rectangle and is a rectangular box around the object 160. The bounding box 176 includes bounding box 176a, bounding box 176b, bounding box 176c, etc., and continues up to bounding box 176n, where 174n represents any suitable number. and is a rectangular box around the object 160. The bounding box 176 includes bounding box 176a, bounding box 176b, bounding box 176c, etc., and continues up to bounding box 176n, where 174n represents any suitable number.

[0024] In certain embodiments, the image capture module 150 configures a bounding box 176 around the object 160 (or a portion thereof) in the image 172. For example, the object reconstruction module 170 may identify a first portion of the object 160 in image 172a and configure a bounding box 176a around the first portion of the object 160 in image 172a. As another example, the object reconstruction module 170 may identify a second portion of the object 160 in image 172b and configure a bounding box 176b around the second portion of the object 160 in image 172b. As yet another example, the object reconstruction module 170 may identify a third portion of the object 160 in transition image 174a and configure a bounding box 176c around the third portion of the object 160 in transition image 176a.

[0025] In some embodiments, the region of object 160 is defined by a bounding box 176. For example, in the case of an object having five regions, the first portion of object 160 enclosed by the bounding box 176a in image 172a may define the first, second, third, and fourth regions of object 160; the second portion of object 160 enclosed by the bounding box 176b in image 172b may define the fifth region of object 160; and the third portion of object 160 enclosed by the bounding box 176c in transition image 174a may define the second, third, fourth, and fifth regions of object 160.

[0026] In a particular embodiment, the object reconstruction module 170 generates a transition image 174 from an image 172. The transition image 174 is an image that shares the same pixels as one or more other images 172. For example, image 172a and transition image 174a may share a predetermined ratio of the same pixels (e.g., 25, 50, or 75%). The transition image 174 includes transition images 174a, 174b, and so on, up to transition image 174n, where 176n represents any appropriate number.

[0027] In some embodiments, the object reconstruction module 170 generates a transition image 174 by superimposing two or more images 172. For example, the object reconstruction module 170 may generate a transition image 174a by superimposing a portion of image 172a (e.g., 50%) and a portion of image 172b (e.g., 50%) such that the transition image 174a includes the overlapping portion of image 172a and image 172b. In some embodiments, the image capture module 150 captures the transition image 174. For example, the image capture module 150 may be configured to capture a subsequent image 172 with a predetermined amount of overlap (e.g., 25, 50, or 75 percent overlap).

[0028] In certain embodiments, each transition image 174 includes one or more regions of the object 160. For example, in the case of an object having five regions, image 172a may include the first, second, third, and fourth regions, image 172b may include the fifth region, and transition image 174a may include the second, third, and fourth regions of image 172a and the fifth region of image 172b.

[0029] In a particular embodiment, the object reconstruction module 170 determines one or more equivalent regions 178. The equivalent regions 178 are images 172 and transition images 178 that share the same pixels. This is a region within 74. Equivalent region 178 includes equivalent region 178a, equivalent region 178b, etc., up to equivalent region 178n, where 178n represents any appropriate number. The object reconstruction module 170 may determine equivalent regions 178 between one or more images 172 and one or more transition images 174. For example, the object reconstruction module 170 may determine that both image 172a and transition image 174a contain the second, third, and fourth regions of object 160. In certain embodiments, the object reconstruction module 170 excludes equivalent regions 178 from object reconstruction 180. In some embodiments, the object reconstruction module 170 uses IoU, detection label equivalence, and / or confidence score thresholds to determine that the first region of object 160 in transition image 174 and the first region of object 160 in image 172a are equivalent regions 178.

[0030] In certain embodiments, the object reconstruction module 170 generates an object reconstruction 180. Each object reconstruction 180 is an overall representation of object 160. For example, an object reconstruction 180 may include the first, second, third, and fourth regions of image 172a and the fifth region of image 172b. The object reconstructions 180 include object reconstruction 180a, object reconstruction 180b, etc., and continue up to object reconstruction 180n, where 180n represents any appropriate number. In certain embodiments, the object reconstruction module 170 generates an object reconstruction 180 using one or more images 172 and one or more transition images 174. For example, the object reconstruction module 170 may generate an object reconstruction 180a by concatenating the first, second, third, and fourth regions from image 172a and the fifth region from transition image 174a. In certain embodiments, when generating an object reconstruction 180, the object reconstruction module 170 excludes equivalent regions 178 to avoid overlapping regions of object 160. For example, if both image 172a and transition image 174a include the second, third, and fourth regions of object 160, the object reconstruction module 170 may exclude the second, third, and fourth regions of transition image 174a to avoid overlapping of these regions in object reconstruction 180a.

[0031] During operation, the image capture module 150 of system 100 is attached to the railway vehicle 140.

[0032] The image capture module captures images 172a and 172b of object 160 while the railway vehicle 140 moves along the railway track 130 in the railway environment 120. The object reconstruction module 170 of system 100 receives images 172a and 172b of object 160 (e.g., rail joint bar) from the image capture module 150 via the network 110. The object reconstruction module 170 uses one or more object detection models to detect the first portion of object 160 in image 172a. The object reconstruction module 170 constructs a bounding box 176a around the first portion of object 160 in image 172a. Representing object 160 as five regions, the first portion of image 172a enclosed by the bounding box 176a includes the first, second, third, and fourth regions of object 160. Image 172b includes the second portion and fifth region of object 160. However, the object reconstruction module 170 may fail to detect the second portion of object 160 due to the size of the second portion relative to the overall size of image 172b.

[0033] The object reconstruction module 170 generates a transition image 174a by superimposing image 172a and image 172b such that the transition image 174a includes half of image 172a and half of image 172b. The object reconstruction module 170 detects the third portion of object 160 in the transition image 174a, and the third portion of object 160 A bounding box 176c is formed around the object. The bounding box 176c includes the second, third, fourth, and fifth regions of object 160. The object reconstruction module 170 determines that the second, third, and fourth regions of image 172a and transition image 174a are equivalent regions 178. To avoid overlapping regions of object 160, the object reconstruction module 170 excludes equivalent regions 178 (e.g., the third, fourth, and fifth regions of transition image 174a) in the object reconstruction 180. The object reconstruction module 170 generates the object reconstruction 180 of object 160 by concatenating the first, second, third, and fourth regions of image 172a with the fifth region of transition image 174a. In this way, the system 100 can be used to accurately regenerate the entire object 160 even if a small portion of object 160 is not detected in image 172b.

[0034] Figure 1 shows a specific arrangement of network 110, railway environment 120, railway track 130, railway vehicle 140, image capture module 150, object 160, object reconstruction module 170, image 172, bounding box 176, transition image 174, equivalent region 178, and object reconstruction 180. This specification intends for any suitable arrangement of network 110, railway environment 120, railway track 130, railway vehicle 140, image capture module 150, object 160, object reconstruction module 170, image 172, bounding box 176, transition image 174, equivalent region 178, and object reconstruction 180.

[0035] For example, the image capture module 150 and the object reconstruction module 170 may be combined into a single module.

[0036] Figure 1 shows a specific number of network 110, railway environment 120, railway track 130, railway vehicle 140, image capture module 150, object 160, object reconstruction module 170, image 172, bounding box 176, transition image 174, equivalent region 178, and object reconstruction 180. This specification intends any appropriate number of network 110, railway environment 120, railway track 130, railway vehicle 140, image capture module 150, object 160, object reconstruction module 170, image 172, transition image 174, bounding box 176, equivalent region 178, and object reconstruction 180. For example, system 100 may include multiple railway environments 120, multiple image capture modules 150, and / or multiple object reconstruction modules 170.

[0037] Figure 2 shows an exemplary image 172 that may be used by the system of Figure 1. In the embodiment shown in Figure 2, image 172 includes image 172a and image 172b. Image 172a includes a portion of an object (e.g., object 160 in Figure 1), and image 172b includes the remaining portion of the same object. The portion of the object in image 172a is enclosed by a bounding box 176a, and the portion of the object in image 172b is enclosed by a bounding box 176b.

[0038] To help understand the entire object in image 172, we divide the object into five regions. The portion of the object in image 172a includes region 1, region 2, region 3, and region 4, and the portion of the object in image 172b includes region 5. If the object is represented as two independent images 172, the reconstruction of the entire object is bounding box 176a from image 172a (including regions 1 through 4) followed by bounding box 176b from image 172b (including region 5). The concatenation of bounding box 176a and bounding box 176b generates the complete set of regions 1 through 5.

[0039] As shown in Figure 2, image 172b contains one-fifth of the object, which is a relatively small portion of the object. The object reconstruction module in Figure 1 can be trained to accurately detect the relatively small portion of the object in image 172b, but this process can significantly reduce the object detection speed. In certain embodiments, the object reconstruction module may not be able to absorb the delay penalty of detecting region 5 in image 172b. However, the object reconstruction module needs to capture the entire object. To solve this problem, the object reconstruction module can use transition images, as described in Figure 3 below, to accurately detect the entire object without incurring a full delay penalty.

[0040] Figure 3 shows a transition image 174a that may be used by the system in Figure 1. Images 172a and 172b are each divided into two parts, namely a first part and a second part. The second part of image 172a includes regions 2, 3, and 4 of the bounding box 176a, and the first part of image 172b includes region 5 of the bounding box 176b. In certain embodiments, the object reconstruction module may generate a transition image 174a that includes the second part of image 172a and the first part of image 172b by superimposing the second part of image 172a with the first part of image 172b. In some embodiments, the transition image 174a is generated by an image capture module.

[0041] Independent analogies are performed for each of the images 172a, 172b, and transition image 174a in Figure 3. In response to the execution of the analogies, bounding boxes 176a are formed on image 172a, bounding boxes 176b are formed on image 172b, and bounding boxes 176c are formed on transition image 174a. Image 172a includes regions 1, 2, 3, and 4, image 172b includes region 5, and transition image 174a includes regions 2, 3, 4, and 5.

[0042] Because of the presence of transition image 174a, the object reconstruction module, which sacrifices accuracy for speed, can accurately detect the entire object in images 172a and 172b. Even if the object reconstruction module cannot detect region 5 of the object in image 172b, the pixels originally contained in image 172b are also contained in transition image 174a. Therefore, the detection of the object in images 172a and transition image 174a is a complete detection. Analogy to the additional image (e.g., transition image 174a) results in a delay penalty, but in certain embodiments, this delay penalty may be less than the delay penalty resulting from accurately detecting region 5 of the object in image 172b. Figures 4 and 5 below illustrate additional processing applied to the image to avoid overlap of regions 1 through 5.

[0043] Figure 4 shows an exemplary equivalent region 178a that may be determined by the system of Figure 1. Image 172a includes a bounding box 176a (enclosing regions 1, 2, 3, and 4 of the object), and image 172b includes a bounding box 176b (enclosing region 5 of the object). In certain embodiments, each region may be enclosed by its own bounding box 176. For example, region 1 in image 172a may be enclosed by a first bounding box 176, region 2 in image 172a may be enclosed by a second bounding box 176, and so on. In certain embodiments, bounding boxes may be nested within each other. For example, if two other overlapping objects are detected, region 1 may be enclosed by a first bounding box 176 that can be nested within a larger bounding box 176.

[0044] As shown in Figure 4, the bounding box 176a of image 172a and the bounding box 176c of transition image 174a are not equivalent. Bounding boxes 176a and 176c occupy different coordinate spaces and contain different regions of the object. However, the pixels contained in the second part of image 172a are the same pixels contained in the first part of transition image 174a. A simple concatenation of the bounding box 176a of image 172a and the bounding box 176c of transition image 174a produces region 12342345, which is an inaccurate overlap of regions 2, 3, and 4 of the entire object.

[0045] To accurately reconstruct the entire object without overlapping regions, image 172a and / or transition image 174a are scaled down to account for overlapping pixels. In certain embodiments, the object reconstruction module in Figure 1 determines equivalent regions 178a in image 172a and transition image 174a. Equivalent regions 178a are regions in two or more images that share the same pixels. The object reconstruction module 170 may use IoU, detection label equivalence, and / or confidence score thresholds to determine equivalent regions 178a. For example, the object reconstruction module 170 may use IoU to determine that the bounding boxes 176 surrounding regions 2, 3, and 4 in image 172a and regions 2, 3, and 4 in transition image 174a are equivalent regions 178a. In certain embodiments, the object reconstruction module calculates the IoU of detections in image 172a and transition image 174a and determines whether the IoU is greater than a predetermined threshold. If the IoU is greater than a predetermined threshold, the object reconstruction module 170 determines that the overlap between image 172a and transition image 174a defines an equivalent region 178a. As illustrated in Figure 5 below, the equivalent region 178a may be excluded from the final object reconstruction.

[0046] Figure 5 shows an exemplary object reconstruction 180a that can be generated by the system in Figure 1. As shown in Figure 4 above, regions 2, 3, and 4 in image 172a and transition image 174a are equivalent regions 178a. Equivalent regions 178a are excluded from object reconstruction 180a to avoid duplication of such regions (i.e., regions 2, 3, and 4 are included in object reconstruction 180a only once). The object reconstruction module in Figure 1 generates object reconstruction 180a by concatenating regions 1, 2, 3, 4, and 5. The concatenation of regions 1 through 5 can be represented as 12345 so that object reconstruction 180a represents an object (e.g., object 160 in Figure 1 as a whole). If the object reconstruction module detects region 5 of an object in image 172b, the same process may be applied to avoid duplication of equivalent regions 178b.

[0047] Figure 6 shows an exemplary method 600 for reconstructing an object using transition images (e.g., transition image 174a in Figure 1). Method 600 begins in step 610. In step 620, an object reconstruction module (e.g., object reconstruction module 170 in Figure 1) receives two or more images of an object (e.g., object 160 in Figure 1) via a network (e.g., network 110 in Figure 1). For example, the object reconstruction module may receive a first image (e.g., image 172a in Figure 1) and a second image (e.g., image 172b in Figure 1) of the object from an image capture module (e.g., image capture module 150 in Figure 1). The image capture module may be mounted on a railway vehicle (e.g., railway vehicle 140 in Figure 1). In a particular embodiment, the image capture module captures images of the object while the railway vehicle is moving along a railway track (e.g., railway track 130) in a railway environment (e.g., railway environment 120 in Figure 1). Next, method 600 moves from step 620 to step 630.

[0048] In step 630 of method 600, the object reconstruction module decides whether or not to use a transition image (e.g., transition image 174a in Figure 3) for object reconstruction. For example, the object reconstruction module may decide to use a transition image if it prioritizes object detection speed over object detection accuracy (e.g., detecting a relatively small portion of an object in the received image). If the object reconstruction module decides not to use a transition image for object reconstruction, it proceeds from step 630 to step 670, where it generates an object reconstruction from the received image. For example, the first image received by the object reconstruction module may include the first, second, and third regions of the object, and the second image received by the object reconstruction module may include the fourth and fifth regions of the object.

[0049] The object reconstruction module may detect a first, second, and third region of an object in the first image. The object reconstruction module may detect a fourth and fifth region of an object in the second image. The object reconstruction module may generate an object reconstruction by concatenating the first, second, and third regions detected in the first image with the fourth and fifth regions detected in the second image, so that the object reconstruction represents the entire object. Next, method 600 moves from step 670 to step 680, where method 600 terminates.

[0050] If, in step 630, the object reconstruction module decides to use a transition image for object reconstruction, the object reconstruction module may create a transition image by superimposing the first image and the second image such that the transition image includes the overlapping portions of each image. For example, the first image received by the object reconstruction module may include the first, second, third, and fourth regions of the object, and the second image received by the object reconstruction module may include the fifth region of the object. The object reconstruction module may generate a transition image by superimposing the first image and the second image such that it includes the second, third, fourth, and fifth regions of the object. In a particular embodiment, the object reconstruction module receives a transition image from the image capture module. Method 600 then moves from step 630 to step 640.

[0051] In step 640 of method 600, the object reconstruction module performs an inference for each image. For example, the object reconstruction module may perform an inference for the first image, the second image, and the transition image by executing an object detection algorithm to detect objects in the first image, the second image, and the transition image. In response to performing the inference and detecting objects, the object reconstruction module may generate one or more bounding boxes around the detected object (or part thereof). Each bounding box may contain one or more regions of the object. Method 600 then moves from step 640 to step 650.

[0052] In step 650 of Method 600, the object reconstruction module determines equivalent regions in the received image (e.g., equivalent region 178a in Figure 4). Equivalent regions are regions that share the same pixels. In certain embodiments, the object reconstruction module performs bounding box and overlapping region crossing of the first image, second image, and transition image to normalize bounding boxes that may extend beyond overlapping regions before calculating the IoU of two or more boxes. In some embodiments, the object reconstruction module uses IoU, detected label equivalence, and / or confidence score thresholds to determine overlapping regions. For example, the object reconstruction module may use IoU, detected label equivalence, and / or confidence score thresholds to determine that the second, third, and fourth regions overlap in the first image and transition image. As another example, object reconstruction The module may use IoU, detection label equivalence, and / or confidence score thresholds to determine if the fifth region overlaps in the second image and the transition image. Next, method 600 moves from step 650 to step 660.

[0053] In step 660 of method 600, the object reconstruction module excludes equivalent regions from the object reconstruction. For example, if the object reconstruction module determines that the second, third, and fourth regions from the first image and the transition image are equivalent regions, the object reconstruction module may exclude the equivalent regions in the object reconstruction so that the object reconstruction includes only one copy of the second, third, and fourth regions. As another example, if the object reconstruction module determines that the fifth region from the second image and the transition image are equivalent regions, the object reconstruction module may exclude the equivalent regions in the object reconstruction so that the object reconstruction includes only one copy of the fifth region. Next, method 600 moves from step 660 to step 670.

[0054] In step 670 of method 600, the object reconstruction module generates an object reconstruction (e.g., object reconstruction 180a in Figure 5) from one or more received images and one or more transition images. For example, the first image received by the object reconstruction module may include the first, second, third, and fourth regions of the object, and the transition image generated in step 630 of method 600 may include the second, third, fourth, and fifth regions of the object. The object reconstruction module may generate the object reconstruction by concatenating the first, second, third, and fourth regions from the first image with the fifth region of the transition image so that the object reconstruction represents the entire object. Next, method 600 moves from step 670 to step 680, where method 600 terminates. Thus, method 600 can be used to reconstruct an object as a whole even if a specific region of the object is not detected in the received image.

[0055] Method 600 shown in Figure 4 can be modified, added to, or omitted. Method 600 may include more, fewer, or other steps. For example, Method 600 may include utilizing a second transition image. The steps may be performed in parallel or in any suitable order. Although discussed as specific components that complete the steps of Method 600, any suitable component may perform any step of Method 600. For example, one or more steps of Method 600 may be performed by an image capture module.

[0056] Figure 7 shows an exemplary computer system that may be used by the systems and methods described herein. For example, one or more components of system 100 in Figure 1 (e.g., image capture module 150 and / or object reconstruction module 170) may include one or more interfaces 710, processing circuits 720, memory 730, and / or other suitable elements. Interface 710 receives inputs, transmits outputs, processes inputs and / or outputs, and / or performs other suitable operations. Interface 710 may include hardware and / or software.

[0057] The processing circuit 720 executes or manages the processing of the components. The processing circuit 720 may include hardware and / or software. Examples of the processing circuit include one or more computers, one or more microprocessors, one or more applications, etc. In certain embodiments, the processing circuit 720 executes logic (e.g., instructions) to perform actions (e.g., processing), such as producing an output from an input. The logic executed by the processing circuit 720 may be encoded in one or more tangible non-temporary computer-readable media (e.g., memory 730). For example, the logic may be a computer program, software, etc. The embodiments may include computer-executable instructions and / or instructions that can be executed by a computer. In certain embodiments, the processing of the embodiments may be performed by one or more computer-readable media containing, storing, implementing and / or encoding a computer program and / or the stored and / or encoded computer program.

[0058] Memory 730 (or memory unit) stores information. Memory 730 may include one or more non-temporary, tangible, computer-readable, and / or computer-executable storage media. Examples of memory 730 include computer memory (e.g., RAM or ROM), mass storage media (e.g., hard disks), removable storage media (e.g., compact discs (CDs) or digital video discs (DVDs)), databases and / or network storage (e.g., servers), and / or other computer-readable media.

[0059] In this specification, a computer-readable non-temporary storage medium or medium may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, SD (Secure Digital) cards or drives, or other suitable computer-readable non-temporary storage mediums, or any suitable combination of two or more of these. The computer-readable non-temporary storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, as required.

[0060] In this specification, “or” is inclusive and non-exclusive unless expressly indicated otherwise or contextually indicated otherwise. Accordingly, in this specification, “A or B” means “A, B, or both” unless expressly indicated otherwise or contextually indicated otherwise. Also, “and” means joint and plural unless expressly indicated otherwise or contextually indicated otherwise. Accordingly, in this specification, “A and B” means “A and B, jointly, or individually” unless expressly indicated otherwise or contextually indicated otherwise. The scope of this disclosure includes all changes, substitutions, modifications, alterations and modifications to the exemplary embodiments described or illustrated herein as understandable to those skilled in the art. The scope of this disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, while this disclosure describes and illustrates each embodiment of this specification as including specific components, elements, features, functions, processes, or steps, any of these embodiments may include any combination or permutation of components, elements, features, functions, processes, or steps described or illustrated anywhere in this specification as understandable to those skilled in the art. Furthermore, any reference in the appended claims to a device or system or component thereof that is appropriately configured and arranged to perform a particular function and is operational or operable includes the system or component of such device, system, or component whether it is appropriately configured and arranged, operational or operable, or whether a particular function has been activated or deactivated. Furthermore, while this disclosure describes or illustrates certain embodiments as providing certain advantages, a particular embodiment may provide none, some, or all of these advantages.

Claims

1. A system for reconstructing an object using transition images, One or more processors, One or more computer-readable non-temporary storage media coupled to the one or more processors And when executed by one or more processors, it causes the device to perform processing. One or more computer-readable non-temporary storage media containing instructions, Includes, The aforementioned process is, A process of capturing one or more images of an object using a sensor, wherein the object is a transportation infrastructure component, a railway environment component, or a railway track component, A process of detecting the object in the image using an object detection algorithm, A process of identifying at least a first portion of the object in the first image and constructing a bounding box around at least a first portion of the object in the first image, The process involves identifying at least a second portion of the object in the second image and constructing a bounding box around at least a second portion of the object in the second image. A process by which one or more processors generate transition images from the images, wherein the transition images share at least one of the images and at least some pixels, and include one or more regions of the object, A process for determining an equivalent region between the aforementioned image and one or more of the aforementioned transition images, A process by which one or more processors reconstruct the object from at least one image and at least one transition image, A process to exclude the equivalent region from the reconstruction of the object. including, system.

2. The system according to claim 1, wherein the object is a road, railway, airway, waterway, canal, pipeline, terminal component, railway track, debris, pedestrian, animal, vegetation, joint, switch, frog, railhead, anchor, fastener, gauge plate, ballast, or sleeper.

3. The system according to claim 1, wherein the image includes at least a portion of one or more of the objects.

4. The system according to claim 1, wherein the bounding box is an outline that identifies the object.

5. The system according to claim 1, wherein at least one transition image shares a predetermined percentage of pixels with at least one of the images.

6. The system according to claim 1, wherein the transition image is generated by the overlapping of two or more images.

7. The system according to claim 1, wherein the equivalent region is a region in the image and the transition image that shares the same pixels.

8. The system according to claim 1, further comprising determining that both at least one of the images and at least one of the transition images include a shared region of the object.

9. The system according to claim 1, wherein a threshold for the equivalence or confidence score of the detection labels is used to determine that the first region of the object in the first transition image and the first region of the object in the first image are equivalent regions.

10. The system according to claim 1, wherein the reconstruction of the object represents the entire object.

11. A method for reconstructing an object using transition images, A step of capturing one or more images of an object using a sensor, wherein the object is a transportation infrastructure component, a railway environment component, or a railway track component, A step of detecting the object in the image using an object detection algorithm, The steps include identifying at least a first portion of the object in the first image and constructing a bounding box around at least a first portion of the object in the first image, The steps include identifying at least a second portion of the object in the second image and constructing a bounding box around at least a second portion of the object in the second image, A step of generating a transition image from the aforementioned image, wherein the transition image shares at least one of the aforementioned images with at least some pixels and includes one or more regions of the object, The steps include determining an equivalent region between the aforementioned image and one or more of the aforementioned transition images, The steps of reconstructing the object from at least one image and at least one transition image, A step of excluding the equivalent region from the reconstruction of the object. including, method.

12. The method according to claim 11, wherein the object is a road, railway, airway, waterway, canal, pipeline, terminal component, railway track, debris, pedestrian, animal, vegetation, joint, switch, frog, railhead, anchor, fastener, gauge plate, ballast, or sleeper.

13. The method according to claim 11, wherein the image includes at least a portion of one or more of the objects.

14. The method according to claim 11, wherein the bounding box is an outline that identifies the object.

15. The method according to claim 11, wherein at least one transition image shares a predetermined percentage of pixels with at least one of the images.

16. The method according to claim 11, wherein the transition image is generated by the overlapping of two or more images.

17. The method according to claim 11, wherein the equivalent region is a region in the image and the transition image that shares the same pixels.

18. The method according to claim 11, further comprising determining that both at least one of the images and at least one of the transition images include a shared region of the object.

19. The method according to claim 11, wherein a threshold for the equivalence or confidence score of the detection labels is used to determine that the first region of the object in the first transition image and the first region of the object in the first image are equivalent regions.

20. The method according to claim 11, wherein the reconstruction of the object represents the entire object.

Images