Calibration system for multiple image sensors of different spectra

Abstract

A calibration system for a plurality of image sensors includes a manufacturing apparatus, a first imaging sensor, a second imaging sensor located at a different location than the first imaging sensor and configured to capture at least one different wavelength than the first imaging sensor, a calibration member, and an imaging calibration module. The imaging calibration module is configured to obtain first and second images of the calibration member via the first and second imaging sensors, determine first and second pixel mappings between the first and second images and a common coordinate system according to positions of the calibration member in the first and second images, an image of the manufacturing object captured by the first imaging sensor is converted into a common coordinate system, and an image of the manufacturing object captured by the second imaging sensor is converted into a common coordinate system.

Description

[0001] introduction

[0002] The information provided in this section is for the purpose of generally presenting the context of the present disclosure. To the extent that it is described in this section, the work of the presently named inventors, and aspects that may not otherwise qualify as prior art at the time of filing, are neither explicitly nor implicitly admitted to be prior art against the present disclosure.

[0003] The present disclosure generally relates to a calibration system for multiple image sensors of different light spectra, including a manufacturing system having an infrared camera for thermal imaging and a visible light camera.

[0004] During the manufacture of an object (e.g., a vehicle component), different imaging sensors may be used to capture one or more images (e.g., image sequences) of the object at different spectral wavelengths, such as an infrared camera and a visible light camera for thermal imaging. The different imaging sensors are typically located at different positions relative to the object and at different azimuth angles relative to the object, resulting in captured images (or image sequences) having different sizes, resolutions, etc. Summary of the invention

[0005] A calibration system for multiple image sensors includes: a manufacturing device configured to perform a manufacturing operation on a manufacturing object; a first imaging sensor configured to capture an image of the manufacturing object; a second imaging sensor configured to capture an image of the manufacturing object, wherein the second imaging sensor is located at a different position from the first imaging sensor and is configured to capture at least one wavelength different from the first imaging sensor; a calibration artifact located within a plane of the manufacturing device; and an imaging calibration module. The imaging calibration module is configured to obtain a first image of the calibration artifact via the first imaging sensor, obtain a second image of the calibration artifact via the second imaging sensor, determine a first pixel mapping between the first image and a common coordinate system based on a position of the calibration artifact in the first image, determine a second pixel mapping between the second image and the common coordinate system based on a position of the calibration artifact in the second image, transform the image of the manufacturing object captured by the first imaging sensor to the common coordinate system based on the first pixel mapping, and transform the image of the manufacturing object captured by the second imaging sensor to the common coordinate system based on the second pixel mapping.

[0006] In other features, the imaging calibration module is configured to perform image processing on an image of the manufactured object captured by the first imaging sensor to identify a target region, and apply the target region to an image of the manufactured object captured by the second imaging sensor using a common coordinate system.

[0007] In other features, the imaging calibration module is configured to control the manufacturing device to perform a manufacturing operation on the manufacturing object according to the target area applied to the image of the manufacturing object captured by the second imaging sensor.

[0008] In other features, the manufacturing device is a weld inspection machine, and the manufacturing operations include weld inspection operations performed on the manufactured object.

[0009] In other features, the first imaging sensor is a visible light camera and the second imaging sensor is an infrared camera.

[0010] In other features, the imaging calibration module is configured to activate a light source to illuminate at least one of a front surface of the calibration object or a rear surface of the calibration object when a first image of the calibration object is captured using a visible light camera, and to thermally stimulate at least a portion of the calibration object when a second image of the calibration object is captured using an infrared camera.

[0011] In other features, the imaging calibration module is configured to capture a sequence of images while thermally actuating at least a portion of the calibration member, and select one of the sequence of images having a most uniform thermal profile as the second image for determining the second pixel map.

[0012] In other features, the front surface of the calibration object includes a material configured to absorb and emit heat in the form of infrared radiation.

[0013] In other features, the imaging calibration module is configured to apply infrared radiation to a front surface of the calibration object to uniformly heat the calibration object while capturing a second image of the calibration object using the infrared camera.

[0014] In other features, the material on the front surface of the calibration article includes a paint material configured to absorb and emit heat in the form of infrared radiation.

[0015] In other features, the calibration member includes a rectangular plate having a front side, a back side, at least one chamfered corner, and an array of openings defined from the front side to the back side.

[0016] In other features, the rectangular plate defines a thickness from a front side to a back side, and each opening and each edge of the rectangular plate is chamfered.

[0017] In other features, the first imaging sensor is at a different distance from the calibration object than the second imaging sensor.

[0018] In other features, the first imaging sensor is oriented at a different angle relative to the calibration object than the second imaging sensor.

[0019] A method for calibrating multiple image sensors includes obtaining a first image of a calibration piece via a first imaging sensor, the calibration piece being located within a plane of a manufacturing device configured to perform a manufacturing operation on a manufacturing object, obtaining a second image of the calibration piece via a second imaging sensor, wherein the second imaging sensor is located at a different position than the first imaging sensor and is configured to capture at least one wavelength different from the first imaging sensor, determining a first pixel mapping between the first image and a common coordinate system based on a position of the calibration piece in the first image, determining a second pixel mapping between the second image and the common coordinate system based on a position of the calibration piece in the second image, converting an image of the manufacturing object captured by the first imaging sensor to the common coordinate system based on the first pixel mapping, and converting an image of the manufacturing object captured by the second imaging sensor to the common coordinate system based on the second pixel mapping.

[0020] In other features, the method includes performing image processing on an image of the manufactured object captured by the first imaging sensor to identify a target region, and applying the target region to an image of the manufactured object captured by the second imaging sensor using a common coordinate system.

[0021] In other features, the method includes controlling the manufacturing device to perform a manufacturing operation on the manufacturing object based on the target area applied to the image of the manufacturing object captured by the second imaging sensor.

[0022] In other features, the manufacturing device is a weld inspection machine, and the manufacturing operations include weld inspection operations performed on the manufactured object.

[0023] In other features, the first imaging sensor is a visible light camera and the second imaging sensor is an infrared camera.

[0024] In other features, the method includes activating a light source to illuminate at least one of a front side of the calibration article or a back side of the calibration article while capturing a first image of the calibration article using a visible light camera, and thermally stimulating at least a portion of the calibration article while capturing a second image of the calibration article using an infrared camera.

[0025] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims and drawings.The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.

[0026] The present invention also includes the following solutions.

[0027] Solution 1. A calibration system for multiple image sensors, the calibration system comprising:

[0028] a manufacturing device configured to perform a manufacturing operation on a manufacturing object;

[0029] a first imaging sensor configured to capture an image of a manufactured object;

[0030] a second imaging sensor configured to capture an image of the manufactured object, wherein the second imaging sensor is located at a different location than the first imaging sensor and is configured to capture at least one different wavelength than the first imaging sensor;

[0031] A calibration piece located within the plane of the manufacturing device; and

[0032] The imaging calibration module is configured as follows:

[0033] Acquiring a first image of the calibration object via a first imaging sensor;

[0034] obtaining a second image of the calibration object via a second imaging sensor;

[0035] Determining a first pixel mapping between the first image and the common coordinate system according to the position of the calibration member in the first image;

[0036] determining a second pixel mapping between the second image and the common coordinate system based on the position of the calibration member in the second image;

[0037] transforming an image of the manufactured object captured by the first imaging sensor to a common coordinate system according to the first pixel map; and

[0038] An image of the manufactured object captured by the second imaging sensor is transformed to the common coordinate system according to the second pixel map.

[0039] Solution 2. The system according to solution 1, wherein the imaging calibration module is configured to:

[0040] performing image processing on an image of the manufactured object captured by the first imaging sensor to identify a target area; and

[0041] The common coordinate system is used to apply the target region to an image of the manufactured object captured by the second imaging sensor.

[0042] Option 3. A system according to Option 2, wherein the imaging calibration module is configured to control the manufacturing device to perform the manufacturing operation on the manufacturing object based on the target area applied to the image of the manufacturing object captured by the second imaging sensor.

[0043] Option 4. The system according to Option 3, wherein:

[0044] The manufacturing device is a weld inspection machine; and

[0045] Manufacturing operations include weld inspection operations performed on manufactured objects.

[0046] Option 5. The system according to Option 1, wherein:

[0047] The first imaging sensor is a visible light camera; and

[0048] The second imaging sensor is an infrared camera.

[0049] Solution 6. The system according to Solution 5, wherein the imaging calibration module is configured to:

[0050] activating a light source to illuminate at least one of a front surface of the calibration article or a back surface of the calibration article while capturing a first image of the calibration article using the visible light camera; and

[0051] At least a portion of the calibration article is thermally actuated while capturing a second image of the calibration article using the infrared camera.

[0052] Solution 7. The system according to Solution 6, wherein the imaging calibration module is configured to:

[0053] capturing a sequence of images while thermally actuating at least a portion of the calibration element; and

[0054] The one of the sequence of images having the most uniform thermal profile is selected as the second image for determining the second pixel map.

[0055] Option 8. A system according to Option 7, wherein the front surface of the calibration piece includes a material configured to absorb and emit heat in the form of infrared radiation.

[0056] Option 9. A system according to Option 8, wherein the imaging calibration module is configured to apply infrared radiation to the front surface of the calibration piece to uniformly heat the calibration piece, while using the infrared camera to capture a second image of the calibration piece.

[0057] Option 10. A system according to Option 8, wherein the material on the front surface of the calibration piece includes a coating material configured to absorb and emit heat in the form of infrared radiation.

[0058] Option 11. A system according to Option 1, wherein the calibration member includes a rectangular plate having a front side, a back side, at least one chamfer, and an array of openings defined from the front side to the back side.

[0059] Option 12. A system according to Option 11, wherein the rectangular plate defines a thickness from the front side to the back side, and each opening and each edge of the rectangular plate is chamfered.

[0060] Option 13. A system according to Option 1, wherein the first imaging sensor is at a different distance from the calibration piece than the second imaging sensor.

[0061] Embodiment 14. The system of embodiment 1, wherein the first imaging sensor is oriented at a different angle relative to the calibration member than the second imaging sensor.

[0062] Solution 15. A method for calibrating a plurality of image sensors, the method comprising:

[0063] obtaining, via a first imaging sensor, a first image of a calibration object located within a plane of a manufacturing device configured to perform a manufacturing operation on a manufacturing object;

[0064] obtaining a second image of the calibration article via a second imaging sensor, wherein the second imaging sensor is located at a different position than the first imaging sensor and is configured to capture at least one wavelength different from the first imaging sensor;

[0065] determining a first pixel mapping between the first image and the common coordinate system according to the position of the calibration member in the first image;

[0066] determining a second pixel mapping between the second image and the common coordinate system based on the position of the calibration member in the second image;

[0067] transforming an image of the manufactured object captured by the first imaging sensor to a common coordinate system according to the first pixel map; and

[0068] An image of the manufactured object captured by the second imaging sensor is transformed to the common coordinate system according to the second pixel map.

[0069] Solution 16. The method according to Solution 15, further comprising:

[0070] performing image processing on an image of the manufactured object captured by the first imaging sensor to identify a target area; and

[0071] The target region is applied to an image of the manufactured object captured by the second imaging sensor using the common coordinate system.

[0072] Option 17. The method according to Option 16 further includes controlling a manufacturing device to perform a manufacturing operation on the manufacturing object according to a target area applied to the image of the manufacturing object captured by the second imaging sensor.

[0073] Scheme 18. The method according to Scheme 17, wherein:

[0074] The manufacturing device is a weld inspection machine; and

[0075] Manufacturing operations include weld inspection operations performed on manufactured objects.

[0076] Scheme 19. The method according to Scheme 15, wherein:

[0077] The first imaging sensor is a visible light camera; and

[0078] The second imaging sensor is an infrared camera.

[0079] Solution 20. The method according to solution 19, further comprising:

[0080] activating a light source to illuminate at least one of a front side of the calibration article or a back side of the calibration article while capturing a first image of the calibration article using the visible light camera; and

[0081] At least a portion of the calibration article is thermally actuated while capturing a second image of the calibration article using the infrared camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The present disclosure will become more fully understood from the detailed description and accompanying drawings, in which:

[0083] Figure 1 is a diagram of an example calibration system for multiple imaging sensors.

[0084] Figure 2 is Figure 1 Figure 2 shows an example calibration part used in the calibration system of FIG.

[0085] Figure 3 is a block diagram illustrating an example process for mapping identified target regions between images captured by different imaging sensors using a common coordinate system.

[0086] Figure 4 is a flow chart depicting an example process for calibrating different image sensors to a common coordinate system.

[0087] Figure 5 is a flow chart depicting an example process for mapping identified target regions between images captured by different imaging sensors using a common coordinate system.

[0088] In the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0089] Some example embodiments described herein include systems and methods for fusing image data streams (or image sequences) from imaging sensors operating at different spectral wavelengths to enable multispectral region of interest selection and mapping between sensors. For example, a hybrid calibration facilitates conversion of different sensors to a common world coordinate system (WCS), wherein the hybrid calibration includes calibration patterns detectable at different spectral wavelengths of different sensors.

[0090] Once the sensor data has been appropriately mapped to a common WCS, region of interest identification based on image data from one sensor can be mapped to image data from another sensor (and vice versa) to improve the robustness of analysis of a given object observed in all spectral wavelengths of the sensors.

[0091] For example, weld locations may be identified with greater accuracy in images captured by a visible light camera (e.g., due to the higher resolution of the visible light camera), and the identified weld locations may be applied to the same locations in the infrared image using a common coordinate system mapping. While some example embodiments described herein relate to mid-wavelength infrared (MWIR) sensors and visible light sensors, other example embodiments may include imager sensors in other spectral ranges.

[0092] Figure 1 A calibration system 10 for a plurality of image sensors is provided. The calibration system includes a manufacturing device 16 configured to perform a manufacturing operation on a manufacturing object 20. For example, the manufacturing device 16 may be a welding machine configured to perform a welding operation on a component for assembling a vehicle.

[0093] The calibration system 10 includes a first imaging sensor 12 configured to capture an image of a manufactured object 20, and a second imaging sensor 14 configured to capture an image of the manufactured object 20. The second imaging sensor 14 is located at a different location than the first imaging sensor 12. In various embodiments, the calibration system 10 first acquires data from a calibration object, such as a calibration part 18. The actual part is then moved into view so that data can be acquired.

[0094] For example, the first imaging sensor 12 and the second imaging sensor 14 may be located at different distances from the manufactured object 20 (or may be located at the same distance), may be oriented at different angles relative to the manufactured object 20 (or may be located at the same azimuth), etc. Figure 1 A first imaging sensor 12 is shown having an image capture field 24 that is oriented at a different angle than the image capture field 22 of the second imaging sensor 14 .

[0095] First imaging sensor 12 is configured to capture at least one different wavelength than second imaging sensor 14. For example, first imaging sensor 12 may be an optical camera configured to capture visible light images of manufactured object 20, and second imaging sensor 14 may be an infrared camera configured to capture infrared images of manufactured object 20.

[0096] The calibration piece 18 is located in the plane of the manufacturing device 16. For example, the manufacturing device 16 may include a conveyor or the like configured to move the manufacturing object 20 to a position where the first imaging sensor 12 and the second imaging sensor 14 capture images of the manufacturing object 20. The calibration piece 18 may be located in substantially the same position where the manufacturing object 20 is typically located, so that the first imaging sensor 12 and the second imaging sensor 14 can capture images of the calibration piece 18 at a position similar to the manufacturing object 20.

[0097] The calibration system 10 further includes an imaging calibration module 26 . The imaging calibration module 26 is configured to obtain a first image of the calibration object 18 via the first imaging sensor 12 , and obtain a second image of the calibration object 18 via the second imaging sensor 14 .

[0098] The imaging calibration module 26 is configured to determine a first pixel mapping between the first image and a common coordinate system based on the position of the calibration member 18 in the first image. For example, a common world coordinate system (WCS) can be used to calibrate the first imaging sensor 12 and the second imaging sensor 14 to facilitate conversion of image data between images captured by each imaging sensor.

[0099] Any suitable technique may be used to generate a pixel mapping from the image of the calibration article 18 to the common coordinate system, which may account for sensor format, measured angles relative to the plane of the calibration article 18, optical curvature, etc. The imaging calibration module 26 is configured to determine a second pixel mapping between the second image and the common coordinate system based on the position of the calibration article 18 in the second image.

[0100] Imaging calibration module 26 is further configured to transform images of manufactured object 20 captured by first imaging sensor 12 to a common coordinate system according to a first pixel map and to transform images of manufactured object 20 captured by second imaging sensor 14 to a common coordinate system according to a second pixel map.

[0101] For example, imaging calibration module 26 may be configured to perform image processing on an image of manufactured object 20 captured by first imaging sensor 12 to identify a target region, and apply the target region to an image of manufactured object 20 captured by second imaging sensor 14 using a common coordinate system. In various embodiments, imaging calibration module 26 may perform image processing using images captured from a visible light camera and identify a target region of interest (ROI), such as identifying a weld location, where image processing is more accurate on visible light camera images having greater detail.

[0102] The imaging calibration module 26 may then use the common coordinate system to map the identified target area in the visible light image to the same target area in the infrared image captured by the infrared camera to highlight the weld location in the infrared image. For example, the infrared image may have lower accuracy, resolution, contrast, etc. than the visible light image, which may reduce the accuracy of image processing of the infrared image (e.g., making it more difficult to identify the weld location in the infrared image).

[0103] The use of a common coordinate system allows for more accurate determination of target areas in visible light images, which can then be converted into infrared images for use in performing operations or inspections on manufactured parts. For example, imaging calibration module 26 can be configured to control manufacturing device 16 to perform manufacturing operations on manufacturing object 20 based on target areas applied to images of manufacturing object 20 captured by second imaging sensor 14.

[0104] The imaging calibration module 26 can be configured to activate a light source to illuminate at least one of the front surface of the calibration piece 18 or the back surface of the calibration piece 18 while using the visible light camera to capture a first image of the calibration piece 18. For example, a backlight of an LED or another light source can be activated to increase the contrast of the calibration piece features captured by the visible light camera. The front array of LEDs can illuminate the manufactured object 20 to increase the accuracy and detail of the image of the manufactured object captured by the visible light camera.

[0105] The imaging calibration module 26 can be configured to thermally stimulate at least a portion of the calibration member 18 while using the infrared camera to capture a second image of the calibration member 18. For example, the front surface of the calibration member 18 can include a material with high absorptivity and emissivity, such as a black paint material. The imaging calibration module 26 can be configured to apply light to the front surface of the calibration member to uniformly heat the calibration member 18 while using the infrared camera to capture a second image of the calibration member 18.

[0106] In some example embodiments, imaging calibration module 26 is configured to capture a sequence of images while thermally actuating calibration member 18. Imaging calibration module 26 may then select one of the sequence of images having the most uniform thermal profile as a second image for determining a second pixel map. For example, the heating distribution on calibration member 18 may vary in different areas depending on how calibration member 18 is thermally actuated. Imaging calibration module 26 may capture a sequence of images over time and select which image has the most uniform thermal distribution for calibration purposes.

[0107] Although some example embodiments are described herein with respect to visible light cameras and infrared cameras, other embodiments may use other suitable types of image sensors that can capture different wavelengths of light.

[0108] In various embodiments, each sensor may have a line of sight to the calibration piece, with the entire calibration piece visible and in focus.Each imaging sensor may have a sensor format with a sufficiently high pixel density to observe the associated calibration piece pattern.

[0109] Each sensor is configured to detect the calibration piece, such as observing the calibration piece via thermal excitation of an infrared camera. Each sensor can observe an illumination condition or excitation scheme that produces a spatially uniform spectral emission. In some example embodiments, the calibration piece can have a geometry that is tuned depending on which sensor(s) deviate from normal incidence. For example, image artifacts related to the geometry caused by a particular sensor can be reduced, such as by using chamfers on the edges and openings of the calibration piece to reduce edge effects and shadow effects in imaging that deviate from normal incidence.

[0110] In some example embodiments, the lighting, calibration pieces, imaging sensors, etc. may be located at fixed locations. Suitable materials may be selected for the calibration pieces, such as matte black for the infrared spectrum when using an infrared camera for thermal imaging. Each calibration piece may be designed to be detectable by all imaging sensors in the system.

[0111] Figure 2 is Figure 1 FIG. 1 is a diagram of an example calibration part used in the calibration system 10 of FIG. Figure 2 As shown in FIG, the index 18 has a rectangular (eg, square) shape with an array of openings 32. The openings 32 are defined from a front side 30 of the index 18 to a back side of the index.

[0112] The opening 32 may have any suitable shape, such as circular, rectangular, triangular, etc. Figure 2 An array having six rows and six columns of openings 32 is illustrated, but other example embodiments may have other suitable calibrated patterns.

[0113] The opening 32 can allow for pattern detection in visible light image capture (e.g., by shining a backlight through the opening), as well as pattern detection in infrared light images (e.g., where the material of the calibration piece 18 has an increased thermal profile when thermally stimulated, but the opening 32 does not). As mentioned above, other embodiments can use image sensors other than a visible light camera and an infrared camera for infrared images. The calibration piece 18 can use a different pattern type than the opening 32, as long as the same pattern is detectable in both image sensor types. In some examples, the calibration piece can have a pattern that enables the sensor to understand the orientation of the calibration piece in three-dimensional space. The pattern can have a high contrast with the background. For thermal imaging, one example includes creating a pattern from a hole, but in other examples, the pattern can be created from two different materials.

[0114] like Figure 2 As shown in , calibration piece 18 includes chamfer 34 (which may be cut at a 45 degree angle, etc.). Chamfer 34 allows the orientation of the pattern of calibration piece 18 to be determined, and may be useful in situations where different image sensors may have different rotational orientations. Other example embodiments may have more or fewer chamfers, calibration pieces with different shapes, other indicators for calibrating the orientation of an item, etc.

[0115] In some example embodiments, the rectangular plate of the calibration member 18 defines a thickness from the front side to the back side, so the image sensor may be able to detect the pattern or calibration in three dimensions. If the image sensor is positioned at a different angle relative to the calibration member 18, each opening 32 and each edge of the rectangular plate may be chamfered to reduce distortion and artifacts caused by "shadows" / off-axis perspective.

[0116] In some example embodiments, the calibration artifact facilitates coordinate system fusion between multiple sensors in different spectral regimes simultaneously. This is achieved by designing the calibration artifact to be detectable across all image sensor spectral regimes in the system.

[0117] For example, the calibration element may include features that achieve high contrast between the workpiece calibration pattern and the background. The calibration element may have materials and geometries that reduce or minimize objectionable image features caused by spectral reflections, shadows, noise, perspective, etc.

[0118] The calibration piece can achieve uniform illumination and / or emission of the workpiece calibration pattern and background, and can increase or maximize the workpiece pattern contrast for each image sensor spectral system in the system. Example calibration piece excitation / illumination methods can achieve and optimize feature detection of the calibration piece within the spectral system of the image sensors used in the system. The excitation strategy can depend on the potential impact of the calibration piece due to excitation in the spectral system of interest (e.g., by implementing an excitation sequence strategy when switching between spectral systems of different image sensors).

[0119] Figure 3 1 is a block diagram illustrating an example process 100 for mapping identified target regions between images captured by different imaging sensors using a common coordinate system. Figure 3 As shown in , a first image 102 is captured by a first image sensor, such as a visible light optical camera.

[0120] The second image 104 is captured by a second image sensor, such as an infrared camera. The first image 102 and the second image 104 can have different sizes, different rotational orientations, etc. In various embodiments, the calibration member 18 does not move between image captures. In other words, the image data should be acquired from a fixed system position.

[0121] The first image 102 and the second image 104 are mapped to the common coordinate view 106. For example, a pixel mapping conversion process may use calibration patterns in the first image 102 and the second image 104 to map each image to the common coordinate view 106 based on a stored relationship between the calibration patterns and the common coordinate view 106.

[0122] At 108, the imaging calibration module acquires optical data of a manufacturing object, such as an object for welding. At 110, the imaging calibration module determines a segment welding region (or target region of interest). This can be performed by performing image processing on a visible light image, etc.

[0123] At 112, the imaging calibration module transforms the visible light image to a common coordinate view. Figure 3 It is illustrated that the object region of interest is determined before converting the visible light image into the common coordinate view, but in other example embodiments, the system may convert the visible light image into the common coordinate view before determining the object region of interest in the visible light image.

[0124] The imaging calibration module is configured to acquire object infrared (IR) data at 114, such as via infrared image capture of a manufactured object. At 116, control is performed to transform the ROI found in the visible light image to infrared data using a common world coordinate view mapping. For example, the identified ROI from the visible light image can be applied to the infrared image to identify the welding position in the infrared image for control inspection for performing welding operations, etc. In various embodiments, before applying the identified ROI from the visible light image, the infrared image capture of the object can be converted to a common coordinate view, and the identified ROI can be mapped from the common coordinate view to infrared image coordinates, etc. based on pixel mapping calibration. In some examples, the infrared image data can provide a better (e.g., more accurate) ROI selection means. In various embodiments, the infrared image data can be used to select the ROI, and the selected ROI can then be applied to the visible light image.

[0125] Figure 4 is a flow chart depicting an example process for calibrating different image sensors to a common coordinate system. Figure 4 The process illustrated in the figure can be done by, for example Figure 1 The imaging calibration module 26 is used to perform the operation.

[0126] At 404, the process begins by placing a calibration object in view of the two image sensors. Then, at 408, the imaging calibration module is configured to apply a first imaging sensor excitation, such as by turning on an LED backlight. In some examples, the sensor excitation can be built into the calibration object itself. For example, the calibration object can include an embedded heating coil for a thermal excitation system, or the LED backlight can be part of the calibration object.

[0127] At 412, the imaging calibration module is configured to capture first imaging sensor data via a first imaging sensor (e.g., a visible light camera). At 416, the imaging calibration module is configured to apply a second imaging sensor excitation to the calibration member, such as a thermal excitation calibration member. The imaging calibration module then captures second imaging sensor data via a second imaging sensor (e.g., an infrared camera).

[0128] At 424, the imaging calibration module is configured to identify a pattern of the calibration member in the first image data.Then, at 428, the imaging calibration module generates a first pixel map from the identified first image pattern to common coordinates.

[0129] At 432, control is configured to select a first of a sequence of image captures from the second imaging sensor data. For example, the imaging sensor may capture a series of images, wherein control then attempts to select one of the images in which the calibration piece is uniformly heated. Figure 4The second imaging sensor data comprising a sequence of images is illustrated, but in other example embodiments, the second imaging sensor data may comprise a single snapshot from the second sensor. In that case, control may proceed to 444 to generate a second pixel map based on a single snapshot from the second imaging sensor data.

[0130] If the second imaging sensor data does include an image sequence, then at 436, control determines whether the calibration piece pattern is detected in the image capture of the selected image due to approximately uniform heating. If not, control proceeds to 440 to select the next capture from the second imaging sensor data sequence and returns to 436 to determine whether the next selected image has uniform heating to adequately detect the calibration piece pattern. Once the appropriate image capture is selected at 436, control proceeds to 444 to generate a second pixel map from the identified second image pattern to a common coordinate view (e.g., a common world coordinate system (WCS)).

[0131] At 448, control is configured to store the determined first pixel map for application to future object images captured by the first imaging sensor. At 452, control is configured to store the determined second pixel map for application to future object images captured by the second imaging sensor.

[0132] Figure 5 is a flow chart depicting an example process for mapping an identified target area between images captured by different imaging sensors using a common coordinate system. At 504, control begins by placing a manufactured object in the views of two imaging sensors.

[0133] At 508, control applies a first imaging sensor stimulus, such as an illumination array, to the manufactured object. Control then captures first imaging sensor data, such as a visible light image of the manufactured object, at 512.

[0134] At 516 , control is configured to apply a second imaging sensor stimulus to the manufactured object, such as thermally stimulating the manufactured object. Control then captures second imaging sensor data, such as infrared image data of the manufactured object, at 520 .

[0135] The imaging calibration module is configured to map the first image to the common coordinate system at 524 and map the second image to the common coordinate system at 528. In some examples, one of the images may already be mapped to the common coordinate system such that only one of the images needs to be mapped to the common coordinate system.

[0136] Then, at the portion at 532, control determines which image will be used for the region of interest. For example, in some embodiments, the visible light image may provide an image with higher accuracy for ROI selection, and the ROI selected based on the visible light image may be applied to the infrared image. In other examples, the infrared image may provide an image with higher accuracy for ROI selection, and the ROI selected based on the infrared image may be applied to the visible light image.

[0137] If the first image is used for region of interest detection at 536, control proceeds to 540 to determine the region of interest based on the image from the first imaging sensor. Then, at 544, control applies the identified region of interest to the second image from the second imaging sensor using the common coordinate system.

[0138] Alternatively, if control determines at 536 that the first image is not to be used for region of interest detection, control proceeds to 548 to determine the region of interest based on the second image from the second imaging sensor. Then, at 552, control applies the identified region of interest to the first image from the first imaging sensor using the common coordinates.

[0139] After applying the region of interest to another image using the common coordinate system, control performs a manufacturing operation on the object based on the identified region of interest at 556. For example, control may perform a weld inspection operation on the component based on the identified weld locations, or specify whether the part has passed a quality inspection.

[0140] The foregoing description is merely illustrative in nature and is by no means intended to limit the present disclosure, its application or use. The broad teachings of the present disclosure can be implemented in a variety of forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be so limited, because other modifications will become apparent when studying the drawings, the specification and the following claims. It should be understood that one or more steps in the method can be performed in different orders (or simultaneously) without changing the principles of the present disclosure. In addition, although each embodiment is described above as having certain features, any one or more of those features described with respect to any embodiment of the present disclosure can be implemented in any other embodiment and / or combined with the features of any other embodiment, even if that combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the replacement of one or more embodiments with each other is still within the scope of the present disclosure.

[0141] Various terms are used to describe the spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.), including "connected," "engaged," "coupled," "adjacent," "next to," "on top of," "above," "below," and "disposed." Unless explicitly described as "direct," when a relationship between a first and a second element is described in the above disclosure, that relationship can be a direct relationship in which there are no other intervening elements between the first and second elements, but can also be an indirect relationship in which there are one or more intervening elements (spatially or functionally) between the first and second elements. As used herein, the phrase at least one of A, B, and C should be interpreted to mean a logical (A or B or C), using a non-exclusive logical "or," and should not be interpreted to mean "at least one of A, at least one of B, and at least one of C."

[0142] In a diagram, the direction of the arrow indicated by the arrow generally indicates the flow of information (such as data or instructions) of interest to the diagram. For example, when element A and element B exchange multiple types of information, but the information transmitted from element A to element B is relevant to the diagram, the arrow may point from element A to element B. This one-way arrow does not mean that no other information is transmitted from element B to element A. In addition, for information sent from element A to element B, element B may send a request for the information or an acknowledgment of its receipt to element A.

[0143] In this application, including the definitions below, the term "module" or the term "controller" may be replaced with the term "circuit". The term "module" may refer to an application-specific integrated circuit (ASIC); a digital, analog or mixed analog / digital discrete circuit; a digital, analog or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated or group) that executes code; a memory circuit (shared, dedicated or group) that stores code executed by the processor circuit; other suitable hardware components that provide the functionality; or a combination of some or all of the above (e.g., in a system on a chip), is a part of them, or includes them.

[0144] The module may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functions of any given module of the present disclosure may be distributed among multiple modules connected via the interface circuit. For example, multiple modules may allow load balancing. In another example, a server (also referred to as a remote or cloud) module may perform some functions on behalf of a client module.

[0145] The term "code" used above may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" includes a single processor circuit that executes some or all code from multiple modules. The term "group processor circuit" includes a processor circuit that combines with additional processor circuits to execute some or all code from one or more modules. References to multiple processor circuits include multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term "shared memory circuit" includes a single memory circuit that stores some or all code from multiple modules. The term "group memory circuit" includes a memory circuit that combines with additional memory to store some or all code from one or more modules.

[0146] The term "memory circuit" is a subset of the term "computer-readable medium". The term "computer-readable medium" as used herein does not include transient electrical or electromagnetic signals propagated through a medium (e.g., on a carrier wave); therefore, the term "computer-readable medium" may be considered to be tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuits (e.g., flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (e.g., static random access memory circuits or dynamic random access memory circuits), magnetic storage media (e.g., analog or digital magnetic tape or hard disk drives), and optical storage media (e.g., CDs, DVDs, or Blu-ray discs).

[0147] The apparatus and methods described in this application may be implemented in part or in whole by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions contained in a computer program. The functional blocks, flow chart components, and other elements described above serve as software specifications that can be converted into computer programs by the routine work of a skilled technician or programmer.

[0148] The computer program includes processor executable instructions stored on at least one non-transitory tangible computer readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0149] A computer program may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated by a compiler from source code; (iv) source code executed by an interpreter; (v) source code compiled and executed by a real-time compiler, etc. By way of example only, the source code may be compiled using languages ​​including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language Fifth Revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK, and The grammar of the language is used to write it.

Claims

1. A calibration system for multiple image sensors, the calibration system comprising: a manufacturing device configured to perform a manufacturing operation on a manufacturing object; a first imaging sensor configured to capture an image of a manufactured object; a second imaging sensor configured to capture an image of the manufactured object, wherein the second imaging sensor is located at a different location than the first imaging sensor and is configured to capture at least one different wavelength than the first imaging sensor; A calibration piece located in the plane of the manufacturing device; and The imaging calibration module is configured as follows: Acquiring a first image of the calibration object via a first imaging sensor; obtaining a second image of the calibration object via a second imaging sensor; Determining a first pixel mapping between the first image and the common coordinate system according to the position of the calibration member in the first image; determining a second pixel mapping between the second image and the common coordinate system based on the position of the calibration member in the second image; transforming an image of the manufactured object captured by the first imaging sensor to a common coordinate system according to the first pixel map; and An image of the manufactured object captured by the second imaging sensor is transformed to the common coordinate system according to the second pixel map.

2. The system according to claim 1, wherein the imaging calibration module is configured to: performing image processing on an image of the manufactured object captured by the first imaging sensor to identify a target area; and The common coordinate system is used to apply the target region to an image of the manufactured object captured by the second imaging sensor.

3. The system of claim 2, wherein the imaging calibration module is configured to control the manufacturing device to perform the manufacturing operation on the manufacturing object according to the target area applied to the image of the manufacturing object captured by the second imaging sensor.

4. The system of claim 3, wherein: The manufacturing device is a weld inspection machine; and Manufacturing operations include weld inspection operations performed on manufactured objects.

5. The system of claim 1, wherein: The first imaging sensor is a visible light camera; and The second imaging sensor is an infrared camera.

6. The system according to claim 5, wherein the imaging calibration module is configured to: activating a light source to illuminate at least one of a front surface of the calibration article or a back surface of the calibration article while capturing a first image of the calibration article using the visible light camera; and At least a portion of the calibration article is thermally actuated while capturing a second image of the calibration article using the infrared camera.

7. The system according to claim 6, wherein the imaging calibration module is configured to: capturing a sequence of images while thermally actuating at least a portion of the calibration element; and The one of the sequence of images having the most uniform thermal profile is selected as the second image for determining the second pixel map.

8. The system according to claim 7, wherein: The front surface of the calibration object includes a material configured to absorb and emit heat in the form of infrared radiation.

9. The system according to claim 8, wherein: The imaging calibration module is configured to apply infrared radiation to the front surface of the calibration object to uniformly heat the calibration object, while using the infrared camera to capture a second image of the calibration object.

10. A method for calibrating a plurality of image sensors, the method comprising: obtaining, via a first imaging sensor, a first image of a calibration object located within a plane of a manufacturing device configured to perform a manufacturing operation on a manufacturing object; obtaining a second image of the calibration article via a second imaging sensor, wherein the second imaging sensor is located at a different position than the first imaging sensor and is configured to capture at least one wavelength different from the first imaging sensor; determining a first pixel mapping between the first image and the common coordinate system according to the position of the calibration member in the first image; determining a second pixel mapping between the second image and the common coordinate system based on the position of the calibration member in the second image; transforming an image of the manufactured object captured by the first imaging sensor to a common coordinate system according to the first pixel map; and An image of the manufactured object captured by the second imaging sensor is transformed to the common coordinate system according to the second pixel map.

Images