A near-infrared camera multi-frame time-division welding observation device, system and method
By using a near-infrared camera time-sharing welding observation device, the problems of complex welding torch end structure and unified expression of visual information in multi-sensor schemes are solved. It realizes stable acquisition and unified expression of multiple types of visual information, and supports online welding measurement and trajectory tracking.
Patent Information
- Application Number
- CN202610965925.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-01
- Publication Date
- 2026-11-17
AI Technical Summary
In existing robotic arc welding vision systems, multi-sensor solutions result in a large number of sensors at the welding torch end and a complex installation structure, making it difficult to uniformly express multiple types of visual information under the same imaging link and coordinate system. Furthermore, structured light measurement and illumination enhancement observation interfere with each other, and time-division observation results are difficult to uniformly express in motion welding scenarios.
A near-infrared camera time-sharing welding observation device is adopted. Through the synchronous control of the near-infrared camera, near-infrared laser and near-infrared pulse illumination source, near-infrared images are acquired in a time-sharing exposure sequence. The molten pool state information, two-dimensional information of the welding neighborhood and three-dimensional geometric information of the weld or weld bead are obtained respectively. The results are unified to the same coordinate system through frame type identification and spatiotemporal alignment mechanism.
It achieves stable acquisition of multiple types of visual information under the same near-infrared camera, reduces the structural complexity of the welding torch end observation device, reduces light source crosstalk, and provides a unified observation basis for online welding measurement, trajectory tracking, and deviation correction.
Smart Images

Figure CN123103117A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of welding visual perception technology, and more specifically, to a near-infrared camera multi-frame time-division welding observation device, a welding and observation system, and a welding observation method. Background Technology
[0002] In robotic arc welding, the state of the molten pool, the position of the welding wire tip, the weld boundary, the groove geometry, the interlayer weld bead profile, and the post-weld bead formation all affect welding trajectory planning, welding torch posture adjustment, and welding quality stability. Therefore, welding vision systems typically need to acquire multiple types of observation information simultaneously, including near-infrared images for judging the state of the molten pool or welding area, two-dimensional image information for identifying the welding wire tip or weld boundary, and structured light images for three-dimensional measurement of the weld, groove, or weld bead.
[0003] In existing robotic arc welding vision systems, the aforementioned various types of visual information are typically acquired by different sensors. For example, a line laser sensor is used to acquire the three-dimensional geometric information of the weld, bevel, or weld bead in front of the welding torch; a molten pool camera is used to observe the molten pool area; and a weld bead camera or auxiliary lighting camera is used to acquire information about the welding wire tip, weld boundary, weld bead boundary, or post-weld formation. While this multi-sensor approach can accomplish different observation tasks separately, it results in a large number of sensors at the welding torch tip, a complex installation structure, and an increased number of cables and protective components, which is detrimental to miniaturized integration and applications in confined welding spaces.
[0004] Meanwhile, multi-sensor solutions typically require multi-camera extrinsic parameter calibration, multi-sensor time synchronization, and coordinate registration between different data sources. During welding, the welding torch tip is affected by robot acceleration and deceleration, cable dragging, heat input, fumes, spatter, and vibration. The relative installation relationship between different sensors is prone to slight changes, which in turn introduces extrinsic parameter drift and registration errors, affecting the unified expression of molten pool state information, two-dimensional boundary information, and three-dimensional geometric information.
[0005] Existing active weld pool observation schemes typically employ short exposures, high-peak-pulse light sources, and narrow-band filters to suppress welding arc light and acquire images of the weld pool area. These schemes primarily focus on weld pool condition observation and struggle to simultaneously obtain two-dimensional boundary information of the weld or weld bead, as well as three-dimensional geometric information of the weld, bevel, or weld bead. When weld pool condition observation, two-dimensional identification of the welding neighborhood, and three-dimensional measurement using structured light need to jointly serve robot closed-loop control, the challenge of consistently acquiring multiple types of visual information within the same imaging link and coordinate system remains.
[0006] Furthermore, if both line laser and near-infrared illumination sources are activated simultaneously within the same camera's field of view, the enhanced background light may reduce the contrast of the line laser stripes relative to the background. The line laser stripes and their reflection or scattering on the metal surface may also affect the identification of wire ends, weld boundaries, or weld bead boundaries in the enhanced illumination image. Therefore, simply placing the line laser and illumination source within the same camera's field of view makes it difficult to stably complete structured light 3D measurements and enhanced illumination observations.
[0007] Therefore, it is necessary to provide a near-infrared camera multi-frame time-division welding observation device, welding and observation system, and welding observation method to acquire molten pool state information, two-dimensional information of the welding neighborhood, and three-dimensional geometric information of the weld or weld bead in time division under the same imaging optical path using the same near-infrared camera. Frame identification and spatiotemporal alignment are performed to address the time difference in robot movement caused by time division acquisition, thereby uniformly transforming at least some observation results to the same coordinate system, providing a stable visual observation basis for online welding measurement, trajectory tracking, and deviation correction. Summary of the Invention
[0008] The main objective of this invention is to provide a near-infrared camera multi-frame time-division welding observation device, welding and observation system, and welding observation method to solve the problems of complex multi-sensor structure, difficult coordinate registration, mutual interference between structured light measurement and illumination enhancement observation, and difficulty in uniformly expressing time-division observation results in motion welding scenarios in existing welding vision systems.
[0009] To achieve the above objectives, according to one aspect of the present invention, a near-infrared camera multi-frame time-division welding observation device is provided. The device includes a near-infrared camera, a near-infrared laser, a near-infrared pulsed illumination source, and a synchronization controller, wherein the synchronization controller is electrically connected to the near-infrared camera, the near-infrared laser, and the near-infrared pulsed illumination source, respectively.
[0010] The synchronization controller is configured to control the near-infrared camera to acquire near-infrared images according to a time-division exposure sequence and generate a frame type identifier corresponding to the current exposure frame. The time-division exposure sequence includes an active light source shutdown frame, a line laser frame, and an illumination enhancement frame, and the active light source shutdown frame, the line laser frame, and the illumination enhancement frame are acquired by the same near-infrared camera under the same imaging optical path, the same camera intrinsic parameters, and the same camera mounting extrinsic parameters.
[0011] In the active light source off frame, both the near-infrared laser and the near-infrared pulsed illumination source are turned off to acquire near-infrared images for extracting at least one of the following: molten pool state information, welding area brightness characteristics, arc stability characteristics, or droplet transition state characteristics.
[0012] In the line laser frame, the near-infrared laser is turned on and the near-infrared pulsed illumination source is turned off to acquire structured light images for obtaining three-dimensional geometric information of the weld, bevel, or weld bead.
[0013] In the illumination enhancement frame, the near-infrared pulsed illumination source is turned on and the near-infrared laser is turned off, so as to acquire an illumination enhancement image that does not contain the structured light stripes of the current frame and is used to extract two-dimensional information of the wire tip, weld boundary, weld bead boundary or welding neighborhood.
[0014] The synchronization controller causes the high-level drive of the near-infrared laser in the line laser frame to cover the exposure window of the near-infrared camera, and causes the high-level drive of the near-infrared pulsed illumination source in the illumination enhancement frame to cover the exposure window of the near-infrared camera, and sets a light source shutdown protection interval between adjacent frames where both the near-infrared laser and the near-infrared pulsed illumination source are turned off.
[0015] Optionally, the synchronization controller has independent synchronization output channels corresponding to the near-infrared camera, the near-infrared laser, and the near-infrared pulsed illumination source, and outputs camera exposure trigger signal, line laser drive signal, illumination source drive signal, frame type identifier, and frame timestamp or frame sequence number for spatiotemporal alignment through the independent synchronization output channels.
[0016] Optionally, the line laser projected by the near-infrared laser is located in at least one of the weld area, bevel area, interlayer weld area, or post-weld weld area in front of the welding torch TCP; a preset angle is formed between the laser plane of the near-infrared laser and the imaging optical axis of the near-infrared camera.
[0017] Optionally, the near-infrared pulsed illumination source includes at least one near-infrared illumination unit, which is used to form a planar or array-shaped near-infrared illumination area in the illumination enhancement frame.
[0018] Optionally, the device further includes a near-infrared filter, which is disposed in the imaging optical path of the near-infrared camera, and the transmission band of the near-infrared filter matches the center wavelength of the near-infrared laser and / or the near-infrared pulsed illumination source.
[0019] Optionally, the device further includes a processor, which calls corresponding exposure compensation, image preprocessing, or feature extraction parameters according to the frame type identifier, and extracts at least one of the following from the active light source off frame, line laser frame, and illumination enhancement frame: molten pool state information, welding area state information, welding neighborhood two-dimensional information, and three-dimensional geometric information. The processor also performs spatiotemporal alignment on the extraction results of the three types of frames according to the frame timestamp or frame sequence number, the welding robot pose information, the intrinsic parameters of the near-infrared camera, the line laser plane calibration parameters, and the extrinsic parameters of the near-infrared camera relative to the welding torch TCP, and transforms the extraction results to the welding torch TCP coordinate system, the robot base coordinate system, or the workpiece coordinate system. For the two-dimensional information in the illumination enhancement frame, the processor performs coordinate transformation based on the known workpiece plane constraints, weld local plane constraints, or the three-dimensional geometric information obtained from the line laser frame.
[0020] According to another aspect of the present invention, a welding and observation system is provided. The system includes any of the aforementioned near-infrared camera multi-frame time-division welding observation devices, a welding torch, and a welding robot that holds and controls the movement of the welding torch.
[0021] According to another aspect of the present invention, a multi-frame time-division welding observation method using a near-infrared camera is provided. The method includes: controlling a near-infrared camera to acquire near-infrared images according to a time-division exposure sequence including active light source shutdown frames, line laser frames, and illumination enhancement frames; generating a frame type identifier and a frame timestamp or frame sequence number for each frame; shutting down the near-infrared laser and near-infrared pulsed illumination source in the active light source shutdown frames; turning on the near-infrared laser and shutting down the near-infrared pulsed illumination source in the line laser frames; turning on the near-infrared pulsed illumination source and shutting down the near-infrared laser in the illumination enhancement frames; covering the exposure window of the near-infrared camera with the driving high level of the corresponding light source, and setting a light source shutdown protection interval between adjacent frames; and spatiotemporally aligning the processing results of the three types of frames according to the frame type, frame timestamp or frame sequence number, and the pose information of the welding robot.
[0022] By applying the technical solution of this invention, the molten pool state information, two-dimensional information of the welding neighborhood, and three-dimensional geometric information of the weld or weld bead can be acquired in time-division using the same near-infrared camera; inter-frame crosstalk can be reduced by frame type identification, guard interval, and corresponding light source exposure coverage control; and at least some observation results can be unified to the same coordinate system by the same imaging optical path, common calibration parameters, and spatiotemporal alignment mechanism, thereby providing an observation basis for online welding measurement, trajectory tracking, and deviation correction. Attached Figure Description
[0023] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0024] Figure 1 A schematic diagram of a near-infrared camera multi-frame time-division welding observation device according to an embodiment of the present invention is shown.
[0025] Figure 2 A schematic diagram of a welding and observation system according to an embodiment of the present invention is shown.
[0026] Figure 3 A schematic diagram of the spatial arrangement of a near-infrared camera, a near-infrared laser, and a near-infrared pulsed illumination module relative to a welding torch is shown according to an embodiment of the present invention.
[0027] Figure 4 A schematic diagram of the camera ROI acquisition area, the line laser stripe projection area, and the illumination enhancement area according to an embodiment of the present invention is shown.
[0028] Figure 5 A time-division timing diagram of an active light source shutdown frame, a line laser frame, and an illumination enhancement frame according to an embodiment of the present invention is shown.
[0029] Figure 6 A schematic diagram illustrating the image processing and welding observation state quantity generation process for different frame types according to an embodiment of the present invention is shown.
[0030] The above-mentioned figures include the following reference numerals: 10, near-infrared camera; 11, lens; 12, near-infrared filter; 20, near-infrared laser; 30, near-infrared illumination unit; 31, near-infrared pulse illumination module; 40, synchronization controller; 50, processor; 60, welding torch; 61, welding wire; 70, welding robot; 80, weldment; 81, weld seam; 82, camera ROI acquisition area; 83, line laser stripe projection area; 84, illumination enhancement area; 100, near-infrared camera multi-frame time-division welding observation device. Detailed Implementation
[0031] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings. It should be noted that the following embodiments are used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Unless otherwise specified, the embodiments and technical features in the embodiments of the present invention can be combined with each other.
[0032] First aspect embodiment: Near-infrared camera multi-frame time-division welding observation device
[0033] like Figures 1 to 3 As shown, this embodiment of the invention provides a near-infrared camera multi-frame time-division welding observation device 100. The device includes a near-infrared camera 10, a near-infrared laser 20, a near-infrared pulsed illumination source 31, and a synchronization controller 40. Optionally, the device also includes a near-infrared filter 12 and a processor 50.
[0034] The near-infrared camera 10 is used to acquire near-infrared images of the vicinity of the welding torch 60. The near-infrared camera 10 includes a lens 11 and a near-infrared filter 12 disposed in the imaging optical path of the near-infrared camera 10 to allow light of a preset near-infrared band to enter the near-infrared camera 10 and to suppress some of the arc light, spatter reflection light or ambient stray light of non-target bands.
[0035] Near-infrared laser 20 is used to project line laser light onto the weld seam 81, bevel, or weld bead area on the weldment 80 to form a line laser stripe projection area 83. Near-infrared camera 10 acquires images containing structured light stripes in the line laser frame, and processor 50 can recover the three-dimensional geometric information of the weld seam, bevel, or weld bead based on the deformation information of the structured light stripes.
[0036] The near-infrared pulsed illumination source 31 includes at least one near-infrared illumination unit 30. The near-infrared illumination unit 30 can be a near-infrared LED, a VCSEL, a near-infrared laser diode, or a combination thereof. The near-infrared pulsed illumination source 31 is used to form an illumination enhancement region 84 in the illumination enhancement frame to enhance the image contrast of the wire 61 tip, weld 81 boundary, weld bead boundary, weld bead surface, or weld neighborhood texture.
[0037] The synchronization controller 40 is connected to the near-infrared camera 10, the near-infrared laser 20, and the near-infrared pulsed illumination source 31, respectively. The synchronization controller 40 may include at least one of the following: hardware timer, microcontroller, FPGA, PLC, motion control card, or real-time controller. It is used to control the near-infrared camera 10, the near-infrared laser 20, and the near-infrared pulsed illumination source 31 to operate according to a preset hardware timing sequence, and to output frame type identifier, frame timestamp, or frame sequence number.
[0038] The processor 50 receives images acquired by the near-infrared camera 10 and calls the corresponding processing flow according to the frame type of the image. The processor 50 can be set up independently or integrated into the control system of the synchronization controller 40, the camera control unit, or the welding robot 70.
[0039] Second aspect of the implementation: Multi-frame time-division exposure control
[0040] like Figure 4 and Figure 5 As shown, the synchronization controller 40 controls the near-infrared camera 10 to acquire near-infrared images according to a time-division exposure sequence. The time-division exposure sequence includes active light source off frames, line laser frames, and illumination enhancement frames. These three types of frames are acquired by the same near-infrared camera 10 under the same imaging optical path, the same camera intrinsic parameters, and the same camera mounting extrinsic parameters.
[0041] During the active light source shutdown frame, the synchronization controller 40 controls both the near-infrared laser 20 and the near-infrared pulsed illumination source 31 to be in a shutdown state, and the near-infrared camera 10 acquires near-infrared images within the camera ROI acquisition area 82. These near-infrared images can be used to extract molten pool state information, welding area brightness characteristics, arc light state information, or droplet transition state information.
[0042] In the online laser frame, the synchronization controller 40 controls the near-infrared laser 20 to turn on and the near-infrared pulse illumination source 31 to turn off. The near-infrared camera 10 acquires structured light images within the line laser stripe projection area 83 to obtain three-dimensional geometric information of the weld 81, bevel, interlayer weld bead, or post-weld weld bead.
[0043] In the illumination enhancement frame, the synchronization controller 40 controls the near-infrared pulse illumination source 31 to turn on and the near-infrared laser 20 to turn off. The near-infrared camera 10 acquires illumination enhancement images within the illumination enhancement region 84. Since the near-infrared laser 20 is turned off in this frame, the illumination enhancement image does not contain the structured light stripes of the current frame and can be used to extract two-dimensional information of the welding wire 61 end, weld seam 81 boundary, weld bead boundary, or welding neighborhood.
[0044] like Figure 5 As shown, in the online laser frame, the high-level drive of the near-infrared laser 20 covers the exposure window of the near-infrared camera 10; in the illumination enhancement frame, the high-level drive of the near-infrared pulsed illumination source 31 covers the exposure window of the near-infrared camera 10. The high level can be turned on by a predetermined advance amount before the start of the exposure window and turned off by a predetermined delay amount after the end of the exposure window, thereby avoiding image brightness fluctuations caused by the rising or falling edge of the light source falling into the main exposure range.
[0045] like Figure 5 As shown, the synchronization controller 40 sets a light source shutdown protection interval between adjacent frames. During the light source shutdown protection interval, both the near-infrared laser 20 and the near-infrared pulsed illumination source 31 remain off to reduce light source crosstalk between adjacent frames and to provide time margin for light source shutdown, frame type switching, camera exposure trigger jitter, and transient reflection attenuation of metal surfaces.
[0046] In one optional implementation, the synchronization controller 40 also generates a frame type identifier, frame timestamp, or frame sequence number corresponding to the current exposure frame. The frame type identifier can be sent to the processor 50 or written into the timestamp, frame sequence number, or metadata of the image acquired by the near-infrared camera 10. The processor 50 distinguishes between active light source shutdown frames, line laser frames, and illumination enhancement frames based on the frame type identifier, and performs spatiotemporal alignment of the extraction results for different frame types by combining the frame timestamp or frame sequence number with the pose information of the welding robot 70.
[0047] Third aspect of the embodiment: Image processing of different frame types
[0048] like Figure 6 As shown, the processor 50 processes the images acquired by the near-infrared camera 10 according to the frame type to obtain welding area status information, welding neighborhood two-dimensional information and three-dimensional geometric information respectively, and associates the results of different frame types with the same welding observation cycle according to the frame timestamp or frame sequence number.
[0049] For frames where the active light source is off, the processor 50 can preprocess the near-infrared image in the camera ROI acquisition area 82 and extract at least one of the following: molten pool center, molten pool width, molten pool length, molten pool area, welding area brightness features, arc stability features, or droplet transition state features. Since both the near-infrared laser 20 and the near-infrared pulsed illumination source 31 are off in this frame, the risk of strong reflections from the applied near-infrared active light on the molten pool, welding wire 61, or weldment 80 surface, which could affect image feature extraction, can be reduced.
[0050] For line laser frames, processor 50 can perform background suppression, stripe candidate region extraction, stripe centerline extraction, and subpixel localization on the structured light image, and combine the intrinsic parameters of near-infrared camera 10 and line laser plane calibration parameters to recover the three-dimensional geometric information of weld 81, bevel, or weld bead. The three-dimensional geometric information may include at least one of weld center position, bevel width, bevel depth, misalignment, weld bead height, or weld bead three-dimensional contour.
[0051] For the illumination-enhanced frame, the processor 50 can perform grayscale normalization, edge enhancement, target detection, or semantic segmentation on the illumination-enhanced image, and extract at least one of the following: the position of the welding wire 61 end, the boundary of the weld 81, the weld bead boundary, the weld bead surface texture, or two-dimensional information of the welding neighborhood. Since the near-infrared laser 20 is turned off in this frame, the influence of the structured light stripes of the current frame on the identification of the welding wire end, weld boundary, or weld bead boundary can be reduced. For two-dimensional information that requires coordinate transformation, the processor 50 can use known workpiece plane constraints, weld local plane constraints, or three-dimensional geometric information obtained from the line laser frame to determine the spatial position corresponding to the two-dimensional image features.
[0052] The processor 50 can also uniformly transform the molten pool state information extracted from the active light source off frame, the three-dimensional geometric information extracted from the line laser frame, and the two-dimensional boundary information extracted from the illumination enhancement frame to the welding torch TCP coordinate system, the robot base coordinate system, or the workpiece coordinate system to generate welding observation state quantities. This transformation can be based on the intrinsic parameters of the near-infrared camera 10, the line laser plane calibration parameters, the extrinsic parameters of the near-infrared camera 10 relative to the welding torch TCP, the workpiece coordinate system calibration parameters, and the pose information of the welding robot 70 at each exposure time. The welding observation state quantities can include at least one of the following: molten pool center, molten pool width, welding wire end position, weld boundary, weld bead boundary, weld bead three-dimensional contour, groove width, groove depth, misalignment, and visual confidence level.
[0053] Fourth aspect of the embodiment: Welding and observation system
[0054] like Figure 2 As shown, this embodiment of the invention also provides a welding and observation system. The system includes the aforementioned near-infrared camera multi-frame time-division welding observation device 100, a welding torch 60, and a welding robot 70 that clamps and controls the movement of the welding torch 60.
[0055] A near-infrared camera multi-frame time-division welding observation device 100 is positioned in the vicinity of the welding torch 60 and maintains a fixed relative posture or preset calibration relationship with the welding torch 60, moving with the welding torch 60. The welding robot 70 can adjust the position, attitude, welding speed, or oscillation parameters of the welding torch 60 according to the welding observation state quantities generated by the processor 50, so as to realize online welding measurement, trajectory tracking, and deviation correction.
[0056] Fifth aspect embodiment: Near-infrared camera multi-frame time-division welding observation method
[0057] This invention also provides a method for multi-frame time-division welding observation using a near-infrared camera, comprising the following steps:
[0058] The near-infrared camera 10 is controlled to acquire near-infrared images according to a time-division exposure sequence including active light source shutdown frames, line laser frames, and illumination enhancement frames, and to generate a frame type identifier and a frame timestamp or frame sequence number for each frame.
[0059] In the active light source shutdown frame, the near-infrared laser 20 and the near-infrared pulse illumination source 31 are turned off, and near-infrared images are acquired to extract molten pool state information or welding area state information.
[0060] In the online laser frame, the near-infrared laser 20 is turned on and the near-infrared pulse illumination source 31 is turned off, and a structured light image is acquired to obtain the three-dimensional geometric information of the weld 81, bevel, or weld bead.
[0061] In the illumination enhancement frame, the near-infrared pulse illumination source 31 is turned on and the near-infrared laser 20 is turned off, and an illumination enhancement image is acquired to extract two-dimensional information of the end of the welding wire 61, the boundary of the weld seam 81, the boundary of the weld bead, or the welding neighborhood.
[0062] The near-infrared laser 20 in the line laser frame is driven to a high level to cover the exposure window of the near-infrared camera 10, and the near-infrared pulse illumination source 31 in the illumination enhancement frame is driven to a high level to cover the exposure window of the near-infrared camera 10. A light source shutdown protection interval is set between adjacent frames where both active light sources are turned off.
[0063] The near-infrared image, structured light image, and illumination-enhanced image are processed according to the frame type. The processing results of the three types of frames are spatiotemporally aligned according to the frame timestamp or frame sequence number, the pose information of the welding robot 70, and the calibration parameters. The processing results are then converted to the welding torch TCP coordinate system, the robot base coordinate system, or the workpiece coordinate system.
[0064] In one optional implementation, the active light source shutdown frame, line laser frame, and illumination enhancement frame can be acquired cyclically in a fixed order. For example, the time-division exposure sequence can be an active light source shutdown frame, line laser frame, and illumination enhancement frame cycling sequentially, or it can be set to an active light source shutdown frame, line laser frame, active light source shutdown frame, and illumination enhancement frame cycling sequentially according to different welding task requirements.
[0065] In another alternative implementation, the acquisition ratio of active light source off frames, line laser frames, and illumination enhancement frames can be adjusted according to the observation task. For example, when it is necessary to focus on acquiring the state of the molten pool, the acquisition ratio of active light source off frames can be increased; when it is necessary to focus on acquiring the three-dimensional geometric information of the weld, groove, or weld bead, the acquisition ratio of line laser frames can be increased; and when it is necessary to focus on identifying the wire tip, weld boundary, or weld bead boundary, the acquisition ratio of illumination enhancement frames can be increased.
[0066] In one optional implementation, the processor 50 or the synchronization controller 40 can adaptively adjust the time-division exposure sequence based on the image quality of the current frame. The image quality may include at least one of the following: weld area image saturation ratio, structured light stripe signal-to-noise ratio, target detection confidence level of the illumination-enhanced image, wire end recognition confidence level, or weld boundary recognition confidence level. During adaptive adjustment, at least one active light source shutdown frame, at least one line laser frame, and at least one illumination-enhanced frame can be retained within a preset time window to maintain the continuity of multiple types of observation information.
[0067] In one alternative implementation, the active light source shutdown frame, line laser frame, and illumination enhancement frame can each be configured with different exposure times, camera gains, light source pulse widths, or light source drive currents. This allows for the acquisition of appropriate image brightness and contrast for molten pool state observation, structured light 3D measurement, and illumination enhancement recognition, respectively.
[0068] For example, the exposure time of the active light source shutdown frame can be from 50 μs to 500 μs, the exposure time of the line laser frame can be from 20 μs to 200 μs, and the exposure time of the illumination enhancement frame can be from 100 μs to 1000 μs; the light source shutdown protection interval between adjacent frames can be from 1 μs to 500 μs. The light source shutdown protection interval can be determined based on at least one of the following: near-infrared laser shutdown delay, near-infrared pulsed illumination source shutdown delay, camera exposure trigger jitter, and transient reflection attenuation time of the welded metal surface. The above parameters are only for illustrating possible implementations; those skilled in the art can adjust them based on welding current, material reflection characteristics, working distance, filter transmission band, and near-infrared camera response characteristics.
[0069] In one specific embodiment, the near-infrared camera 10 has a frame rate of 60 fps, and the time-division exposure sequence consists of an active light source shutdown frame, a line laser frame, and an illumination enhancement frame, which cycle sequentially. The exposure time for the active light source shutdown frame is 200 μs, the exposure time for the line laser frame is 100 μs, and the exposure time for the illumination enhancement frame is 500 μs. The light source shutdown protection interval between adjacent frames is 100 μs. The above specific parameters are used to illustrate one possible implementation of the present invention and do not constitute a limitation on the scope of protection of the present invention.
[0070] In one alternative implementation, the near-infrared laser 20 and the near-infrared pulsed illumination source 31 can use the same center wavelength or different center wavelengths. When they use the same center wavelength, the near-infrared filter 12 can be a single-passband narrowband filter that matches the center wavelength; when they use different center wavelengths, the near-infrared filter 12 can be a multi-passband filter or a broadband near-infrared filter.
[0071] In one optional implementation, the processor 50 uniformly transforms the molten pool state information extracted from the active light source shutdown frame, the three-dimensional geometric information extracted from the line laser frame, and the two-dimensional boundary information extracted from the illumination enhancement frame to the welding torch TCP coordinate system, the robot base coordinate system, or the workpiece coordinate system. For the three-dimensional geometric information in the line laser frame, coordinate transformation can be performed based on camera intrinsic parameters, line laser plane calibration parameters, and camera extrinsic parameters. For the two-dimensional boundary information in the illumination enhancement frame, the correspondence between image points and spatial points can be established based on the known workpiece plane, the local plane of the weld, or the three-dimensional geometric information recovered from the line laser frame. For the molten pool state information in the active light source shutdown frame, observations relative to the welding torch can be generated based on the position of the molten pool center or contour in the image and the welding torch TCP extrinsic parameters. Thus, the molten pool state, the two-dimensional boundary of the weld or weld bead, and the three-dimensional geometric information of the weld or weld bead can be expressed in the same coordinate system.
[0072] In one optional evaluation method, the end-of-line integration complexity of the present invention and the multi-sensor observation scheme can be evaluated by the end-of-line assembly mass, installation volume, and number of cables; coordinate consistency can be evaluated by the repeated measurement residuals of standard calibration boards or standard parts under different welding torch postures; the relative timing deviation between the camera exposure trigger signal, the line laser drive signal, and the illumination source drive signal can be measured by an oscilloscope; the inter-frame crosstalk control effect can be evaluated by the structured light stripe signal-to-noise ratio in the line laser frame, the weld boundary recognition stability in the illumination enhancement frame, and the proportion of falsely triggered bright spots or residual stripes in adjacent frames; and the spatiotemporal alignment effect can be evaluated by the reprojection error of the same weld feature before and after robot pose interpolation.
[0073] In the above manner, the present invention can acquire active light source shutdown frames, line laser frames, and illumination enhancement frames in 10-minute intervals using the same near-infrared camera, thereby reducing the structural complexity of the welding torch end observation device; by using the temporal mutual exclusion of line laser frames and illumination enhancement frames and the light source shutdown protection interval of adjacent frames, the mutual interference between structured light stripes and illumination enhancement images is reduced; and the state of the molten pool or welding area under the condition of no external near-infrared active illumination is obtained through the active light source shutdown frames.
[0074] Furthermore, the present invention can acquire three-dimensional geometric information of weld, groove or weld bead through line laser frames, acquire two-dimensional information such as wire end, weld boundary or weld bead boundary through illumination enhancement frames, and complete spatiotemporal alignment and unified coordinate transformation through frame type identifier, frame timestamp or frame sequence number and robot pose information, providing consistent welding observation state quantities for online welding measurement, trajectory tracking and deviation correction.
[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A near-infrared camera multi-frame time-division welding observation device, characterized in that, include: The system includes a near-infrared camera, a near-infrared laser, a near-infrared pulsed illumination source, and a synchronization controller. The synchronization controller is electrically connected to the near-infrared camera, the near-infrared laser, and the near-infrared pulsed illumination source, and is configured to generate a frame type identifier corresponding to the exposure frame. The synchronization controller controls the near-infrared camera to acquire near-infrared images according to a time-division exposure sequence. The time-division exposure sequence includes active light source shutdown frames, line laser frames, and illumination enhancement frames. The frame type identifier corresponds to the active light source shutdown frame, the line laser frame, or the illumination enhancement frame. All three types of frames are acquired by the same near-infrared camera under the same imaging optical path, the same camera intrinsic parameters, and the same camera installation extrinsic parameters. In the active light source off frame, both the near-infrared laser and the near-infrared pulsed illumination source are turned off to acquire near-infrared images for extracting at least one of the following: molten pool state information, welding area brightness characteristics, arc stability characteristics, or droplet transition state characteristics. In the line laser frame, the near-infrared laser is turned on and the near-infrared pulsed illumination source is turned off to acquire structured light images for obtaining three-dimensional geometric information of welds, weld beads, or bevels. In the illumination enhancement frame, the near-infrared pulse illumination source is turned on and the near-infrared laser is turned off, so as to acquire an illumination enhancement image that does not contain the structured light stripes of the current frame and is used to extract two-dimensional information of the welding wire end, weld boundary, weld bead boundary or welding neighborhood; The synchronization controller causes the high-level drive of the near-infrared laser in the line laser frame to cover the exposure window of the near-infrared camera, and causes the high-level drive of the near-infrared pulsed illumination source in the illumination enhancement frame to cover the exposure window of the near-infrared camera, and sets a light source shutdown protection interval between adjacent frames where both the near-infrared laser and the near-infrared pulsed illumination source are turned off.
2. The near-infrared camera multi-frame time-division welding observation device according to claim 1, characterized in that, The synchronization controller has independent synchronization output channels corresponding to the near-infrared camera, the near-infrared laser, and the near-infrared pulsed illumination source, and outputs camera exposure trigger signal, line laser drive signal, illumination source drive signal, frame type identifier, and frame timestamp or frame sequence number for spatiotemporal alignment through the independent synchronization output channels.
3. The near-infrared camera multi-frame time-division welding observation device according to claim 1, characterized in that, The light source shutdown protection interval is from 1 μs to 500 μs, and is determined based on at least one of the following: near-infrared laser shutdown delay, near-infrared pulsed illumination source shutdown delay, camera exposure trigger jitter, or transient reflection attenuation time of welded metal surface; within the light source shutdown protection interval, both the near-infrared laser and the near-infrared pulsed illumination source remain off.
4. The near-infrared camera multi-frame time-division welding observation device according to claim 1, characterized in that, The line laser projected by the near-infrared laser is located in at least one of the unwelded bevel area, weld area, interlayer weld area, or post-weld weld area in front of the welding torch TCP, and the laser plane of the near-infrared laser forms an angle of 20° to 60° with the imaging optical axis of the near-infrared camera.
5. The near-infrared camera multi-frame time-division welding observation device according to claim 1, characterized in that, The near-infrared pulsed illumination source is used to form a planar or array-type near-infrared illumination area in the illumination enhancement frame to extract at least one of the wire tip, bevel edge, molten pool neighborhood, weld boundary, weld bead boundary, or weld bead surface under conditions without interference from current frame line laser stripes.
6. The near-infrared camera multi-frame time-division welding observation device according to claim 1, characterized in that, The device further includes a near-infrared filter, which is disposed in the imaging optical path of the near-infrared camera, and the transmission band of the near-infrared filter matches the center wavelength of the near-infrared laser and / or the near-infrared pulsed illumination source.
7. The near-infrared camera multi-frame time-division welding observation device according to claim 1, characterized in that, The time-sharing exposure sequence is either a periodic alternating sequence or an adaptive scaling sequence. In the adaptive scaling sequence, the synchronization controller adjusts the acquisition ratio of the three types of frames according to the saturation ratio of the welding area image, the signal-to-noise ratio of the structured light stripes, the target detection confidence of the illumination-enhanced image, the wire end recognition confidence, or the weld boundary recognition confidence, and ensures that at least one frame of each type of frame is acquired within a preset time window.
8. The near-infrared camera multi-frame time-division welding observation device according to claim 2, characterized in that, The device further includes a processor, which calls corresponding exposure compensation, image preprocessing, or feature extraction parameters according to the frame type identifier, and extracts at least one of the following from the active light source off frame, the line laser frame, and the illumination enhancement frame: molten pool state information, welding area state information, weld bead three-dimensional geometric information, welding wire end information, weld boundary information, weld bead boundary information, or welding neighborhood two-dimensional information. The processor also spatiotemporally aligns and transforms the extracted results to the welding torch TCP coordinate system, the robot base coordinate system, or the workpiece coordinate system based on the frame timestamp or frame number, the welding robot pose information, the intrinsic parameters of the near-infrared camera, the line laser plane calibration parameters, and the extrinsic parameters of the near-infrared camera relative to the welding torch TCP. For the two-dimensional information in the illumination enhancement frame, the processor performs coordinate transformation based on known workpiece plane constraints, weld local plane constraints, or the three-dimensional geometric information obtained from the line laser frame.
9. A welding and observation system, characterized in that, include: The near-infrared camera multi-frame time-division welding observation device, welding torch, and welding robot that clamps and controls the movement of the welding torch according to any one of claims 1 to 8, wherein the near-infrared camera multi-frame time-division welding observation device is disposed in the vicinity of the welding torch and maintains a fixed relative posture or preset calibration relationship with the welding torch, and moves with the welding torch.
10. A multi-frame time-division welding observation method using a near-infrared camera applied to the welding and observation system of claim 9, characterized in that, include: Control the near-infrared camera to acquire near-infrared images according to a time-division exposure sequence including active light source shutdown frames, line laser frames, and illumination enhancement frames, and generate a frame type identifier and a frame timestamp or frame sequence number for each frame; In the active light source shutdown frame, the near-infrared laser and the near-infrared pulse illumination source are turned off, and near-infrared images are acquired to extract at least one of the following: molten pool state information, welding area brightness characteristics, arc stability characteristics, or droplet transition state characteristics. In the line laser frame, the near-infrared laser is turned on and the near-infrared pulse illumination source is turned off, and a structured light image is acquired to obtain three-dimensional geometric information of the weld, weld bead, or bevel. In the illumination enhancement frame, the near-infrared pulse illumination source is turned on and the near-infrared laser is turned off, and an illumination enhancement image is acquired to extract two-dimensional information of the welding wire tip, weld boundary, weld bead boundary or welding neighborhood. The near-infrared laser in the line laser frame is driven at a high level to cover the exposure window of the near-infrared camera, and the near-infrared pulse illumination source in the illumination enhancement frame is driven at a high level to cover the exposure window of the near-infrared camera. A light source shutdown protection interval is set between adjacent frames where both active light sources are turned off. The near-infrared image, the structured light image, and the illumination-enhanced image are processed according to their frame types respectively; Based on the frame timestamp or frame sequence number, the welding robot's pose information, and calibration parameters, the processing results of the three types of frames are spatiotemporally aligned, and the processing results are converted to the welding torch TCP coordinate system, the robot base coordinate system, or the workpiece coordinate system.
