A method and system for vehicle welding status recognition based on multi-source vision fusion

By using protected regions and reference constraints in multi-source image processing, the problems of absorption of real geometric changes and repeated error accounting in the welding of electric tricycle frames are solved, thus achieving accurate identification of welding status.

CN123119002APending Publication Date: 2026-11-17XUZHOU MEIBANG ELECTRIC VEHICLE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611618101.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

During the welding process of electric tricycle frames, real geometric changes in multi-source image processing are easily absorbed by registration transformation, common errors are repeatedly included, and supplementary observations lack specificity, making it difficult to distinguish between normal and abnormal.

Method used

By associating multi-source images of the same welding position and stage, the protection area is delineated. The imaging parameters and the displacement range of the base material are determined using fixture and base material reference. The images are then combined with compatible parameters and projected onto the original images. Combinations that meet the valid measured characteristics and error constraints are retained. Normal and abnormal candidates are distinguished, and the final state is confirmed by reviewing the images.

Benefits of technology

It effectively distinguishes between normal and abnormal welding conditions, avoids double counting of errors, ensures targeted observation, prevents misjudgment due to lack of evidence, and achieves accurate welding condition identification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123119002A_ABST
    Figure CN123119002A_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for vehicle welding status recognition based on multi-source visual fusion, belonging to the field of image processing technology, and applied to the welding of electric tricycle frames. The method associates multi-source images at the same location and stage, delineates a protected area, and determines the interrelated imaging parameters and base material displacement range only by external references. Shared parameters are used to project candidate geometry onto the original image, retaining both normal and abnormal candidates. When separable heterogeneous candidates are predicted, a verification position is selected, the prediction is recorded, the actual image is verified, and incompatible candidates are excluded. The status is output based on reference, coverage, and candidate retention. This invention restricts changes to the reference estimate in the area to be inspected, maintains the common error relationship between boundaries, and ensures that the verification targets unresolved geometric ambiguities.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image processing and multi-view image analysis technology, and in particular to a method and system for state recognition of welding of electric tricycle frames. Background Technology

[0002] The frame of an electric tricycle typically includes tubular components, crossbeams, and connecting brackets. Welded areas are characterized by curved surfaces meeting flat surfaces, obstruction by adjacent components, and changes in the position of thin-walled base material due to heating. During welding, intense arc light, spatter, fumes, and specular reflection affect the weld boundary in the image. Post-weld observation further alters the weld's position in the image due to changes in camera pose and base material displacement. Therefore, the joint processing of visible light and infrared images requires addressing the correlation of the same location, determining the accuracy of boundaries, and distinguishing between imaging changes and actual geometric changes.

[0003] In one multi-source image processing method, different images are first registered, then weld features are extracted separately and the recognition results are fused. If ambiguity exists in the recognition, additional images are acquired. If the weld to be inspected or the heat-deformed area is allowed to participate in free fitting during registration, the actual offset or narrowing may be absorbed into the registration transformation. If independent errors are added to both boundaries, errors such as common camera translation will be repeatedly included in the width range. Both of these processes may weaken the distinction between real anomalies and imaging interference.

[0004] On the other hand, even if the boundaries of the supplementary acquired images are clearer, they may not necessarily be able to distinguish between the currently valid normal and abnormal morphologies. If it is not determined beforehand where the two types of morphologies produce different visible boundaries, or if parameters are arbitrarily adjusted to interpret the new images after they arrive, it is difficult to confirm whether the verification images provide independent criteria for discrimination. Therefore, an image processing method is needed that can constrain the reference estimation source, maintain the common error relationship between multiple boundaries, and select the verification location according to the actual discrimination needs. Summary of the Invention

[0005] This application provides a vehicle welding status recognition method and system based on multi-source visual fusion, which solves the technical problems in multi-source image processing, such as the absorption of real geometric changes by registration transformation, the repeated inclusion of common errors, and the lack of specificity in supplementary observation, which make it difficult to distinguish between normal and abnormal conditions.

[0006] The first aspect of this application provides a method for vehicle welding status recognition based on multi-source visual fusion, the method comprising: Step 1: Associate multi-source images of the same welding location and stage to delineate the protected area containing the weld to be inspected; Step 2: Determine the interrelated imaging parameters and the displacement range of the base material solely based on the clamp reference outside the protected area and the base material reference; Step 3: Project the candidate weld geometry and the compatibility parameter combination within the range onto the original image, so that the relevant boundaries in the same candidate combination share the corresponding imaging parameters and base material displacement parameters, retain the combination that meets the effective measured characteristics and error constraints, and distinguish normal candidates from abnormal candidates according to the geometric allowable range of the detection item. Step 4: If neither type of candidate is excluded, predict its boundary range in the available verification image, select the position where the predicted range of at least one pair of heterogeneous candidates can be separated after accounting for the error, and record the prediction and parameter version. Step 5: When the selected location is available, obtain the actual image, verify the validity of the reference outside the area and the stage, and after the verification is passed, exclude candidates based on the actual boundary and the incompatibility of the recorded prediction, without expanding the parameter range based on the weld to be inspected. Step 6: When the image coverage of the reference and corresponding detection items is valid, and one type of normal candidate and abnormal candidate has a specific combination that meets all constraints, while the other type of candidate is completely excluded, output the normal state or abnormal state corresponding to the type with the specific combination. If the normal or abnormal state output conditions are not met, output "cannot be determined" and retain the unexcluded candidate. If both types of candidates still coexist and the review deadline has not been reached, continue the review. If the candidate combination test has not been completed, continue the candidate test.

[0007] Furthermore, the multi-source images include visible light images and infrared images from different viewing directions. The protected area is formed by merging a preset weld corridor, suspected abnormal areas, infrared radiation change areas, and image saturation areas, and expanding them according to the mapping error; newly added areas within the same event to be inspected are incorporated into the protected area, and pixels within the protected area are used for detection rather than for fitting imaging parameters and base material displacement.

[0008] Furthermore, the camera imaging changes are constrained by a fixed reference using a fixture, and the positional changes of the base material relative to the pre-welding datum are constrained by base material references on both sides of the protected area. Reference marks not involved in the fitting are retained to check the projection residuals; if the reference distribution degrades or the residuals exceed limits, the reference estimate is recorded as invalid.

[0009] Furthermore, weld boundary locations, orientations, and occlusion / saturation markers are extracted from the original visible light images in the multi-source images. Candidate verification retains the shared relationships between different boundaries using the same camera, scale, and base material reference. Multiple branches are retained for boundaries that cannot be confirmed; candidates for occlusion or saturation locations are not rejected due to missing boundaries. Exclusion is performed using interval cells covering all allowed parameter values ​​and their projected bounding ranges; cells that have not yet undergone judgment are still retained.

[0010] Furthermore, when both reference estimation and image coverage are valid, the individual state is determined based on the exclusion of normal and abnormal candidates and specific feasible combinations; when both types of candidates are excluded, the recorded image constraints are inconsistent, and it is not directly judged as a welding anomaly. If the discrimination cannot be completed, the state that cannot be determined is output.

[0011] Furthermore, the boundary location ranges of the two types of candidates are predicted in the candidate review images. Imaging errors, camera pose prediction errors, and weld change errors are included in the prediction. Normal candidate branches are paired with abnormal candidate branches, and the location where the interval between the predicted ranges of at least one pair of outlier branches exceeds the minimum resolvable interval is selected. The prediction and parameter versions are recorded, and then the actual image content to be inspected is read to complete the elimination using the actual boundaries. When all candidates are incompatible, the parameters are re-estimated only based on the new reference outside the protected area, and the version record is retained.

[0012] Furthermore, status records are established according to workpiece, weld, pass, position, and stage, respectively, and the identification results of the welding process and the post-weld formed surface are saved. The current joint displacement, weld offset relative to the current joint, and weld offset relative to the pre-weld reference are recorded respectively. When the review cutoff condition is reached, the results and unresolved positions are saved.

[0013] A second aspect of this application provides a multi-source vision fusion vehicle welding status recognition system, the system comprising: The image association module is used to receive multi-source images and observation conditions, associate images of the same welding position and stage, delineate the protection area containing the weld to be inspected, transfer the associated images and protection area to the reference estimation module and the candidate inspection module, and transfer the position, stage and available observation conditions to the verification selection module. The reference estimation module is used to determine the interrelated imaging parameters and the displacement range of the base material solely from the clamp reference outside the protected area and the base material reference. The range, correlation, reference validity, and parameter version are then transmitted to the candidate verification module, the review selection module, and the review update module. The reference validity is also transmitted to the status output module. The candidate inspection module receives the associated image, reference estimation result, and geometric allowable range of the inspection item. It projects the candidate weld geometry and the compatible parameter combination in the imaging parameters and base material displacement range of the reference estimation result onto the original image, so that the relevant boundaries in the same candidate combination share the corresponding imaging parameters and base material displacement parameters, retains the combination that meets the effective measured characteristics and error constraints, distinguishes between normal candidates and abnormal candidates, and transmits the candidates, specific feasible combinations, and effective image coverage to the verification selection module and the status output module. The review selection module is used to predict the boundary range of each candidate based on the received candidates, parameter range, stage and available observation conditions when neither type of candidate is excluded. It selects at least one pair of dissimilar candidates whose predicted ranges can be separated after accounting for errors, and transmits the recorded predictions, parameter versions and selected positions to the review update module. If no position is selected, the result is transmitted to the status output module. The verification and update module is used to obtain the actual image when the selected location exists. The image association module and the reference estimation module verify the validity of the external reference and the stage. After the verification is passed, the candidates are excluded based on the incompatibility between the actual boundary and the recorded prediction. The parameter range is not extended based on the weld to be inspected. The actual image constraint and updated candidates are sent back to the candidate inspection module. The reason for the verification failure is sent to the status output module. The status output module is used to receive reference validity, candidate test results and review results. When the reference and coverage are valid, there is a specific combination that meets all constraints and all other candidates are excluded, the corresponding normal or abnormal status is output. Otherwise, the output cannot be determined and the unexcluded candidates are retained. When the review deadline has not been reached, the review selection module is fed back the information to be reviewed.

[0014] The technical solution provided in this application has the following technical effects: The protected area prevents the weld to be inspected, suspected anomalies, and areas of strong radiation change from changing the reference parameters used to interpret the image in reverse, thus limiting the possibility of fitting local real anomalies as camera changes from the data source perspective; the fixture reference and the base material reference constrain the imaging changes and actual displacement, respectively, so that the thermal movement of the base material can be retained as a state result.

[0015] By sharing imaging parameters within the same candidate combination, the weld width, center offset, and original boundary maintain a consistent geometric relationship. The effects of common translation on the difference between the two boundary sides are canceled out within the same combination, while errors such as scale that cannot be canceled out are retained. This avoids repeatedly amplifying the same error as two independent errors, and also avoids relying on boundary positions that are advantageous but cannot be simultaneously valid to form a normal weld.

[0016] Verification images are selected based on the predicted separation locations of normal and abnormal candidates, ensuring that supplementary observations correspond to specific unresolved geometric ambiguities. Predictions are saved before actual images are examined, and the use of the region to be examined to expand the reference range is prohibited. This limits the possibility of modifying interpretation conditions in reverse based on new results, ensuring that candidate exclusion has traceable actual image evidence.

[0017] For insufficient coverage, incomplete calculations, and inconsistent constraints, the reasons for not being able to be determined are retained to prevent missing evidence from being treated as normal or abnormal evidence. The above effect comes from the coordination between image region segmentation, geometric projection constraints, and verification data processing, without changing the welding current, voltage, or welding torch motion control parameters. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the multi-source vision fusion method for vehicle welding status recognition provided in this application embodiment.

[0019] Figure 2 This is a schematic diagram of the modules and data transmission of the multi-source vision fusion vehicle welding status recognition system provided in the embodiments of this application.

[0020] Figure 3 This is a schematic diagram showing the positional relationship between the protected area and the fixture reference and the base material reference provided in the embodiments of this application. Detailed Implementation

[0021] The technical solution of this application will be described below with reference to the accompanying drawings. The welding state described herein refers to the weld width, center offset, visible surface interruption, and base material position change that can be supported by actual images. The normal state in the surface image is not equated with internal fusion, internal porosity, or weld joint strength meeting the requirements. Normal and abnormal refer to the listed inspection items and covered welding positions.

[0022] Example 1 The multi-source images in this application can be composed of visible light images from different viewing directions, or infrared images can be added to them. Visible light images are used for boundary and reference feature extraction; when infrared images are added, their radiation variation areas are used for protection area expansion and location correlation. Without infrared images, the protection area is still expanded by taking the union of the preset weld corridor, suspected visible light anomaly areas, and saturation areas according to the mapping error. The original image refers to the image that retains the actual content acquired by each camera and has not been replaced by the fusion result; it does not refer to a separate image source. The following specific configuration is illustrated with the case of adding infrared images; candidate boundary checks still use the corresponding visible light images.

[0023] This embodiment illustrates the complete processing procedure of the method. For example... Figure 1 As shown, multi-source images of the same welding position and stage are processed according to steps one through six. After step three is completed, the output conditions of step six are checked first; if the conditions for determining the conclusion are met, the current detection item ends and no further verification is required. Step five is only entered when both types of candidates have not been eliminated and there is a separable observation position; if there is no selected position or the verification fails, step six is ​​entered, the output cannot be determined, and the system waits for other available observations before the deadline. Figure 1 These conditions are represented by bypass and loop, but this does not mean that all mapping steps are performed for each detection.

[0024] Step 1: Associate multiple source images of the same welding location and stage to delineate the protected area containing the weld to be inspected.

[0025] Specifically, taking the gas-shielded welding station for thin-walled steel pipes and connecting brackets of an electric tricycle as an example, visible light cameras with different observation directions are configured on both sides of the welding torch, along with an infrared camera. Visible light images are used to extract boundaries and reference features, while infrared images are used to identify areas of radiation variation and assist in associating the welding position. Each camera uses a trigger signal or a unified clock to record the start and end times of exposure, with the midpoint between the two times serving as the exposure center time. The original image, along with camera identification, exposure information, and timestamp, is written to a cache; the original image is not saved as a fused image.

[0026] Before welding, the intrinsic parameters, distortion parameters, and positional relationships between cameras are obtained using a calibration board of known dimensions. Coordinate relationships between infrared and visible light are established using calibration patterns that are recognizable in both visible and infrared light. For patterns not involved in calibration fitting, images are repeatedly taken at different field-of-view positions, and the projection residual range is recorded. The image position changes corresponding to camera fixation errors, marker positioning errors, and synchronization errors are included in the error table. Images exceeding the calibration field of view or exposure range cannot directly use the original error limits and should be marked as requiring recalibration.

[0027] Reference marks are fixed to the fixture, and continuously trackable base material textures, contour corner points, or preset reference marks are selected on both sides of the weld, outside the expected heated and highly reflective areas. The fixture reference determines the camera's position relative to the fixture, and the base material reference determines the workpiece's position relative to the fixture; their data are recorded separately. The reference marks are positioned within the camera's visible range, and their spatial distribution constrains the degrees of freedom of the selected imaging transformation.

[0028] Before welding, the joint centerline is established from the joint contour in the binocular image. The arc length along the centerline is used as the position index, and the direction perpendicular to the centerline on the local surface is used as the lateral direction. The planar support part uses planar local coordinates, and the pipe part uses curved surface local coordinates supported by binocular measurement. The correspondence between adjacent coordinate pieces is recorded at the interface. The weld coordinates save the material position identity. Subsequent base material displacement is added separately. The displacement cannot be zeroed by resetting the pre-weld reference. For positions lacking reliable depth, only the projected displacement range supported by existing images is used. Three-dimensional displacements not supported by observations are not output.

[0029] Each record includes at least the workpiece identifier, weld identifier, pass number, arc length range, welding stage, and original image index. The image is correlated to the arc length range using the pre-weld joint, reference markers in the image, and camera position. Robot pose can be used to determine the processing range, but cannot replace reference verification in the image or serve as a signal for judging welding quality. Positional changes caused by time differences are accounted for as error based on the range of motion of the reference features in adjacent frames; if this change cannot be constrained, the two frames are not considered common evidence of the same instantaneous geometry.

[0030] In this embodiment, the welding stage is established as part of the image association record. When an event to be inspected is established, a corresponding stage time interval is established according to the pre-set shooting sequence of the weld position, and the stage time interval is saved together with the workpiece identifier, weld identifier, welding pass number, and weld position. After the image is acquired, the stage time interval into which the image falls is determined according to the exposure center time of the image, and the stage identifier of the image is formed by combining the workpiece identifier, weld identifier, welding pass number, and weld position. Images that are near the stage time boundary, whose exposure time spans two stage time intervals, or whose stage cannot be uniquely determined due to time synchronization errors are not used as the basis for cross-stage candidate exclusion.

[0031] When verifying the validity of a review image at each stage, the workpiece identifier, weld identifier, welding pass, weld position, and stage identifier are compared with the corresponding information saved when generating the original candidate prediction. If the review image and the original candidate belong to the same stage, and the interval between the exposure center time and the original observation time does not exceed the applicable time range of geometric changes that have been verified for that stage, the stage condition is considered met. If the stage identifier has changed, the review image crosses the stage boundary, or although the stage identifier is the same, the observation time interval has exceeded the verifiable geometric change range, then the original candidate prediction is no longer directly used for the exclusionary judgment of the review image.

[0032] When the original candidate prediction is no longer applicable to the verification image, the existing candidate and state records of the original stage are retained. If the image belongs to a new welding stage, the state record is re-established according to the stage corresponding to the image; if it still belongs to the same stage but lacks actual evidence to constrain the geometric changes within that time interval, the image can be used as a new observation record for the current moment, but it is not used to reversely exclude candidates from the original observation moment. This avoids misinterpreting different stages or unconstrained morphological changes as the original candidate being invalid.

[0033] Based on the maximum permissible weld width, maximum permissible offset, and joint positioning error specified in the process documentation, a pre-defined weld corridor is formed on both sides of the joint before welding. This corridor is part of the protection zone from the moment the event is established, regardless of whether an anomaly has been detected. Gradient edges are extracted laterally along the joint in the visible light image. Pixel connectivity areas that extend beyond the pre-defined weld corridor or experience contour interruptions on adjacent sampling lines are designated as suspected anomaly areas; this marking is only used to expand the protection zone and is not directly considered an anomaly conclusion. Saturated pixels are marked according to the camera's actual full-scale range and the near-saturation value determined by calibration. For infrared images, a stable reference area is first used to correct for common variations in the overall radiation readings. Then, areas where the variation relative to the pre-weld reference exceeds the infrared repetition imaging error envelope are marked as radiation variation areas; this marking cannot be used to directly infer the melting or solidification state.

[0034] After mapping the above regions to weld coordinates and taking the union of the sets, the results are then expanded in all directions according to the local coordinate mapping error, yielding... Figure 3 The protected area is shown. The initial expansion uses offline calibration error; subsequent expansions use the error range determined by the most recent valid reference estimate and the current external reference residual. New protected areas added during the current event are merged with existing protected areas and do not shrink due to reduced infrared radiation or temporarily undetected anomalies. If mapping cannot be completed, the original protected area is retained and the reference is marked as invalid; a reference point is not forcibly obtained by reducing the protected area.

[0035] Step two: Determine the interrelated imaging parameters and the range of displacement of the base material solely based on the external fixture and the base material reference.

[0036] Specifically, imaging parameters include the camera pose relative to the fixture and applicable scale corrections; calibrated intrinsic and distortion parameters are kept within their calibration error range. For local images of planar supports, the relationship can be represented by a projection transformation derived from planar geometry; for curved pipe surfaces, the relationship is represented by the camera projection relationship and the pre-weld surface coordinates. It is not possible to freely fit an arbitrarily bendable transformation across the entire area spanning the pipe and support.

[0037] Select a fitting group and an independent inspection group from the fixture marks outside the protected area. The marks in the independent inspection group are called independent inspection reference marks and do not participate in the imaging parameter fitting. Initial values ​​of the imaging parameters are obtained using the known fixture coordinates and actual image coordinates of the fitting group, preserving the parameter range that satisfies the error limits of each fitting point. Furthermore, the projection positions of the independent inspection group must also fall within the corresponding measurement error range. The error limits are derived from offline calibration residuals, the current feature positioning range, and synchronization errors, and should not be increased due to the weld not meeting expectations. If the number of marks meets the requirements but is concentrated in a single line, the same small area, or cannot constrain the degrees of freedom of the transformation used, they are still treated as invalid references. This condition can be checked by the rank of the fitting equation and the coverage of the independent inspection points in the field of view.

[0038] Under each set of permissible imaging parameters, the current image position of the base material references on both sides is compared with the pre-weld position to determine the displacement of the base material relative to the fixture. For locally rigid areas, common translation or rigid body displacement is used; when the reference points support a changing trend, the displacement field obtained by linear interpolation from the reference positions at both ends of the area is used. Interpolation residuals are checked using other base material reference points, and the residuals and errors propagating to the protected area are included in the base material displacement range. When references are sparse, no freely deformable local parameters are added; when the error limit propagating to the inspected position cannot be given, the relevant position is marked as undeterminable.

[0039] Changes in the fixture reference and changes in the base material reference are saved separately. For example, if the fixture mark shifts as a whole in the image while the position of the base material relative to the fixture remains unchanged, the camera change is recorded; if, after removing the imaging changes, the base material reference still moves relative to the fixture, the base material displacement is recorded. The base material displacement can change the current position of the joint, so it is saved as the actual state to be output and cannot be used only to restore the weld image to its pre-weld position and then discard it.

[0040] Step two outputs the reference validity, imaging parameter range, parent material displacement range, corresponding error sources, and parameter version number. Each range retains its common variables and related constraints; for example, the same scale parameter may act on both boundaries simultaneously, or the same fixture reference may act on two cameras simultaneously. These relationships can be saved using interval records with common parameter indices, avoiding the direct conversion of related ranges into independent error terms.

[0041] The interrelated ranges are recorded and stored using the same parameters: each imaging parameter interval is associated with a displacement interval within that interval that satisfies the parent material reference constraints, as well as a reference residual constraint that both must simultaneously satisfy. These reference constraints are re-examined during candidate testing, and it is prohibited to take an independent extreme value from different records and then concatenate them into an incompatible combination. Separate estimations indicate that the two physical sources are different, but do not imply that they are statistically or geometrically independent.

[0042] Reference residual limits are established based on pixel units in image coordinates; tooling size errors given in millimeters are first converted to pixel ranges according to the corresponding imaging scale, including projections to each marker position during rotation. The calibration residual envelope, the current marker positioning range, and positional changes caused by time alignment are recorded separately and then merged outwards; the same error source is only registered once. Independent test points use the same merging limit; the raw calibration residual is not used as its judgment threshold alone.

[0043] Step 3: Project the candidate weld geometry and the compatibility parameter combination within the range onto the original image, so that the relevant boundaries in the same candidate combination share the corresponding imaging parameters and base material displacement parameters, retain the combinations that meet the effective measured characteristics and error constraints, and distinguish normal candidates from abnormal candidates according to the geometric allowable range of the detection items.

[0044] Among them, the candidate weld geometry is the possible range of geometry to be inspected established within the calibrated observation area and the corresponding local coordinate range of the joint. It covers the normal and abnormal geometry that can be inspected by the current inspection item; the allowable range of geometry for the inspection item is obtained from the process parameters of the corresponding workpiece and welding pass.

[0045] Specifically, distortion is corrected in the original visible light image according to calibration data, preserving the correspondence between pixels before and after correction. Local extrema of the gray-level gradient are extracted along the sampling lines transverse to the seam, and candidate edges are formed by connecting extrema with continuous positions and directions. The edge position is determined by the range of local gray-level changes, and the edge direction is determined by the connection direction of adjacent edge points. Edge position and direction errors are estimated using the same exposure, edge extraction rules, and contrast conditions as during calibration. When multiple possible correspondences exist, such as weld boundaries and reflection boundaries, they are stored separately, and contours that do not conform to the expected normal width are not directly deleted.

[0046] Edge extraction can be performed on the distorted image, but the measured points used for joint verification must be returned to their original image coordinates via the saved pixel correspondence, and their location range is also transferred with the mapping; candidate projections include the corresponding distortion effects. Therefore, projected points and measured points are compared in the same original image coordinates, and the distorted coordinates are not directly subtracted from the original coordinates. Orientation differences are compared by the angle between undirected edges, taking the angle between two straight lines that does not exceed a right angle, expressed in degrees, and using the same directional location error limit.

[0047] Width, center offset, and continuity are recorded separately for each inspection item. The center offset item uses the lateral position of the midpoint of the left and right boundaries relative to the current joint; relative offset represents this center offset, and no other undefined offset is set. Projection position error is expressed in pixels, while process width, center offset, and interruption length are expressed in millimeters in the local coordinates of the weld. The conversion between the two is completed through the same parameter combination.

[0048] Invalid locations are identified using saturation masks, known component projections, occlusion boundaries between preceding and following frames, and visibility checks in both views. If blurring causes the boundary localization range to exceed the selected detection resolution, that location is marked as uncovered, and smoothed transitions between light and dark areas are not considered reliable boundaries. The fusion process combines features from each original image under common coordinate and parameter constraints, outputting the geometric range and state; it does not require the generation of a single synthetic image as the sole criterion.

[0049] The candidate weld geometry describes the left and right boundaries, center position, and visible interruption endpoints within the inspection area. One feasible approach is to represent the two boundaries with fixed nodes along the weld direction, connected by straight lines. Nodes are added at corners or locations where the measured edge significantly changes direction. The upper limit of the deviation between nodes and the actual curve is determined based on the calibration target and actual edge sampling, taking into account projection error. If this upper limit cannot be obtained, the area is reduced or marked as an undetermined location, and unobserved interruptions are not artificially crossed using straight line connections.

[0050] The candidate initial geometric range is determined by the calibrated observation area and the local coordinate range of the joint, covering both normal and abnormal geometry within this range; width or offset cannot be limited to only normal values. If either side boundary exceeds the inspectable range, the output cannot determine and the reason for exceeding the inspectable range will be recorded; the outside of the observation range cannot be interpreted as the absence of anomalies. The allowable geometric range is read from the process document corresponding to the workpiece and welding pass, specifying at least the upper and lower limits of width, offset limits or allowable interruption length, and the applicable stage. A reference sample is used to verify boundary extraction and error limits; its appearance does not replace the allowable process range.

[0051] When creating parameter units, the normal process range and its complement within the inspected geometry should be included; values ​​should not be taken only around a single extracted contour. The lower and upper width limits themselves fall within the normal range; values ​​below the lower limit or above the upper limit are classified as abnormal. Offsets and interruption lengths are handled according to their respective allowable limits. The upper bound of the geometric residual between nodes is only used for already verified areas, observation scales, and edge conditions; when actual edges show bends not covered by this upper bound or new interruptions, nodes should be added and re-covered; actual anomalies cannot be inferred from the complete exclusion by old units. For visible interruptions, endpoints and branches with or without interruptions can be enumerated according to image-resolvable seam-side units; interruptions with resolution less than this are not excluded.

[0052] The initial coordinate mapping of the protected area uses pre-weld calibration and pre-weld reference, and subsequent mapping only uses existing valid reference versions; therefore, it is not necessary to re-estimate the parameters of the weld within the protected area before generating the protected area. If the external reference cannot transmit a valid displacement limit to the heated zone, then that area remains undetermined. Whether the required limit can be obtained through on-site marking distribution and binocular imaging must be determined by actual verification; a finite number of reference residuals themselves do not constitute absolute error proof for all future operating conditions.

[0053] Each candidate combination consists of a set of candidate geometries, a set of camera imaging parameters, and compatible parent material displacements. The geometric nodes are first transformed to the current fixture coordinates according to the parent material displacement, and then projected onto the original image of that camera using the same camera pose, intrinsic parameters, and distortion relationships. The same set of parameters is used for the left and right boundaries in the same image; the fixture reference and dimensional errors shared by multiple cameras use the same variables. The projected boundary position and orientation are compared with the corresponding measured boundaries and their error ranges; if any valid correspondence is incompatible, the combination is excluded.

[0054] For multiple possible contours, retain the corresponding branches and examine all valid boundaries within each branch. An edge should not be labeled as reflective simply because it does not support a normal candidate, nor should the occlusion area be temporarily expanded to retain a normal candidate. Uncertain areas caused by occlusion, saturation, and reflection must be formed using actual image quality detection and visibility information and saved with the original image.

[0055] For cases where a boundary is predicted but not detected in the actual image, first confirm that the location is within the camera's field of view, unobstructed, unsaturated, and that the image sharpness meets calibration requirements. Then confirm that the edge contrast under the same material, viewing angle, and exposure conditions is within the verified detectable range. Only when all the above conditions are met can the prediction be excluded due to boundary absence; if any condition is not met, the record lacks effective coverage. Normal image brightness alone cannot guarantee that a boundary can be detected.

[0056] To avoid using a small number of discrete samples to represent all possible parameters, this embodiment uses parameter interval cells to store candidates. The initial cell covers all allowable imaging parameters, parent material displacement, and candidate geometric ranges; shared variable indices are retained within the same cell. The boundary position obtained by coordinate transformation and projection of this cell is represented by an enclosing range that includes all possible projections. During calculation, the range is rounded outwards, and the projected depth range is required to be within the effective imaging space of the camera. If the enclosing range does not intersect with a certain measured boundary error range, there is no combination within the cell that satisfies the constraint, and it can be deleted.

[0057] For elements that still have overlap, first select the parameter interval with the largest projection variation span and bisecte it from the midpoint. Then, calculate the outer bound and geometrically permissible range relationships of the sub-elements separately. Projection operations can be performed item-by-item interval operations. Planar projections can use linear enclosing relationships with residual terms within the permissible range. The mere presence of overlap does not indicate that a practically feasible combination has been found. For specific parameter values ​​selected from within the element or obtained through constraint solving, recalculate all effective boundary residuals item by item. Only after confirming that all are within the permissible error should a feasible mark be recorded. Specific parameter values ​​that are at the numerical rounding boundary and cannot be confirmed are not marked as feasible.

[0058] Candidates are categorized into normal and abnormal candidates based on whether their specific geometric values ​​meet the allowable range of the corresponding detection item. Interval cells crossing allowable boundaries are split according to that boundary, or both labels are retained simultaneously; it is not permissible to classify all cross-boundary cells as normal. Cells that cannot be deleted when the computation time limit, minimum subdivision scale, or numerical discrimination capability boundary is reached are marked as not yet subject to discrimination. For a class of candidates to be completely excluded, all cells covering the entire initial range of that class must be proven incompatible; the absence of a class cannot be based on the solver not finding a sample, local solution failure, or timeout.

[0059] The width is obtained by subtracting the right and left boundaries of the same lateral position in the retained combination, and the relative offset is obtained by comparing the midpoint of the two boundaries with the current seam position. Each calculation uses the same set of common parameters, and the results of all combinations are aggregated to form the geometric range. Discontinuous interval branches are retained internally; even if the total bounding range is given during display, gaps between branches cannot be filled. This output is further used for candidate classification and verification prediction. Specific feasible combinations are used to prove the existence of feasible solutions for the corresponding category; the geometric range should cover the retained unit and all unexcluded combinations under its constraints, and cannot be replaced by the minimum and maximum values ​​of the found specific feasible combinations. When the interval operation cannot obtain a precise boundary, the outer bounding range is retained, and candidates that have not yet been judged are not deleted.

[0060] Step 4: If neither type of candidate is excluded, predict its boundary range in the available verification image, select the position where the predicted range of at least one pair of heterogeneous candidates can be separated after accounting for the error, and record the prediction and parameter version.

[0061] Specifically, each normal candidate branch is paired with each abnormal candidate branch to determine the weld location and inspection item corresponding to the current ambiguity. Candidate verification images can come from another camera at the same time, cached images of the area to be inspected that have not yet been read, or images that the camera can acquire at subsequent times according to a predetermined running path. Candidate positions are given by the camera's field of view, the existing robot trajectory, and known component occlusion relationships; this embodiment does not require changes to the welding control parameters.

[0062] The stage identifier of the candidate verification image is determined in accordance with the method described in step one; the verification image is used for prediction comparison of the original candidate only when its stage identifier is consistent with the stage of the original candidate prediction record and the observation time interval is within the applicable range of geometric changes that have been checked for the corresponding stage.

[0063] For each candidate image, the retained normal and abnormal geometric branches are projected onto their respective observation coordinates. The camera pose range is derived from known installation relationships, trajectory, and external reference changes; the morphological change range within the verification interval is derived from the boundary change range already acquired in the same stage and the corresponding measurement error, and its applicable time interval is verified using images from the same stage that were not involved in determining this range. When extrapolating along time, the boundary range is expanded to cover allowable changes; if the time interval exceeds the verified time interval, extrapolation is not continued. When there is a lack of actual evidence to support the limits of subsequent morphological changes, the current morphology must not be considered fixed; instead, other views at the same time or cached images corresponding to the original observation time should be used; cached images with only the same stage identifier but no time change constraints cannot be used to exclude the original candidate. When crossing the molten pool evolution and post-weld forming stages, state records are established separately; the candidates from the two stages cannot be directly intersected.

[0064] During implementation, images from other observation directions at the same time are prioritized for exclusionary verification. For images at subsequent times, the maximum change in historical samples can only serve as a basis for arranging observations; subsequent images are only added to the original candidate constraints when the applicable stage, time interval, and external references have all been verified, and there is a verified range of change. If the validity of the range of change cannot be maintained, subsequent images are only used to establish a new state record for that time, and the original record is still verified by evidence from the same time. The maximum change in a finite sample is not declared as a necessary upper bound for future physical changes.

[0065] For each pair of boundary locations, imaging error, pose prediction error, and intra-stage geometric change error are respectively added. The range interval is compared at both visible lateral sampling locations. If the ranges overlap, their separable interval is treated as zero; if they do not overlap, the distance between the nearest endpoints of the two ranges is taken. If this distance exceeds the minimum resolvable interval, the pair is considered separable at that location. The minimum resolvable interval is determined by a calibration target with a known edge spacing under the same optical conditions and includes resolution constraints for pixel sampling and edge extraction; if this calibration is not completed, the location is not used for exclusionary verification.

[0066] To ensure that the pair count is unaffected by the number of subdivisions, at the start of the verification selection, normal and abnormal candidate branches are divided based on the value ranges of candidate weld geometry, imaging parameters, and base material displacement. The candidate branch numbers are then frozen, and subsequent interval subdivisions retain the original branch numbers. For all sub-intervals of the same normal and abnormal original branches, a count is only made once if their overall prediction range is separable. Priority is given to the position with the highest number of separable pairs; if the number is the same, the position with the shortest waiting time is selected; if the waiting time is still the same, the position requiring the least image area to be read and processed is selected. Positions that have not yet formed separable pairs remain pending verification; a clearer image is not automatically considered as having discriminative ability.

[0067] The separability condition applies to at least one pair of normal and anomalous branches, rather than requiring that the total envelope of all normal geometries and all anomalous geometries be separated simultaneously. A separable pair only indicates that the observation has a chance to exclude one of the branches; final determination still requires the exclusion of all opposite categories. Especially near the process threshold, the two predictions can always overlap, leaving the outcome undecided and not creating separation by reducing the candidate range.

[0068] If new reference data, new observation locations, or changes in effective image coverage occur during the review waiting period, the review will be rescheduled based on the remaining candidates. If only time has passed and no new data is available, the same set of constraints will not be solved repeatedly. When a reference is invalid, after new reference data arrives, return to step two to estimate the reference, and then perform the candidate test. When the reference is valid and both types of candidates coexist, return to step four only when new available observations are available. The deadline is fixed when the event is established; re-estimation and candidate subdivision do not extend the deadline. If a reference or stage is inapplicable, re-estimation of data outside the region can be requested, or other valid observations can be waited for, but failure predictions must not be used to exclude candidates.

[0069] Before reading the image content of the selected actual region to be inspected, save the normal and abnormal candidate numbers, predicted boundary range, error source, camera and observation position, planned observation time, effective stage, and parameter version. For cached images, it is allowed to read the timestamp, camera position, and external references first to determine candidate availability, but the region to be inspected cannot participate in the previous prediction settings. The saved prediction records constitute the comparison objects for subsequent actual image inspection.

[0070] Step 5: If the selected location is available, obtain the actual image, verify the validity of the reference outside the area and the stage, and after the verification is passed, exclude candidates based on the actual boundary and the recorded incompatibility relationship, without expanding the parameter range of the weld to be inspected.

[0071] Specifically, after reading the actual verification image, first update the saturation and occlusion markers and verify the fixture and base material references outside the protected area. Simultaneously, check the workpiece identification, weld identification, welding pass, weld position, and stage identification of the actual verification image, and verify that the actual observation time is still within the stage and time range applicable to the recorded prediction. If the stage has changed or the range of change within the stage can no longer cover the current observation interval, stop using the original prediction to exclude candidates, and re-establish or update the status record according to the corresponding stage. If the parameters corresponding to the current reference do not fall within the saved prediction parameter range, the image cannot be used for candidate exclusion under the original version; record the reason for the reference inconsistency and proceed to re-estimation. If the reference is valid and the original prediction is applicable, obtain the measured boundary range using the same edge extraction method as in step three, and compare it with the saved prediction range.

[0072] If the actual boundary range is incompatible with a candidate's predicted range, that candidate is excluded; if there is an intersection, the actual boundary is treated as a new constraint and returned to step three for combined testing. Only the branches that an observation can negate are excluded; the exclusion of a single normal branch cannot be considered as the exclusion of all normal candidates. When a new image is occluded or lacks sharpness, only the coverage status is updated; candidates are not deleted simply because no boundary was detected in this instance.

[0073] If all saved candidates are incompatible with the actual image, and the recorded image constraints are inconsistent with the measured image, check for loose reference markers, faulty camera calibration, changes in stage identity, and omissions in contour extraction. Then, create a new parameter version using only reference data outside the protected area. The new version is allowed to use new independent reference residuals and recalibration results, but a wider error range cannot be deduced from the target that the weld to be inspected should be normal. Reprocess the saved actual image according to the new version; old parameters, old predictions, and reasons for inconsistency are retained. If the new version still cannot obtain feasible candidates, the output cannot be determined.

[0074] Step 6: When the image coverage of the reference and corresponding detection items is valid, and one type of normal candidate and abnormal candidate has a specific combination that meets all constraints, while the other type of candidate is completely excluded, output the normal state or abnormal state corresponding to the type with the specific combination. If the normal or abnormal state output conditions are not met, output "cannot be determined" and retain the unexcluded candidate. If both types of candidates still coexist and the review deadline has not been reached, continue the review. If the candidate combination test has not been completed, continue the candidate test.

[0075] Specifically, effective coverage is established based on detection items. The width item requires that both sides of the corresponding cross-section have usable measured constraints; the offset item also requires that the current seam reference has a valid reference; and the continuity item requires that the interval to be judged has a continuous observation range capable of detecting boundaries or interruption endpoints. The adjacent boundaries of the field of view boundary and invalid areas such as occlusion and saturation are shrunk towards the inside of the effective area according to the corresponding positioning and mapping errors. Only the position where the candidate projection falls into the shrunk area under all retained parameters is counted as effective coverage. For adjacent sampling positions, coverage is extended only when the upper limit of boundary change is controllable; otherwise, it is retained as an unjudged gap. The status of the covered sub-interval can be output separately, while the entire requested interval is recorded as insufficient coverage. Partial normality cannot replace full-segment normality.

[0076] When the reference is valid, the requested detection item has complete coverage, all abnormal candidates are excluded, and there are specific feasible normal candidates, output "normal". When all normal candidates are excluded and there are specific feasible abnormal candidates, output "abnormal", and save the abnormal location and geometric range. When there is another type of candidate that has not yet been identified, no definite conclusion is output. When both types of candidates are excluded, output "cannot be determined" and indicate that the constraints are inconsistent; when both types still coexist, output "cannot be determined" externally, record the internal pending review status and continue to step four before the review deadline is reached; save the corresponding reason when the reference is invalid or the coverage is insufficient. If the other type is completely excluded and there are specific feasible combinations in the remaining category, the fact that other intervals in the remaining category are not yet fully subdivided does not prevent the output of the definite status of the category.

[0077] When an event to be inspected is established, cutoff conditions are set based on the time that the same location can remain within the effective field of view and the allowable review time limit of the workstation. If the time limit is reached, the location leaves all available fields of view, or the status cannot be determined after completing all available observations, the review ends and an "Undetermined" message is output. The output record includes the stage, inspection item, status, geometric range, reason for candidate exclusion, undetermined location, original image index, reference version, and review prediction index. Confirmed process anomalies are retained in the corresponding stage record; subsequent normal finished appearances do not overwrite this record.

[0078] The following example inputs illustrate the complete data chain. All values ​​are illustrative parameters set to demonstrate projection and candidate tests and do not represent actual equipment performance, experimental results, or production acceptance values.

[0079] Select a segment of a straight weld within the same forming stage, with the inspection range along the weld direction from 0 mm to 1 mm. The example uses a local plane, fixed depth, and a calibration range where the lateral linear projection is valid. The left and right boundaries of this segment are parallel, and the geometry of the segment is described by the lateral position and width of the left boundary. The example input already covers the entire segment, with no additional bends or interrupted branches between nodes. The position of the left boundary relative to the pre-weld local coordinate origin ranges from -0.10 mm to +0.10 mm, and the candidate width is 1.40 mm to 2.40 mm. The common lateral displacement of the base material is from -0.02 mm to +0.02 mm. The allowable range for the process width is 2.00 mm to 2.40 mm; therefore, candidates with widths less than 2.00 mm are considered width aberration candidates.

[0080] Two cameras observe the same lateral direction. Camera 1 has a lateral scale of 9.98 to 10.02 pixels per millimeter, with an image reference position at pixel 100. Camera 2 has a lateral scale of 19.98 to 20.02 pixels per millimeter, with an image reference position at pixel 200. The common translation correction for both cameras is -0.20 to +0.20 pixels. The listed parameters are given by the example's out-of-area reference constraint, which applies to all combinations of intervals in this example. In practice, if relevant constraints exist, these constraints should be retained, and intervals should not be treated as independent intervals.

[0081] For any boundary, first add its local lateral position to the parent material displacement, then multiply by the same scale value of the selected camera, and finally add the image reference position of that camera and a common translation correction to obtain the predicted pixel position. The left and right boundaries must use the same parent material displacement, the same camera scale, and the same translation correction. The local position of the right boundary is equal to the left boundary position plus the width. This calculation also specifies the parameter units: millimeters multiplied by the number of pixels per millimeter to obtain the pixel value, then added to the pixel reference and translation.

[0082] The actual left boundary positioning range of camera one is from pixel 99 to pixel 101. There are two contours on the right side that meet the edge extraction criteria, with positioning ranges of pixels 114.5 to 115.5 and 120.5 to 121.5 respectively. These two contours form corresponding branches, and one cannot be deleted based on the lower limit of the process. Taking an example where the left boundary position, parent material displacement, and translation correction are all zero, and the scale is 10 pixels per millimeter, a width of 1.50 mm corresponds to pixels 100 and 115, and a width of 2.10 mm corresponds to pixels 100 and 121. These two specific combinations demonstrate that both abnormal and normal patterns can explain the actual image from camera one.

[0083] At the start of the review selection, a width of 1.40 mm to 1.70 mm is designated as a fixed anomalous original branch, while candidates smaller than 2.00 mm are retained in other anomalous branches, and a width of 2.00 mm to 2.40 mm is designated as a normal original branch. This division covers the entire width range in this example, without deleting difficult-to-determine candidates close to 2.00 mm, and the interval subdivision does not generate additional pair counts. Camera 2 is able to observe the aforementioned two side boundaries, and its region from pixel 195 to pixel 255 remains valid after accounting for field of view, saturation, and occlusion errors.

[0084] According to the parameter range projection of Camera 2, the minimum position of the right boundary of the normal original branch is no less than pixel 237.3624, and the maximum position of the right boundary of the selected abnormal original branch is no greater than pixel 236.6364. Then, an extra 0.10 pixels are added outwards as a positioning margin for this edge prediction comparison, and two decimal places are retained, resulting in the lower bound of normal prediction at pixel 237.26 and the upper bound of abnormal prediction at pixel 236.74, with a 0.52 pixel gap between them. The example sets the calibrated minimum resolvable gap to 0.50 pixels, therefore this pair can be separated, and Camera 2 can be selected. Other branches close to the threshold, even if they cannot be separated from the normal branch, are retained.

[0085] Before reading the actual area to be inspected by Camera 2, the aforementioned predictions, parameter versions, branch identities, and observation times are fixed. Camera 2 uses synchronous images from the same exposure time, without introducing subsequent morphological evolution. After actual reading, the external reference and stage verification pass, with the left boundary, including positioning error, ranging from pixel 199.90 to pixel 200.10, and the right boundary from pixel 229.90 to pixel 230.10. The difference between the two boundaries is 29.80 to 30.20 pixels. Since the translation of the same combination and the common displacement of the parent material cancel out the difference, this difference range is divided by a scale range of 19.98 to 20.02 pixels per millimeter, yielding an outwardly rounded width range of 1.4885 to 1.5116 millimeters.

[0086] The entire range is less than 2.00 mm, therefore all normal candidates covered in this example are incompatible with the measured constraints of Camera 2. The left boundary position, parent material displacement, and translation corrections for both cameras are set to zero, with a width of 1.50 mm. The camera scales are 10 pixels and 20 pixels per millimeter, respectively. This yields the boundary positions of pixels 100 and 115 on Camera 1 and pixels 200 and 230 on Camera 2, all within the corresponding actual boundary range. This specific combination provides a recalcible basis for non-empty abnormal candidates. Both the reference and the entire double-boundary coverage are valid, therefore the width is output as abnormal, and the two original images, their correspondence, parameter versions, and width range are saved.

[0087] This example does not rely on the pre-assertion that "other branches have been excluded" to arrive at a conclusion. The exclusion of all normal candidates comes from the width difference constraint under common parameters, and the existence of anomaly categories comes from the given specific parameter combinations. If the boundary is not parallel, the displacement change within the protected area cannot be enclosed by the reference, or there is an unenumerated interruption, this simplified example is not applicable, and the process should be extended according to the aforementioned node and error handling, or the output should be undetermined.

[0088] Example 2 This embodiment illustrates weld width identification when camera translation is present, focusing on the processing of common parameters between the two boundaries. Image acquisition, protected area, reference verification, and status output all adopt the processing relationships of Embodiment 1.

[0089] The following values ​​are illustrative parameters. Assuming that on the same cross section, the calibrated left boundary position is 0.50 mm and the right boundary position is 2.00 mm, the allowable common lateral translation of the fixture reference is -0.60 mm to +0.60 mm, and the positioning error of each of the two boundaries is -0.05 mm to +0.05 mm. Other errors are not considered for the time being.

[0090] If the common translation is applied as two independent quantities to the two side boundaries, a width range of 0.20 mm to 2.80 mm can be constructed. However, some of these upper and lower boundaries require the same camera to move simultaneously in two opposite directions within the same image, which cannot correspond to a single actual parameter combination. In this embodiment, the same translation value is used for both side boundaries in each combination. The difference between the two boundaries is first taken to obtain a width range of 1.40 mm to 1.60 mm. This processing only cancels the common translation and does not eliminate the positioning errors of each boundary itself.

[0091] Furthermore, if the allowed common scale coefficient for independent calibration is between 0.98 and 1.02, and the aforementioned converted position is the result of applying this scale to the actual position, then the aforementioned width range is tested together with the same scale coefficient, yielding an outer width range of approximately 1.3725 mm to 1.6327 mm, where the lower bound is rounded down and the upper bound is rounded up. Assuming the lower limit of the process width for this detection item is 2.00 mm, the reference is valid, and the double boundary coverage is complete, this range is entirely below the lower limit, and normal candidates can be excluded. After confirming that there are specific combinations within the abnormal range that satisfy the image constraints item by item, the output width is insufficient.

[0092] For cases where the dimensional error range cannot be confirmed, this embodiment does not use the above results, but instead retains the influence of the unknown scale on the width and outputs "cannot be determined". This embodiment illustrates that the common parameter relationship is used to exclude error combinations that cannot be simultaneously true, and does not mean that any error can be canceled out, nor does it rely on expanding or shrinking the allowable range of the process to change the detection result.

[0093] Example 3 This embodiment illustrates the handling of situations where both thermal displacement of the base material and weld offset relative to the joint exist simultaneously. Using the fixture reference and base material references on both sides of Embodiment 1, camera changes, joint position changes, and weld offset relative to the joint position are recorded separately.

[0094] The following is an example input. Before welding, the position of a certain area of ​​the joint is saved based on the fixture. The current fixture reference passes inspection. After deducting the changes in camera imaging, the base material references on both sides support the joint of this area to move 0.58 mm to 0.62 mm in the positive lateral direction; the relative offset obtained by the joint boundary and the current joint is 0.18 mm to 0.22 mm. The above ranges form a compatible combination under shared parameter constraints. The outer offset range of the weld center relative to the pre-weld reference is obtained from the joint displacement and relative offset of each combination, which is 0.76 mm to 0.84 mm.

[0095] Assume the example process specifies that the weld offset relative to the current joint should not exceed 0.30 mm, and the joint displacement relative to the pre-weld reference should not exceed 0.40 mm. With complete coverage and feasible combinations of relevant detection items, the weld offset relative to the joint item will output "normal," while the base material position change item will output "abnormal." Simultaneously, the offset range of the weld center relative to the pre-weld reference will be recorded. The system does not merge the two items into a single, undifferentiated "weld offset," nor does it clear the displacement abnormality by re-registering the deformed base material to its original position.

[0096] If one side of the base material reference is obscured, and the displacement transmission error across the protected area cannot be determined, the output of the base material position change item and the offset item dependent on the joint reference cannot be determined; the width item with independent double boundary constraints can still be processed according to its own coverage. This branch ensures that the reference failure of one detection item will not be incorrectly propagated as all detection items being normal or all detection items being abnormal.

[0097] Example 4 This embodiment illustrates the handling of inconsistent constraints, permanent occlusion, and stage transitions. Image acquisition and candidate verification adopt the processing method of Embodiment 1.

[0098] In the first scenario, the system has saved the verification prediction, and the weld boundary of the actual new image is incompatible with all retained candidates. If the independent inspection points of the fixture reference also exceed the original prediction range, the reference verification failure is recorded first; if the fixture reference passes but the weld boundary is still incompatible, the image constraint inconsistency is recorded, and the base material reference, stage association, and contour correspondence are checked. Welding anomalies are not directly output in either case. After obtaining the new version range using the new reference outside the region, the original image is reprocessed; a definitive state is only output when the new version obtains a clear candidate relationship and the coverage meets the requirements, while the old version and inconsistency records are retained.

[0099] In the second scenario, the connecting bracket obscures one side boundary of the weld to be inspected, preventing any available cameras from capturing a valid observation of that location within the review period. This missing boundary cannot be supplemented with measured evidence using interpolation from adjacent segments or predicted images. The system records all remaining normal and abnormal candidates, as well as obstructed locations. When the cutoff condition is met, it outputs that the width detection item cannot be determined, while retaining individual results for other covered locations.

[0100] In the third scenario, an apparent interruption at a certain location has been confirmed using valid original images during the welding process, and a continuous profile appears at that location during the post-weld forming stage. The system establishes process records and forming records for the same workpiece, weld, and arc length range, respectively; the latter updates the continuity results of the formed surface without revoking existing process anomalies based on factual evidence. If there is a lack of verifiable morphological evolution boundaries between the two stages, the forming image is not used to negate process candidates.

[0101] Example 5 like Figure 2 As shown, based on the same inventive concept as the foregoing embodiments, this embodiment provides a vehicle welding status recognition system using multi-source vision fusion. The system includes an image association module, a reference estimation module, a candidate verification module, a review selection module, a review update module, and a status output module. Each module is configured according to... Figure 2 The data relationships shown are used to collaboratively process the original image.

[0102] The image association module receives visible light and infrared images along with their exposure times, camera identifiers, and workpiece information. It then performs time association, weld location association, and protection area merging as described in step one, outputting the original image index, weld coordinates, stage identifier, protection area, and image quality markers. The associated data is passed to the reference estimation module and the candidate verification module, while the location, stage, and available observation conditions are also passed to the review selection module. When the review image arrives, it is associated according to the same rules to prevent images from other locations or stages from being mixed into the current candidate image.

[0103] The available observation conditions consist of the field of view and the positional relationship between cameras in the camera calibration file, the set shooting sequence, the original image cache index, and the predetermined motion trajectory of the workstation. The image association module only organizes and transmits these existing inputs without generating new welding control commands. The candidate inspection module reads the geometric allowable range from the process parameter records of the corresponding workpiece and pass; the status output module reads the verification cutoff conditions from the event record, so these parameters all have a clear external configuration source. Figure 2 The lateral bus in the diagram represents the parallel transmission mentioned above, which does not require a module to receive all the outputs of the previous module in sequence before it can start working.

[0104] The reference estimation module receives the associated image and the protected area, extracts the fixture and base material references only from outside the area, and performs step two: parameter range estimation, independent reference verification, and displacement transfer error determination. This module outputs the imaging parameter range, base material displacement range, validity, and version number, which are then passed to the candidate verification module and the review selection module. The reference validity is directly passed to the status output module, providing the review update module with the imaging parameters of the actual review image and the base material displacement verification results. When a re-estimation request is received, only the reference outside the area is used; range expansion requests based on the condition that the weld should be normal are not accepted.

[0105] The candidate verification module receives the original image features, reference range, and allowable range of process geometry, and performs step three: candidate construction, shared parameter projection, interval exclusion, and specific feasible combination verification. It outputs normal and abnormal candidate branches, interval units that have not yet been judged, the coverage range and geometric value range of each detection item, and passes them to the review selection module and the status output module. After receiving the measured constraints returned by the review update module, it re-verifies, retaining the original branch identity and saving the exclusion reasons.

[0106] The review and selection module receives the two unrejected candidate categories and coverage information from the candidate verification module, the location, stage, and available observation conditions from the image association module, and the interrelated parameter ranges and versions from the reference estimation module. It then performs step four: boundary range prediction and separable location selection. The module outputs the observation location, camera, time, stage, frozen candidate pairings, and prediction versions to the review and update module. If no separable location is found, the module returns the reason to the status output module, preserving the processing entry point for subsequent valid observations.

[0107] The review and update module receives the review task and the new actual image, calls the image association module to determine the location and protected area, receives the verification results from the reference estimation module, and performs the prediction and measurement comparison in step five. This module returns the candidate exclusion results, new measured boundary constraints, and reasons to the candidate verification module, and sends the update results to the status output module. If all candidates are incompatible, it sends a request to the reference estimation module to re-estimate based on data outside the region, while retaining the original prediction and version. The prediction results are only used for selection and comparison and cannot be passed to measured feature data fields.

[0108] The status output module receives reference validity, candidate retention, feasible marker, effective coverage, geometric range, and cutoff conditions, and performs single-item status judgment, stage recording, and result saving in step six. If verification is still required and the cutoff condition has not been met, the remaining candidates and undetermined positions are fed back to the verification selection module; if re-estimation is required but not yet completed, the "undeterminable" marker is retained. Its final output includes the status and the corresponding actual image evidence index, and presents the current joint displacement, weld-to-joint offset, and pre-weld reference offset, respectively.

[0109] The system modules are divided according to the above data relationships, and the specific execution methods are consistent with the aforementioned method embodiments. Image association, out-of-area reference estimation, shared parameter verification, verification location selection, actual image update, and status output together constitute a processing closed loop. The data transmitted between modules retains the workpiece, position, stage, and version identifiers.

[0110] The example values ​​are only for illustrating the processing procedure; the actual parameters and status conclusions are determined according to the above process documents, calibration, measured images, coverage and candidate test results.

Claims

1. A vehicle welding state recognition method based on multi-source vision fusion, applied to electric tricycle frame welding, characterized in that, include: By associating multi-source images of the same welding location and stage, a protective area containing the weld to be inspected is delineated. The interrelated imaging parameters and the displacement range of the base material are determined by the reference of the clamp outside the protected area and the reference of the base material; The candidate weld geometry and the compatibility parameter combination within the range are projected onto the original image, so that the relevant boundaries in the same candidate combination share the corresponding imaging parameters and base material displacement parameters, retaining the combination that meets the effective measured characteristics and error constraints, and distinguishing normal candidates from abnormal candidates according to the geometric allowable range of the detection item; When neither type of candidate is excluded, predict its boundary range in the available verification image, select the position where the predicted range of at least one pair of heterogeneous candidates can be separated after accounting for the error, and record the prediction and parameter version. When a selected location is available, an actual image is obtained. The validity of the reference outside the area and the stage is verified. After the verification is passed, the candidates are excluded based on the incompatibility between the actual boundary and the recorded prediction. The parameter range is not extended based on the weld to be inspected. When the image coverage of the reference and corresponding detection items is effective, and one of the normal candidates and the abnormal candidates has a specific combination that meets all constraints, and the other candidate is completely excluded, the normal state or abnormal state corresponding to the one with the specific combination is output. If the normal or abnormal state output conditions are not met, the output cannot be determined and the unexcluded candidate is retained. Among them, if the two types of candidates still coexist and the review deadline has not been reached, the review continues. If the candidate combination test has not been completed, the candidate test continues.

2. The vehicle welding state recognition method based on multi-source vision fusion according to claim 1, characterized in that, The multi-source images include visible light and infrared images from different viewing directions, and multi-source images associated with the same welding position and stage are used to delineate a protective area containing the weld to be inspected, including: Based on the exposure center time, weld identification and welding position, the images are associated and the position to be inspected and the reference position are mapped to the weld coordinates established along the weld length direction and the transverse direction; The pre-defined weld corridor, the suspected abnormal area in the visible light image, the radiation change area in the infrared image, and the image saturation area are combined under the weld coordinates and then extended outward according to the coordinate mapping error to obtain the protection area. Within the same inspection event, the subsequently generated area is incorporated into the existing protection area, keeping the preset weld corridor within the protection area, and excluding the pixels within the protection area from the fitting data of imaging parameters and base material displacement.

3. The vehicle welding state recognition method based on multi-source vision fusion according to claim 2, characterized in that, The interrelated imaging parameters and the displacement range of the base material are determined by the clamp reference outside the protected area and the base material reference, including: The pose and scale of the camera relative to the fixture are determined by using a reference mark fixed relative to the fixture. The reference mark that is not involved in the fitting is used as an independent test reference mark to test its projection residual. The range of imaging parameters is determined by combining the calibration error. Within the range of the imaging parameters, the positional change of the base material relative to the pre-welding reference is determined by using the base material reference features located on both sides of the protected area, and the base material displacement range is determined based on the reference feature measurement error and the displacement error transmitted to the position to be inspected, and the correspondence between each base material displacement range and the imaging parameters that generate that range is retained. The changes in camera position and the changes in the position of the base material are recorded separately. When the distribution of the reference mark cannot constrain the degrees of freedom of the transformation used, or when the projection residual of the independent inspection reference mark exceeds the corresponding measurement error range including the calibration error and the current positioning error, the reference estimate is marked as invalid and the reference estimate is not restored by reducing the protection area.

4. The vehicle welding state recognition method based on multi-source vision fusion according to claim 1, characterized in that, The candidate weld geometry and the compatibility parameter combination within the specified range are projected onto the original image, so that the relevant boundaries in the same candidate combination share the corresponding imaging parameters and base material displacement parameters. Combinations that meet the valid measured characteristics and error constraints are retained. Normal candidates and abnormal candidates are distinguished according to the geometric allowable range of the detection items, including: The weld boundary position, boundary direction, and occlusion and saturation markers are extracted from the original visible light image in the multi-source image. When there are multiple contours that meet the edge extraction conditions, the contour correspondence is preserved respectively. The candidate weld geometry, imaging parameters and base material displacement are combined, and the same scale, reference position and base material displacement in the same combination are kept consistent between the boundary projections. The projected boundary is compared with the corresponding valid measured boundary to exclude combinations whose position or orientation difference exceeds the corresponding error range; for occluded or saturated locations, the lack of a boundary is not used as a basis for exclusion. Only when there are verified boundary detection conditions at the corresponding location, the combination of predicted boundary absence is excluded by using the missing measured boundary.

5. The vehicle welding state recognition method based on multi-source vision fusion according to claim 4, characterized in that, The retention and exclusion of combinations include: The range of values ​​for candidate weld geometry, imaging parameters, and base material displacement is covered by parameter interval elements, and the projection boundary enclosing range corresponding to each parameter interval element is calculated. The parameter interval cell is excluded only when the error range of the outer bounding range does not intersect with the effective measured boundary, or when all combinations within the parameter interval cell violate the same projection constraint; the parameter interval cells that cannot be excluded are further subdivided, and the parameter interval cells that have not yet been judged are retained. For each specific parameter combination that satisfies all reference constraints and image constraints, a feasible mark is recorded; under the retained parameter interval cell and its constraints, the weld width or center offset is determined based on the two side boundaries in the same combination, and the corresponding geometric value range covering the unexcluded combinations is obtained, without selecting mutually incompatible boundary extrema to construct the weld geometry.

6. The vehicle welding state recognition method based on multi-source vision fusion according to claim 5, characterized in that, When the image coverage of the reference and corresponding detection items is valid, and one type of normal candidate and abnormal candidate has a specific combination that meets all constraints, while the other type of candidate is completely excluded, the normal or abnormal state corresponding to the type with the specific combination is output. If the output conditions for normal or abnormal state are not met, the output "cannot be determined" is output, and the unexcluded candidate is retained. Where both types of candidates still coexist and the review deadline has not been reached, the review continues. If the candidate combination test is not yet completed, the candidate test continues, including: When the reference estimate is valid and the corresponding detection item's detection location has valid image coverage, if all abnormal candidates have been eliminated and there are normal candidates with feasible labels, the normal status of the detection item is output; if all normal candidates have been eliminated and there are abnormal candidates with feasible labels, the abnormal status of the detection item is output. If neither normal nor abnormal candidates are eliminated, the state of "undeterminable" is output externally, and the internal state of "pending review" is recorded before the review deadline is reached; if both types of candidates are eliminated, the reason for the inconsistency between the image constraints and the measured image is recorded and the state of "undeterminable" is output. If the aforementioned normal or abnormal state output conditions are not met, such as invalid reference estimation, insufficient effective image coverage, or the verification cutoff condition being met, the output state cannot be determined, and the corresponding undetermined position is recorded.

7. The vehicle welding status recognition method based on multi-source vision fusion as described in claim 4, characterized in that, When neither candidate class is excluded, predict their boundary range in the available verification image. Select the locations where the predicted ranges of at least one pair of dissimilar candidates can be separated after accounting for errors, and record the prediction and parameter versions, including: Candidate verification images are determined from images of other observation directions or images that can be acquired later at the same welding position and at the same welding stage. Based on the camera pose, visible area and weld geometry of each image during the verification period, the boundary position range of each normal candidate branch and each abnormal candidate branch is predicted, and the heterogeneous branches are paired. The imaging error, pose prediction error, and weld geometry change error are included in the boundary position range. For each pair, the observation position is selected where the interval between the two boundary position ranges exceeds the minimum resolvable interval determined by calibration, and both predicted boundaries of the pair are located within the visible area. When the range of weld geometry changes cannot be obtained, images from other observation directions at the same time, or cached images corresponding to the original observation time, are used. Unconstrained subsequent morphological changes are not used as the basis for excluding the original candidates.

8. The vehicle welding status recognition method based on multi-source vision fusion as described in claim 7, characterized in that, Also includes: At the start of this review selection, normal candidate branches and abnormal candidate branches are divided according to the range of values ​​of candidate weld geometry, imaging parameters and base material displacement, and the branch numbers are fixed. The sub-intervals after the interval subdivision retain the original branch numbers. Only when the prediction range of all sub-intervals of a pair of heterogeneous original branches can be separated in general, the pair is counted once. The observation position with the most separable pairs is selected. If the number is the same, the observation position with the shorter waiting time is selected first. Before reading the actual image content to be inspected, the candidate identifier, the predicted boundary position range, the parameter version and the observation time are recorded. After obtaining the actual image, first verify the imaging parameters, base material displacement and the applicability of the welding stage, the occlusion situation and the boundary detection conditions, and then compare the actual boundary with the recorded prediction. If the actual image makes all recorded candidates incompatible, the inconsistency record is retained, the parameter range is re-determined only based on the new reference data outside the protected area, the actual image is reprocessed with the new parameter version, the old version results are retained, and the predicted boundary is not used as the measured boundary input.

9. The vehicle welding status recognition method based on multi-source vision fusion as described in claim 6, characterized in that, Also includes: Establish status records based on workpiece identification, weld identification, welding pass, welding position and welding stage, and save the status records of the welding process and the status records of the post-weld formed surface separately. Record the position range of the current joint relative to the pre-weld reference based on the displacement range of the base material, and record the offset range of the weld center relative to the current joint and the offset range of the weld center relative to the pre-weld reference. When a new measured image of the same location is obtained, the candidate retention results of the corresponding stage are updated. The normal state of the subsequent stage does not cover the abnormal state that has been measured in the previous stage. When the review time limit is reached or the location to be inspected leaves the observable range, the review is terminated and the status, geometric value range, undetermined location and original image index are saved.

10. A multi-source vision fusion vehicle welding status recognition system, applied to the welding of electric tricycle frames, characterized in that, include: The image association module is used to receive multi-source images and observation conditions, associate images of the same welding position and stage, delineate the protection area containing the weld to be inspected, transfer the associated images and protection area to the reference estimation module and the candidate inspection module, and transfer the position, stage and available observation conditions to the verification selection module. The reference estimation module is used to determine the interrelated imaging parameters and the displacement range of the base material solely from the clamp reference outside the protected area and the base material reference. The range, correlation, reference validity, and parameter version are then transmitted to the candidate verification module, the review selection module, and the review update module. The reference validity is also transmitted to the status output module. The candidate inspection module receives the associated image, reference estimation result, and geometric allowable range of the inspection item. It projects the candidate weld geometry and the compatible parameter combination in the imaging parameters and base material displacement range of the reference estimation result onto the original image, so that the relevant boundaries in the same candidate combination share the corresponding imaging parameters and base material displacement parameters, retains the combination that meets the effective measured characteristics and error constraints, distinguishes between normal candidates and abnormal candidates, and transmits the candidates, specific feasible combinations, and effective image coverage to the verification selection module and the status output module. The review selection module is used to predict the boundary range of each candidate based on the received candidates, parameter range, stage and available observation conditions when neither type of candidate is excluded. It selects at least one pair of dissimilar candidates whose predicted ranges can be separated after accounting for errors, and transmits the recorded predictions, parameter versions and selected positions to the review update module. If no position is selected, the result is transmitted to the status output module. The verification and update module is used to obtain the actual image when the selected location exists. The image association module and the reference estimation module verify the validity of the external reference and the stage. After the verification is passed, the candidates are excluded based on the incompatibility between the actual boundary and the recorded prediction. The parameter range is not extended based on the weld to be inspected. The actual image constraint and updated candidates are sent back to the candidate inspection module. The reason for the verification failure is sent to the status output module. The status output module is used to receive reference validity, candidate test results and review results. When the reference and coverage are valid, there is a specific combination that meets all constraints and all other candidates are excluded, the corresponding normal or abnormal status is output. Otherwise, the output cannot be determined and the unexcluded candidates are retained. When the review deadline has not been reached, the review selection module is fed back the information to be reviewed.