A welding operation process monitoring method and system based on visual perception

Abstract

This invention discloses a visual perception-based welding operation monitoring method and system, belonging to the field of industrial production process monitoring technology. The method includes: acquiring image data containing key points of the human body of both the welding area and the operator; identifying the image data using a human body key point recognition model and a soldering iron tip recognition model to monitor human posture safety; cropping the portion of the human body requiring protection from the image data based on the position information of the human body key points, and identifying it using a corresponding protective gear recognition model to monitor protective gear wearing safety; acquiring the reference position of the solder pad and monitoring the welding contact time in conjunction with the position information of the soldering iron tip; and cropping a portion of the weld point from the image data after welding is completed based on the reference position of the solder pad, and identifying it using a weld point recognition model to monitor welding quality. This invention enables integrated automatic monitoring of welding operations, improving welding consistency and operational safety.

Description

Technical Field

[0001] This invention relates to the field of industrial production process monitoring technology, and in particular to a method and system for monitoring welding operation processes based on visual perception. Background Technology

[0002] As electronic products evolve towards higher density, miniaturization, and higher reliability, the requirements for operator standardization and solder joint consistency in circuit board soldering processes are constantly increasing. Especially in manual or semi-automatic soldering scenarios, soldering quality is not only related to equipment parameters but also highly dependent on whether the soldering personnel follow standard operating procedures, such as whether they wear safety goggles correctly, whether they complete the soldering within the specified time, and whether they control the amount of solder and the soldering angle.

[0003] Currently, post-production quality inspection is commonly used to ensure welding quality, such as AOI automatic optical inspection, manual sampling, or functional testing. These methods can only detect defects such as oversized solder joints, cold solder joints, and solder pad detachment after welding is completed. They cannot reconstruct whether any non-standard operations occurred during the welding process, so they cannot provide real-time corrections and are difficult to create traceable operation records.

[0004] Some production lines also rely on manual on-site supervision to check whether welding personnel are wearing protective equipment and whether they are in dangerous postures such as getting their eyes too close. However, this method depends on human experience and attention, has poor continuity and is highly subjective. It is difficult to cover every welding action and it is also difficult to form quantitative standards and objective evidence.

[0005] While existing welding equipment can limit parameters such as welding temperature and time, these limits primarily target the equipment itself and cannot identify whether human behavior is compliant. Examples include operating without safety goggles, leaving the soldering iron tip on the pad for too long causing it to detach due to heat, or continuing to weld despite poor solder joint formation. Current applications of computer vision in industrial settings often focus on the visual inspection of finished solder joints, rather than monitoring the welding process itself. Therefore, they can only answer whether the welding is good or bad, not whether the welding process was correct, safe, or exceeded the time limit.

[0006] Therefore, existing technologies mainly suffer from the following problems: First, they lack real-time monitoring of welding personnel's behavior, making it impossible to promptly identify safety risks such as not wearing safety goggles or having the eyes too close to the soldering iron tip; second, they lack timing monitoring of welding actions, failing to identify and warn of the duration of the soldering iron tip's contact with the solder pad online, thus preventing intervention before the solder pad is about to detach due to overheating; third, they lack a mechanism to link solder joint quality with operational behavior, making it impossible to trace quality problems such as uneven, rough, or excessively large solder joints back to non-standard operations, thus preventing targeted correction; fourth, they lack integrated hardware and software deployment solutions that can run locally on the production line in real time. Some solutions rely on cloud inference, resulting in high latency and strong network dependence, making them unsuitable for welding, a task requiring a second-level response.

[0007] Based on the above shortcomings, it is necessary to provide a system that can perform visual recognition at the moment of welding. By comprehensively analyzing personnel posture, protective clothing, welding time, and weld formation status, the system can determine in real time whether there are any violations and immediately provide prompts or corrections. At the same time, the monitoring results can be recorded for quality traceability and production management, thereby improving the safety of welding operations and the consistency of weld quality. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a visual perception-based welding operation process monitoring method and system. Without changing the existing welding station operation process, it can simultaneously include welding personnel, welding tools, and weld area in the visual field of view. Then, it identifies the wearing status of protective equipment, personnel posture safety, welding action duration, and weld formation quality in a task-based manner, forming a complete judgment of a welding process, thereby improving the safety of welding operations and the consistency of weld quality.

[0009] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0010] In a first aspect, the present invention provides a method for monitoring welding operation processes based on visual perception, comprising:

[0011] Acquire image data that simultaneously includes key human body points of both the welding area and the worker;

[0012] The image data is identified using a human body key point recognition model and a soldering iron tip recognition model to obtain the position information of human body key points and soldering iron tip.

[0013] Human posture safety monitoring is performed based on the location information of the key points on the human body and the soldering iron tip.

[0014] Based on the location information of the key points of the human body, the local image of the human body that needs to be protected is cropped from the image data, and the local image of the human body is identified by the corresponding protective gear recognition model to monitor the safety of wearing protective gear.

[0015] Obtain the reference position of the solder pad and monitor the soldering contact time in conjunction with the position information of the soldering iron tip;

[0016] Based on the reference position of the solder pad, a local image of the solder joint is cropped from the image data after the welding is completed. The local image of the solder joint is then identified by a solder joint recognition model to monitor the welding quality.

[0017] Optionally, the human body key point recognition model is built based on a convolutional neural network and trained using heatmap regression. A two-dimensional Gaussian heatmap is output for each human body key point, and model training is completed by minimizing the mean square error between the predicted Gaussian heatmap and the labeled heatmap.

[0018] Optionally, the soldering tip recognition model is constructed based on a lightweight target detection network, and the coordinates of the center point of the outer rectangle of the soldering tip output by the network are taken as the soldering tip coordinates.

[0019] Optionally, the human posture safety monitoring based on the position information of the key points on the human body and the soldering iron tip includes:

[0020] The key points of the human body are divided into sensitive key points and reference key points. The sensitive key points are the key points of the human body that need to be kept at a safe distance from the soldering iron tip, and the reference key points are the key points of the human body used to calculate the reference scale of the human body.

[0021] Calculate the normalized distance from the sensitive key point to the soldering tip. :

[0022]

[0023]

[0024] In the formula, For the coordinates of sensitive key points, For the soldering iron tip coordinates, As a normalized reference value, These are two different reference key points. For time frames;

[0025] If normalized distance Less than the corresponding safety threshold Then determine the time frame. The human posture is unsafe.

[0026] Optionally, the protective gear recognition model is constructed based on a lightweight convolutional neural network for binary classification, and its output is whether the corresponding protective gear is worn.

[0027] Optionally, the reference position of the solder pad is obtained, and the soldering contact time is monitored in combination with the position information of the soldering iron tip;

[0028] Calculate the intersection-union ratio (IU / R) of the circumscribed rectangle of the soldering iron tip and the rectangular region of the solder pad. :

[0029]

[0030] In the formula, The outer rectangle of the soldering iron tip. For time frames, This refers to the rectangular area of ​​the solder pads.

[0031] When the intersection and union ratio Greater than the set crossover ratio threshold It assumes that the soldering iron tip is in contact with the solder pad, and sets a binary contact indicator. :

[0032]

[0033] The binary contact indication during the welding time The duration of contact is accumulated to obtain the total contact time. :

[0034] In the formula, These are the time frames for the start and end of the welding process. For video frame rate;

[0035] If the contact duration The dwell time exceeds the maximum allowable dwell time for the corresponding welding process. If so, it is determined that the soldering iron tip has been in contact with the solder pad for too long.

[0036] Optionally, the solder joint recognition model is based on a lightweight solder joint segmentation network, which separates the solder joint region and background pad on the local image of the solder joint to obtain a binary mask image, and uses Dice loss or cross-entropy loss for training to improve the segmentation accuracy of the solder joint edge.

[0037] The area of ​​the solder joint is calculated based on the binary mask image. and perimeter :

[0038]

[0039]

[0040] In the formula, Binary mask image mid-coordinate point The binary value corresponding to the pixel at that location. ; Binary mask image At coordinate point The gradient magnitude of the pixel at that location;

[0041] Based on the area of ​​the solder joint and perimeter Calculate the roundness index of the weld joint :

[0042]

[0043] If roundness index Less than the roundness threshold If the weld joint is not round, the welding quality is deemed unqualified.

[0044] If area If the weld point is not within the standard area range, it is either too large or too small, and the welding quality is deemed unqualified.

[0045] Optionally, the welding quality monitoring also includes:

[0046] The local image of the solder joint is converted into a grayscale image and then smoothed by Gaussian filtering;

[0047] The Sobel operator is used to calculate the horizontal and vertical gradients of the smoothed image. And construct a gradient magnitude map :

[0048]

[0049] In the formula, Gradient magnitude diagrams mid-coordinate point The horizontal and vertical gradients of the pixel;

[0050] Based on the binary mask image of the solder joint Determine the set of pixels in the solder joint area ,gather The number of pixels is ;

[0051] Calculate the average gradient magnitude within the solder joint area. and gradient magnitude variance :

[0052]

[0053]

[0054] In the formula, Gradient magnitude plot mid-coordinate point The gradient magnitude of the pixel at that location;

[0055] If the gradient magnitude variance Greater than the threshold If the surface of the weld joint is not smooth or rough, the welding quality is deemed unqualified.

[0056] Optional, also includes:

[0057] The results of human posture safety monitoring, protective gear wearing safety monitoring, welding contact time monitoring, and welding quality monitoring will be used as indicators of violations.

[0058] Based on the aforementioned violation indicators, a violation severity value is obtained using a neural network or a weighted summation method.

[0059] When the violation severity value exceeds the severity threshold, a warning alert will be triggered.

[0060] Secondly, the present invention provides a welding operation process monitoring system based on visual perception, comprising:

[0061] The human posture safety monitoring module is configured to acquire image data of key human body points that simultaneously include the welding area and the operator; identify the image data using a human key body point recognition model and a soldering iron tip recognition model to obtain the position information of the human key body points and the soldering iron tip; and perform human posture safety monitoring based on the position information of the human key body points and the soldering iron tip.

[0062] The protective gear wearing safety monitoring module is configured to crop out the partial image of the human body to be protected from the image data based on the location information of the key points of the human body, and to identify the partial image of the human body through a corresponding protective gear recognition model to perform protective gear wearing safety monitoring.

[0063] The soldering contact time monitoring module is configured to acquire the reference position of the solder pad and monitor the soldering contact time in conjunction with the position information of the soldering iron tip.

[0064] The welding quality monitoring module is configured to crop a local image of the weld point from the image data after welding is completed based on the reference position of the weld pad, and to identify the local image of the weld point through a weld point recognition model to monitor the welding quality.

[0065] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0066] This invention provides a visual perception-based welding operation process monitoring method and system. It can be completed using a standard RGB camera, without relying on special light sources or markers. All thresholds can be obtained from on-site data statistics, all models can be trained from labeled data, and all calculation formulas are based on image coordinates or image cropping regions, requiring no additional external parameter calibration. Therefore, it has strong engineering feasibility and reproducibility. When implementing this invention, enterprises only need to collect the original video as described above, label key points, soldering iron tip frame, contact time period, solder joint mask, and wearing status according to modules, and then complete the training and deployment using common deep learning processes. This achieves integrated automatic monitoring of safety protection, operating procedures, and solder joint quality, reducing human error and improving welding consistency and operational safety. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the welding operation process monitoring method based on visual perception provided in an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of the hardware deployment of a camera installed at an oblique angle above the welding station, provided in an embodiment of the present invention.

[0069] Figure 3 This is a schematic diagram of the welding contact timing detection and timeout determination process provided in an embodiment of the present invention;

[0070] Figure 4 This is a schematic diagram of a multi-task neural network structure provided in an embodiment of the present invention. Detailed Implementation

[0071] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0072] Example 1

[0073] like Figure 1 As shown, this embodiment of the invention provides a method for monitoring welding operation processes based on visual perception, including:

[0074] Step S1: Acquire image data that simultaneously includes key points of the human body of the welding area and the worker.

[0075] like Figure 2 As shown, in this specific embodiment, a side-above angled camera is set up at the welding station to collect video data that simultaneously includes the welding area and key points of the worker's body, and then decomposes it into image data in frames.

[0076] The advantage of the side-above view is that it allows simultaneous viewing of the welder's head, upper limbs, and the welding station surface, avoiding the situation where the weld iron obscures the weld joint in a direct-above view. It is also superior to the drawback of a pure side view, which easily obscures the welder's face. All subsequent recognition algorithms use this frame of image or its cropped region as input, maintaining the same coordinate system and the same timestamp, which facilitates multi-task fusion.

[0077] Key points on the human body, such as the top of the head, eyes, tip of the nose, neck, shoulders, elbows, and wrists, are points distributed on the head and upper limbs. In other optional implementations, key points on the human body can be set according to actual needs.

[0078] Step S2: The image data is identified using the human body key point recognition model and the soldering iron tip recognition model to obtain the position information of the human body key points and the soldering iron tip.

[0079] The human keypoint recognition model is built upon a convolutional neural network and trained using heatmap regression. A two-dimensional Gaussian heatmap is output for each human keypoint, and model training is completed by minimizing the mean squared error between the predicted and labeled Gaussian heatmaps. The advantage of this approach is its low annotation cost; initial annotation results can be automatically generated using readily available open-source pose annotation tools, followed by minor manual corrections, ensuring that the algorithm described in this invention can be reproduced within an enterprise.

[0080] The soldering tip recognition model is built on a lightweight target detection network, and the coordinates of the center point of the outer rectangle of the output soldering tip are taken as the soldering tip coordinates.

[0081] Step S3: Perform human posture safety monitoring based on the position information of key human body points and the soldering iron tip. Specifically, this includes:

[0082] Step S3.1: Divide the human body key points into sensitive key points and reference key points. Sensitive key points are those that need to be kept at a safe distance from the soldering iron tip, while reference key points are those used to calculate human body reference dimensions. Sensitive key points include the eyes, and reference key points include the top of the head and the shoulders.

[0083] Step S3.2: Calculate the normalized distance from the sensitive key point to the soldering tip. :

[0084]

[0085]

[0086] In the formula, For the coordinates of sensitive key points, For the soldering iron tip coordinates, As a normalized reference value, These are two different reference key points. For time frames.

[0087] Because different staff members are at different distances from the camera, directly using pixel distance for judgment will lead to inconsistent thresholds. Therefore, this invention adopts a normalized distance method, dividing the Euclidean distance from the sensitive key point to the soldering iron tip by the reference scale of the human body in the same frame. The reference scale can be the distance from the top of the head to the shoulders, or the distance from the top of the head to the buttocks, etc.

[0088] Step S3.3, if the normalized distance Less than the corresponding safety threshold Then determine the time frame. The human posture is unsafe.

[0089] If normalized distance Less than the corresponding safety threshold If your eyes get too close to the soldering iron tip, which is at a high temperature during operation, it is considered an unsafe posture.

[0090] Safety threshold This can be obtained by collecting several video clips of actual workers on site under safe conditions and performing statistical analysis. For example, the 5th percentile of the distance distribution of all safe samples can be used as the threshold. This ensures that the algorithm is reproducible and consistent with the actual workstation.

[0091] Step S4: Based on the location information of key points on the human body, crop out the local image of the human body that needs protection from the image data, and use the corresponding protective gear recognition model to identify the local image of the human body and monitor the safety of wearing protective gear.

[0092] Another fundamental requirement in welding operation specifications is the mandatory wearing of protective gear, such as goggles, to prevent eye damage from high-temperature spatter or bright light generated during welding. To address this, this invention adds a protective gear recognition model to the existing human body key point recognition system. Since the camera is already capable of locating the head area, it is unnecessary to perform large-scale detection on the entire image; instead, local cropping is performed based on key points of the head or eyes.

[0093] The protective gear recognition model is built on a lightweight convolutional neural network for binary classification, and its output indicates whether protective gear is being worn. Training only requires labeling each image with whether protective gear is worn, and training can be completed using cross-entropy loss. Since only "wearing / not wearing" needs to be labeled, and precise delineation of the protective gear outline is not required, the labeling cost is very low, making it suitable for rapid deployment when building datasets in factories.

[0094] Step S5: Obtain the reference position of the solder pad and monitor the soldering contact time in conjunction with the position information of the soldering iron tip.

[0095] Solder pad detachment is often not an instantaneous problem that is difficult to detect with the naked eye, but rather caused by localized overheating due to the soldering tip remaining on the solder pad for too long. Therefore, a single frame image is insufficient to identify this violation. This invention introduces soldering contact time monitoring. The position of the solder pads at the soldering station is usually fixed within a work cycle. During the deployment phase, the system can take a reference image of the station, and the rectangular area of ​​the solder pad is marked manually or semi-automatically using template matching, denoted as... In actual operation, the rectangular frame of the soldering iron tip is detected in each frame. ,like Figure 3 As shown, this invention determines whether a frame belongs to the "contacting pad" state by calculating the intersection-union ratio of the two rectangular frames:

[0096] Calculate the intersection-union ratio (IU / R) of the circumscribed rectangle of the soldering iron tip and the rectangular region of the solder pad. :

[0097]

[0098] In the formula, The outer rectangle of the soldering iron tip. For time frames, This refers to the rectangular area of ​​the solder pads.

[0099] When the intersection and union ratio Greater than the set crossover ratio threshold It assumes that the soldering iron tip is in contact with the solder pad, and sets a binary contact indicator. :

[0100]

[0101] Binary contact indication during welding time The duration of contact is accumulated to obtain the total contact time. :

[0102] In the formula, These are the time frames for the start and end of the welding process. For video frame rate;

[0103] If the contact duration The dwell time exceeds the maximum allowable dwell time for the corresponding welding process. If so, it is determined that the soldering iron tip has been in contact with the solder pad for too long.

[0104] The advantage of this approach is that during training, only the start and end times of contact need to be labeled in a video segment to automatically generate a sequence of 0 and 1 labels. It does not require detailed labeling frame by frame, so the labeling workload is small. Furthermore, the formula is simple and clear, and can be fully reproduced in the backend script.

[0105] Step S6: Based on the reference position of the solder pad, crop out the local image of the solder joint from the image data after the welding is completed, and use the solder joint recognition model to identify the local image of the solder joint to monitor the welding quality.

[0106] The final quality of the welding process still depends on the formation of the weld joint; therefore, this invention also monitors the welding quality based on the weld joint itself. After a welding action is detected, the model acquires a local image of the weld joint. To extract the true boundary of the weld joint, the weld joint recognition model is based on a lightweight weld joint segmentation network. It separates the weld joint region and the background pad in the local image to obtain a binary mask image, and uses Dice loss or cross-entropy loss for training to improve the segmentation accuracy of the weld joint edges.

[0107] Calculate the area of ​​solder joints based on binary mask images. and perimeter :

[0108]

[0109]

[0110] In the formula, Binary mask image mid-coordinate point The binary value corresponding to the pixel at that location. ; Binary mask image At coordinate point The gradient magnitude of the pixel at that location;

[0111] Based on the area of ​​the solder joint and perimeter Calculate the roundness index of the weld joint :

[0112]

[0113] If roundness index Less than the roundness threshold If the solder joint is not round, the soldering quality is considered unqualified. When the solder joint is heated and the amount of solder is appropriate, the solder joint should be close to a spherical crown shape, with a roundness index of [missing information]. It will fall within an empirical range; when there is too much solder or uneven heating, the edge of the solder joint will elongate or expand irregularly, resulting in an increase in circumference and a decrease in roundness. Therefore, a roundness threshold can be set. ,when If the solder joint is not round, it needs to be reworked.

[0114] If area If the weld point is not within the standard area range, it is either too large or too small, and the welding quality is deemed unqualified.

[0115] Furthermore, welding quality monitoring also includes:

[0116] The local image of the solder joint is converted into a grayscale image and then smoothed by Gaussian filtering.

[0117] The Sobel operator is used to calculate the horizontal and vertical gradients of a smooth image. And construct a gradient magnitude map :

[0118]

[0119] In the formula, Gradient magnitude diagrams mid-coordinate point The horizontal and vertical gradients of the pixel;

[0120] Based on the binary mask image of the solder joint Determine the set of pixels in the solder joint area ,gather The number of pixels is ;

[0121] Calculate the mean gradient magnitude within the solder joint area. and gradient magnitude variance :

[0122]

[0123]

[0124] In the formula, Gradient magnitude plot mid-coordinate point The gradient magnitude of the pixel at that location;

[0125] If the gradient magnitude variance Greater than the threshold If the surface of the weld joint is not smooth or rough, the welding quality is deemed unqualified.

[0126] All of the above judgments are based on computable geometric and texture features, do not rely on black-box scoring functions, and facilitate the explanation of the algorithm's feasibility during review.

[0127] Step S7: Use the results of human posture safety monitoring, protective gear wearing safety monitoring, welding contact time monitoring, and welding quality monitoring as violation indicators; based on the violation indicators, use a neural network or weighted summation method to obtain the violation severity value; when the violation severity value is greater than the severity threshold, trigger a warning alarm.

[0128] To ensure that this invention can run in real time on edge devices, each of the above sub-models can be exported to ONNX format after training, and then converted into a high-efficiency inference engine file using the edge box's built-in inference engine. Since pose recognition, object detection, binary classification, and segmentation networks are all independent modules, they can be selectively enabled based on actual computing power. For example, solder joint segmentation can be disabled when only safety is a concern, and the pose recognition frame rate can be reduced when only solder joint quality is a concern, thus enabling the reproduction of the invention's functionality under different hardware resources.

[0129] In summary, the visual perception-based welding operation monitoring method provided in this invention collects welding station videos, uses a human key point recognition model to obtain key points such as the welder's eyes, and calculates the normalized distance between sensitive key points and the soldering tip by combining the soldering tip detection results, thereby recognizing dangerous approach behaviors; it uses a protective gear recognition model to detect the wearing status of protective gear such as goggles; it constructs the contact sequence based on the cross-union ratio of the soldering tip and the solder pad area, calculates the contact duration, and identifies excessively long welding stops; it segments the local image of the solder joint and calculates features such as area, perimeter, and roundness to determine if the solder joint is too large, not round, not smooth, or at risk of solder pad detachment. The system integrates the above detection results, outputs real-time audio-visual or interface prompts, and records the entire process data for quality traceability. It can be deployed on existing welding stations to achieve integrated automatic monitoring of safety protection, operating procedures, and solder joint quality, reducing human error and improving welding consistency and operational safety.

[0130] Example 2

[0131] In Example 1, for human posture safety monitoring, protective gear wearing safety monitoring, welding contact time monitoring, and welding quality monitoring, human key point recognition models, soldering iron tip recognition models, protective gear recognition models, and solder joint recognition models were respectively set up to complete different recognition tasks. In this embodiment, as... Figure 4 As shown, a multi-task neural network structure is used to achieve unified visual perception. This neural network takes video image frames collected at the welding station as input, and outputs key human body points, protective gear wearing probability, soldering iron tip detection frame and solder joint segmentation mask through a shared feature extraction backbone network and multiple task branches, thereby providing basic data for subsequent safe distance calculation, welding timing analysis and solder joint quality assessment.

[0132] Input image frames are captured by a camera and, after being resized, normalized, and cropped as necessary by a preprocessing module, are fed into a shared backbone network. The shared backbone network employs lightweight convolutional neural networks, residual networks, or other feature extraction structures suitable for edge deployment to extract hierarchical visual features from the image. The shared backbone network includes an initial feature extraction section and multiple staged feature extraction sections, progressively forming multi-layered feature representations. Shallow features have high spatial resolution, suitable for preserving boundary and texture details, while deep features have strong semantic expressive power, suitable for supporting tasks such as object detection, pose estimation, and region classification.

[0133] To achieve multi-scale perception, a feature pyramid network is incorporated into the neural network. This network receives deep features (C3, C4, C5) from the backbone network output and generates multi-scale feature maps (P3, P4, P5, and ROI) through lateral connections, top-down upsampling, and feature fusion operations. The stride represents the feature map scale. High-resolution feature maps are suitable for fine-grained localization tasks, medium-resolution feature maps balance spatial resolution and semantic information, and low-resolution feature maps are suitable for robust discrimination in small object detection. In this implementation, region feature extraction is also performed on the feature maps based on head region boxes or other prior regions to generate region features for protective gear wearing classification.

[0134] The pose estimation branch receives a high-resolution feature map (P3) as input. This branch first processes the input features through a convolutional feature extraction submodule to obtain local and global representations suitable for keypoint prediction, and then outputs a multi-channel keypoint heatmap through upsampling or deconvolution structures. , Number of key points For time frames, This is a heatmap. The key point heatmap corresponds to the locations of multiple key points on the human body, such as the top of the head, eyes, and shoulders. By taking the position with the maximum response from the heatmap, the coordinates of the key points on the head, eyes, and shoulders of the welding personnel can be obtained, which are then used for subsequent normalized eye-soldering tip distance calculations.

[0135] The protective gear wearing classification branch is used to determine whether welders are wearing protective gear. This branch takes as input the region-of-interest (ROI) features extracted from the head area on the feature map. After the input features are processed by the feature extraction and classification submodules, the output is the probability of wearing protective gear. and the probability of not wearing protective gear The output can be directly used to determine the protective gear wearing status and serves as one of the inputs for subsequent violation fusion determination.

[0136] The soldering tip detection branch receives feature maps (P3+P4+P5) at multiple scales as input. This branch performs convolutional transformations on the input features at different scales to enhance the detectability of the small soldering tip target. Subsequently, it outputs the rectangular bounding box position (x, y, w, h) of the soldering tip in the current image through target classification and bounding box regression, where x and y are the base coordinates, and w and h are the width and height. This detection result can be used to calculate the normalized eye-to-soldering tip distance and can also be spatially correlated with the solder pad region to construct the soldering contact time sequence.

[0137] The solder joint segmentation branch primarily receives a high-resolution feature map (P3) as input. This branch restores the spatial resolution through decoding and progressive upsampling operations, outputting a probability map of the solder joint region. After thresholding, a binary mask of the solder joint region is obtained. The solder joint mask can be used to calculate features such as solder joint area, perimeter, roundness, and surface texture, thereby enabling the evaluation of solder joint forming quality.

[0138] During the training phase, a multi-task loss module can be configured to calculate the corresponding losses for the outputs of the pose estimation branch, protective gear classification branch, soldering iron tip detection branch, and solder joint segmentation branch, and to jointly optimize the shared backbone network and each task branch. The pose estimation branch uses keypoint heatmap regression loss, the protective gear classification branch uses cross-entropy loss, the soldering iron tip detection branch uses target classification loss and bounding box regression loss, and the solder joint segmentation branch uses a combination of binary cross-entropy loss and Dice loss. Through this joint training method, the shared feature extraction part can simultaneously serve multiple welding operation monitoring tasks, thereby reducing the overall model parameter count and improving edge deployment efficiency while ensuring recognition performance.

[0139] In the actual inference phase, after the input image passes through the neural network, it can simultaneously obtain key point output, protective wearing judgment output, soldering iron tip frame output, and solder joint mask output. The system further calculates the normalized eye-to-soldering iron tip distance based on the key point output and soldering iron tip frame output, calculates the welding contact sequence based on the soldering iron tip frame output and the solder pad area, extracts the solder joint morphology and texture features based on the solder joint mask output, and finally completes the identification and correction prompts for welding operation violations through the fusion judgment module.

[0140] Example 3

[0141] This invention provides a vision-based welding operation process monitoring system, comprising:

[0142] The human posture safety monitoring module is configured to acquire image data of key human body points that simultaneously include the welding area and the operator; to identify the image data using a human key point recognition model and a soldering iron tip recognition model to obtain the position information of the human key points and the soldering iron tip; and to perform human posture safety monitoring based on the position information of the human key points and the soldering iron tip.

[0143] The protective gear wearing safety monitoring module is configured to crop out the local human body image that needs protection from the image data based on the location information of key points on the human body, and then identify the local human body image through the corresponding protective gear recognition model to monitor the safety of wearing protective gear.

[0144] The soldering contact time monitoring module is configured to acquire the reference position of the solder pad and monitor the soldering contact time in conjunction with the position information of the soldering iron tip.

[0145] The welding quality monitoring module is configured to crop a local image of the weld point from the image data after welding is completed based on the reference position of the weld pad, and then identify the local image of the weld point through a weld point recognition model to monitor the welding quality.

[0146] Building upon the aforementioned monitoring system, various prompting devices can be further implemented. For example, when a welder is detected not wearing safety goggles, an icon can pop up on the display screen accompanied by a short audible alert; when the eyes are detected to be too close to the soldering iron tip, different audible alerts or more prominent colored graphics can be used to remind the operator to adjust their posture; when the welding time is detected to be too long and there is a risk of solder pad detachment, a prolonged audible and visual alarm can be issued, and the welding record can be marked as a key focus in the quality inspection system; when the solder joint quality inspection results indicate that the solder joint is unqualified, the system can require the operator to immediately rework the solder joint and record the rework information. The data recording and quality inspection system can utilize a local database or connect to the factory's existing manufacturing execution system, linking and storing the time of each weld, operator number, workstation number, inspection indicators, and violations for subsequent traceability and analysis of quality issues.

[0147] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for monitoring welding operation processes based on visual perception, characterized in that, include: Acquire image data that simultaneously includes key human body points of both the welding area and the worker; The image data is identified using a human body key point recognition model and a soldering iron tip recognition model to obtain the position information of human body key points and soldering iron tip. Human posture safety monitoring is performed based on the location information of the key points on the human body and the soldering iron tip. Based on the location information of the key points of the human body, the local image of the human body that needs to be protected is cropped from the image data, and the local image of the human body is identified by the corresponding protective gear recognition model to monitor the safety of wearing protective gear. Obtain the reference position of the solder pad and monitor the soldering contact time in conjunction with the position information of the soldering iron tip; Based on the reference position of the solder pad, a local image of the solder joint is cropped from the image data after the welding is completed. The local image of the solder joint is then identified by a solder joint recognition model to monitor the welding quality.

2. The method for monitoring welding operation process based on visual perception according to claim 1, characterized in that, The human body key point recognition model is built on a convolutional neural network and trained using heatmap regression. A two-dimensional Gaussian heatmap is output for each human body key point, and the model training is completed by minimizing the mean square error between the predicted Gaussian heatmap and the labeled heatmap.

3. The method for monitoring welding operation process based on visual perception according to claim 1, characterized in that, The soldering iron tip recognition model is constructed based on a lightweight target detection network, and the coordinates of the center point of the outer rectangle of the output soldering iron tip are taken as the soldering iron tip coordinates.

4. The method for monitoring welding operation process based on visual perception according to claim 1, characterized in that, The human posture safety monitoring based on the position information of the key points on the human body and the soldering iron tip includes: The key points of the human body are divided into sensitive key points and reference key points. The sensitive key points are the key points of the human body that need to be kept at a safe distance from the soldering iron tip, and the reference key points are the key points of the human body used to calculate the reference scale of the human body. Calculate the normalized distance from the sensitive key point to the soldering tip. : In the formula, For the coordinates of sensitive key points, For the soldering iron tip coordinates, As a normalized reference value, These are two different reference key points. For time frames; If normalized distance Less than the corresponding safety threshold Then determine the time frame. The human posture is unsafe.

5. The method for monitoring welding operation process based on visual perception according to claim 1, characterized in that, The protective gear recognition model is built on a lightweight convolutional neural network for binary classification, and its output is whether the corresponding protective gear is being worn.

6. The method for monitoring welding operation process based on visual perception according to claim 1, characterized in that, The reference position of the solder pad is obtained, and the soldering contact time is monitored in combination with the position information of the soldering iron tip; Calculate the intersection-union ratio (IU / R) of the circumscribed rectangle of the soldering iron tip and the rectangular region of the solder pad. : In the formula, The outer rectangle of the soldering iron tip. For time frames, This refers to the rectangular area of ​​the solder pads. When the intersection and union ratio Greater than the set crossover ratio threshold It assumes that the soldering iron tip is in contact with the solder pad, and sets a binary contact indicator. : The binary contact indication during the welding time The duration of contact is accumulated to obtain the total contact time. : In the formula, These are the time frames for the start and end of the welding process. For video frame rate; If the contact duration The dwell time exceeds the maximum allowable dwell time for the corresponding welding process. If so, it is determined that the soldering iron tip has been in contact with the solder pad for too long.

7. The method for monitoring welding operation process based on visual perception according to claim 1, characterized in that, The solder joint recognition model is based on a lightweight solder joint segmentation network. It separates the solder joint region and the background pad on the local image of the solder joint to obtain a binary mask image. Dice loss or cross-entropy loss is used for training to improve the segmentation accuracy of the solder joint edge. The area of ​​the solder joint is calculated based on the binary mask image. and perimeter : In the formula, Binary mask image mid-coordinate point The binary value corresponding to the pixel at that location. ; Binary mask image At coordinate point The gradient magnitude of the pixel at that location; Based on the area of ​​the solder joint and perimeter Calculate the roundness index of the weld joint : If roundness index Less than the roundness threshold If the weld joint is not round, the welding quality is deemed unqualified. If area If the weld point is not within the standard area range, it is either too large or too small, and the welding quality is deemed unqualified.

8. The method for monitoring welding operation process based on visual perception according to claim 7, characterized in that, The welding quality monitoring also includes: The local image of the solder joint is converted into a grayscale image and then smoothed by Gaussian filtering. The Sobel operator is used to calculate the horizontal and vertical gradients of the smoothed image. And construct a gradient magnitude map : In the formula, Gradient magnitude diagrams mid-coordinate point The horizontal and vertical gradients of the pixel; Based on the binary mask image of the solder joint Determine the set of pixels in the solder joint area ,gather The number of pixels is ; Calculate the mean gradient magnitude within the solder joint area. and gradient magnitude variance : In the formula, Gradient magnitude plot mid-coordinate point The gradient magnitude of the pixel at that location; If the gradient magnitude variance Greater than the threshold If the surface of the weld joint is not smooth or rough, the welding quality is deemed unqualified.

9. The method for monitoring welding operation process based on visual perception according to claim 1, characterized in that, Also includes: The results of human posture safety monitoring, protective gear wearing safety monitoring, welding contact time monitoring, and welding quality monitoring will be used as indicators of violations. Based on the aforementioned violation indicators, a violation severity value is obtained using a neural network or a weighted summation method. When the violation severity value exceeds the severity threshold, a warning alert will be triggered.

10. A welding operation process monitoring system based on visual perception, characterized in that, include: The human posture safety monitoring module is configured to acquire image data of key human body points that simultaneously include the welding area and the worker; The image data is identified using a human body key point recognition model and a soldering iron tip recognition model to obtain the position information of human body key points and soldering iron tip. Human posture safety monitoring is performed based on the location information of the key points on the human body and the soldering iron tip. The protective gear wearing safety monitoring module is configured to crop out the partial image of the human body to be protected from the image data based on the location information of the key points of the human body, and to identify the partial image of the human body through a corresponding protective gear recognition model to perform protective gear wearing safety monitoring. The soldering contact time monitoring module is configured to acquire the reference position of the solder pad and monitor the soldering contact time in conjunction with the position information of the soldering iron tip. The welding quality monitoring module is configured to crop a local image of the weld point from the image data after welding is completed based on the reference position of the weld pad, and to identify the local image of the weld point through a weld point recognition model to monitor the welding quality.

Images