Welding seam grinding and polishing robot visual servo control method, system, device and medium

Abstract

The application provides a welding seam grinding and polishing robot visual servo control method, system, device and medium, collects multiple exposure visible light images of the same welding seam section, synchronously acquires a depth image and structure light data, generates a brightness balanced image through cross-exposure registration and radiation response compensation; fuses the brightness balanced image with the depth / structure light data to extract a welding seam area, a center line and a three-dimensional pose and output a confidence index; adaptively constructs a hybrid error state based on the confidence, inputs a pre-trained strategy model to output a joint speed instruction; the instruction is projected to a feasible set satisfying joint limit, speed upper limit, collision distance threshold and field of view constraint after being limited by a safety constraint layer, and is executed, and is degraded or stopped when the confidence is insufficient or the risk is increased. The application improves the recognition stability, trajectory tracking accuracy and operation safety under complex light and interference.

Description

Technical Field

[0001] This application belongs to the field of intelligent processing and machine vision technology of industrial robots, and particularly relates to a visual servo control method, system, equipment and medium for weld grinding and polishing robot. Background Technology

[0002] Weld polishing is a crucial post-processing step in the manufacturing of welded structural components. Its purpose is to improve the geometry and surface quality of the weld, enhancing appearance consistency and subsequent coating performance. Traditional manual polishing is labor-intensive, operates in harsh environments, and its quality consistency is significantly affected by the operator's experience. Therefore, replacing manual labor with robots for weld polishing has become an important development direction in the field of intelligent manufacturing.

[0003] In actual weld grinding and polishing operations, specular reflection of the metal surface, changes in lighting conditions, and interference from dust and sparks can easily cause saturation in bright areas or loss of detail in dark areas in single-exposure imaging, thus reducing the stability of extracting key features such as weld centerline, boundary, and weld toe. On the other hand, multi-exposure images and multimodal data such as depth and structured light exhibit geometric inconsistencies and differences in radiometric response during acquisition and alignment. Without cross-exposure registration and radiometric response compensation processing, fusion ghosting, brightness distortion, and positioning drift can easily occur.

[0004] In terms of control, existing visual servoing methods typically rely on model and parameter tuning, making it difficult to simultaneously meet the requirements of trajectory matching accuracy, posture stability, and motion smoothness under conditions of contact disturbance and weld morphology changes. Furthermore, they lack a unified constraint mechanism for engineering safety factors such as joint limits, velocity constraints, collision distance, and field-of-view constraints. Existing multi-exposure fusion methods primarily focus on image enhancement, lacking adaptation to dust interference in grinding and polishing scenarios, and their safety constraints do not clearly define collision distance thresholds, making it difficult to meet the stability and safety requirements of weld grinding and polishing contact operations. Summary of the Invention

[0005] In view of this, this application aims to provide a visual servo control method, system, device and medium for a weld grinding and polishing robot to solve at least one of the above problems.

[0006] To achieve the above objectives, the technical solution of this application is implemented as follows: In a first aspect, this application provides a visual servo control method for a weld grinding and polishing robot, comprising: Multiple visible light weld images with different exposure times are acquired on the same weld segment according to the exposure sequence adapted to the weld material, and at least one of depth image and structured light image is acquired simultaneously to obtain multi-exposure multi-modal weld data. Using the selected reference exposure image as a benchmark, feature matching and geometric transformation are performed on the remaining exposure images to align them to the same coordinate system. Radiometric response compensation is performed on the aligned multi-exposure images based on the exposure time and camera radiometric response function. Weights are assigned to each exposure image according to pixel saturation and brightness reliability, and the images are fused to obtain a bright and uniform visible light image. The brightness-equalized visible light image and the acquired depth image and / or structured light image are input into a dual-branch feature coding network to perform feature coding and attention fusion on data from different sources to obtain fused features. Based on the fused features, the weld visual features and confidence index are output. The visual servo control for weld grinding and polishing is modeled as a Markov decision process, and a hybrid visual servo strategy model is obtained through deep reinforcement learning training. The state includes at least weld feature error, robot joint state, and contact information; the action includes joint velocity vectors; the reward function includes trajectory fitting, posture stability, smoothing, safety penalty, and grinding and polishing efficiency factors; the weld feature error is obtained by adaptively selecting or weighting pixel error and 3D pose error according to the confidence index. The system collects multi-exposure, multi-modal weld data in real time during the grinding and polishing process. The extracted weld visual features and robot state are combined to form a strategy model input. The hybrid vision servo strategy model outputs control commands to drive the robot to perform weld grinding and polishing.

[0007] Secondly, based on the same inventive concept, this application also provides a visual servo control system for a weld seam grinding and polishing robot, comprising: The image acquisition module is configured to acquire multiple visible light weld images with different exposure times on the same weld segment according to the exposure sequence adapted to the weld material, and simultaneously acquire at least one of depth images and structured light images to obtain multi-exposure multimodal weld data. The image processing module is configured to use the selected reference exposure image as a reference, perform feature matching and estimate geometric transformation on the remaining exposure images to align them to the same coordinate system, perform radiometric response compensation on the aligned multi-exposure images based on the exposure time and camera radiometric response function, and assign weights to each exposure image according to pixel saturation and brightness reliability, and fuse them to obtain a bright and uniform visible light image. The feature fusion module is configured to input the brightness-equalized visible light image and the acquired depth image and / or structured light image into a dual-branch feature coding network to perform feature coding and attention fusion on data from different sources to obtain fused features, and output weld visual features and confidence index based on the fused features. The model training module is configured to model the visual servo control of weld grinding and polishing as a Markov decision process, and obtain a hybrid visual servo strategy model through deep reinforcement learning. The state includes at least weld feature error, robot joint state, and contact information; the action includes joint velocity vectors; the reward function includes trajectory fitting, posture stability, smoothing, safety penalty, and grinding and polishing efficiency factors; and the weld feature error is obtained by adaptively selecting or weighting pixel error and 3D pose error according to the confidence index. The instruction output module is configured to collect multi-exposure, multi-modal weld data in real time during the grinding and polishing process. The extracted weld visual features and robot state are combined to form a strategy model input. The hybrid vision servo strategy model outputs control instructions to drive the robot to perform weld grinding and polishing.

[0008] Thirdly, based on the same inventive concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0009] Fourthly, based on the same inventive concept, this application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method as described in the first aspect.

[0010] Compared with existing technologies, the visual servo control method, system, equipment, and medium for weld grinding and polishing robots described in this application have the following advantages: This application, through cross-exposure registration and radiometric response compensation, reduces fusion ghosting, brightness distortion, and positioning drift compared to single-exposure imaging or simple fusion methods lacking response compensation, thereby improving the stability and reliability of weld feature extraction. Simultaneously, this application embeds a safety constraint layer into the online control link, limiting and constraining the output control commands of the strategy, and employs a confidence-triggered degradation mechanism to form a fallback-friendly safety closed loop. This effectively improves recognition stability, trajectory tracking accuracy, and operational safety under complex lighting and interference conditions, making it applicable to post-weld polishing operations such as weld reinforcement removal, weld toe transition trimming, and surface defect repair. Attached Figure Description

[0011] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a visual servo control method for a weld grinding and polishing robot according to an embodiment of this application; Figure 2This is a schematic diagram of the structure of a vision servo control system for a weld grinding and polishing robot according to an embodiment of this application; Figure 3 This is a schematic diagram of the hardware structure of the electronic device described in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0013] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0014] The weld grinding and polishing robot system provided in this embodiment includes at least: an industrial robot body, an end effector, an adjustable exposure industrial camera, a computing unit, and a robot controller; and may optionally include a depth camera or structured light sensor, a torque sensor, or a contact stability estimation module.

[0015] The system's data flow includes the following steps: acquiring multi-exposure multimodal data, performing cross-exposure registration and radiation response compensation, performing multimodal fusion and weld feature extraction, constructing strategy input states, outputting control commands from the strategy model, performing amplitude limiting and constraint projection through the safety constraint layer, sending them to the robot controller to perform polishing, and obtaining the next cycle observation to form a closed-loop feedback.

[0016] To ensure the accuracy of visual features and 3D mapping, camera intrinsic and extrinsic parameters can be calibrated before deployment, including camera intrinsic parameters, the calibration relationship between the camera and the robot base, and the extrinsic parameter relationship between multimodal sensors; and a unified expression between image coordinates, depth or structured light coordinates and robot coordinates can be achieved based on the calibration relationship.

[0017] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0018] Please see Figure 1As shown in the figure, this embodiment provides a vision servo control method for a weld seam grinding and polishing robot, which specifically includes the following steps: Step S10: On the same weld segment, acquire multiple visible light weld images with different exposure times according to the exposure sequence adapted to the weld material, and simultaneously acquire at least one of depth images and structured light images to obtain multi-exposure multi-modal weld data.

[0019] Specifically, in this embodiment, let the first... The exposure time of the exposed image is Corresponding pixel coordinates The imaging model can be represented as: ; In the formula, For the first Grayscale / intensity values ​​of the exposed image. It is an equivalent quantity of incident irradiance or scene radiance. Let be the camera's radiation response function.

[0020] For the same weld area, at least two images with different exposure times are acquired in each acquisition cycle. The exposure settings include at least a short exposure time and a long exposure time: the short exposure is used to suppress saturation in the bright areas of the weld, and the long exposure is used to enhance details in the dark areas. The multiple exposure images can be acquired continuously by the same camera in the same acquisition cycle according to a preset exposure sequence, or acquired in parallel by multiple cameras, and aligned by hardware triggering or timestamps to ensure that the exposure sequence corresponds to the observation of the same weld segment. In addition, to improve data quality, this embodiment also performs denoising, distortion correction and normalization on the images, and removes or downweights abnormal frames that are obviously affected by dust and sparks.

[0021] Step S20: Using the selected reference exposure image as a benchmark, feature matching and geometric transformation are performed on the remaining exposure images to align them to the same coordinate system. Radiometric response compensation is performed on the aligned multi-exposure images based on the exposure time and camera radiometric response function. Weights are assigned to each exposure image according to pixel saturation and brightness reliability, and the images are fused to obtain a bright and uniform visible light image.

[0022] Specifically, in this embodiment, this step mainly involves cross-exposure image registration and radiometric response compensation: a short-exposure or medium-exposure image is selected as the reference image, multi-scale feature extraction and matching are performed on the reference image and the remaining exposure images, and mismatched points are eliminated through robust estimation to obtain the geometric transformation matrix of each exposure image relative to the reference image. The geometric transformation is either affine or projective. Subsequently, each exposure image is aligned to the same reference coordinate system to obtain the registered multi-exposure image sequence.

[0023] To eliminate the differences in grayscale response caused by different exposures, a nonlinear mapping model between exposure time and pixel grayscale response is constructed to perform radiometric response compensation and brightness correction on the registered image sequence: short exposure information is preferentially used for grayscale recovery in overexposed areas, and long exposure information is preferentially used for dark areas and local contrast enhancement is performed to obtain a weld image sequence with a balanced dynamic range of brightness.

[0024] In addition, it can output image quality indicators, which can serve as one of the bases for subsequent weld visual feature confidence calculations or safety strategy triggering.

[0025] Specifically, in this embodiment, a reference exposure image is selected. As a baseline, for the remaining exposure images Feature extraction and matching are performed to estimate the geometric transformation relationship of the image relative to the reference image, and projection transformation is used to align it to the same reference coordinate system.

[0026] Wherein, pixel homogeneous coordinates with reference coordinates The relationship can be represented as: In the formula, For the first The transformation matrix from the first image to the reference image.

[0027] Registration quality can be characterized by reprojection residuals, for example: ; In the formula, To match point pairs, is the homogeneous normalized projection operator, and is the number of effective matching points.

[0028] In terms of radiation response compensation, it can be made As the inverse mapping of the camera response function, the radiance estimation of the same pixel in the logarithmic domain satisfies: ; To suppress unreliable observations in overexposed areas, weights can be assigned to each exposure. The radiance estimate is obtained by weighted fusion of multiple exposure results. : ; In the formula, To prevent extremely small quantities with a denominator of zero, weights are used. It can be set according to the proportion of saturated dark pixels or the degree to which the grayscale is close to the median, according to Reconstructable brightness-equalized image Furthermore, smoothing is performed at the region boundaries to suppress brightness abrupt changes, thereby achieving spatial and brightness consistency across exposures and providing stable input for subsequent multimodal fusion and pose estimation.

[0029] This application addresses the problems of underexposure and feature instability caused by strong reflectivity and varying illumination in weld seams. Based on multi-exposure acquisition, it first performs geometric registration on images from different exposures to unify coordinates. Then, it performs radiometric response compensation and brightness correction based on the nonlinear mapping between exposure time and pixel grayscale response, outputting a weld seam image sequence with a balanced dynamic range and spatial consistency. Compared to single-exposure imaging or simple fusion methods lacking response compensation, this application reduces fusion ghosting, brightness distortion, and positioning drift, improving the stability and reliability of weld seam feature extraction.

[0030] Step S30: Input the brightness-equalized visible light image and the acquired depth image and / or structured light image into the dual-branch feature coding network to perform feature coding and attention fusion on data from different sources to obtain fused features, and output the weld visual features and confidence index based on the fused features.

[0031] Specifically, in this embodiment, this step mainly involves multimodal fusion network and weld feature extraction: This embodiment uses a multimodal feature fusion network to achieve stable extraction of key weld features. The network includes a first feature encoding branch and a second feature encoding branch set in parallel: The first branch encodes the multi-exposure visible light weld image to obtain visual features; the second branch encodes the depth image and other modal data (other modal data are depth images and / or depth point clouds reconstructed by structured light, used to provide weld geometric information and assist in three-dimensional pose estimation) to obtain geometric features.

[0032] The features of the two branches are fused by an adaptive weighted fusion module to obtain a fused feature representation. Based on the fused feature, the weld area segmentation result can be output and edge detection can be performed (the purpose of edge detection is mainly to better lock the weld toe line to correct boundary distortion and obtain a continuous and smooth weld contour, which involves fitting the weld center line to the reference and calculating geometric parameters such as weld width). The weld skeleton is calculated or the center curve is fitted within the weld area to obtain the weld center line, and geometric indices such as weld width are estimated.

[0033] Furthermore, by combining camera calibration parameters, depth images, and structured light data, the centerline is mapped from image coordinates to a three-dimensional coordinate system, thus obtaining the three-dimensional centerline and three-dimensional pose information of the weld.

[0034] The aforementioned weld visual features can simultaneously output a confidence index, which can be composed of information such as network output probability, feature consistency test results, and image quality indicators, and is used for online policy input and security mechanism triggering.

[0035] Specifically, this embodiment uses a registered and compensated brightness-equalized image. Other modal data (depth images) or structured light point cloud The input is a multimodal feature fusion network, which encodes and fuses features from different sources to output weld visual features such as weld centerline, weld width, and weld three-dimensional pose.

[0036] Let the output characteristics of the visible light branch encoder be... The output features of the geometric branch encoder are The fusion feature can be represented as: ; In the formula, and For adaptive weighting coefficients, satisfying And it can be generated by attention mechanisms, for example: ; In the formula, For global average pooling, These are trainable parameters.

[0037] Weld seam region segmentation can output pixel-level probability maps and with threshold Forming a weld mask : ; Two-dimensional centerlines are obtained by skeletonization or centerline fitting within the weld mask. 3D mapping is achieved through depth / structured light. For depth image scenes, this can be achieved using the camera intrinsic parameter matrix. Mapping pixels to the camera coordinate system: ; In the formula, For depth value, This represents a 3D point in the camera coordinate system. Further, this is combined with the extrinsic parameter transformation from the camera to the robot's base coordinate system. This yields three-dimensional points in the base coordinate system: ; For the set of discrete points of the centerline It can construct a three-dimensional centerline. The weld width can be estimated by the Euclidean distance between the left and right boundary points of the weld, and the three-dimensional pose of the weld can be determined by the tangent vector along the centerline. and surface normal vector (Obtained by depth gradient or local plane fitting of point cloud) jointly determine, thereby establishing the local coordinate system of the weld. .

[0038] Step S40: Model the visual servo control of weld grinding and polishing as a Markov decision process, and obtain a hybrid visual servo strategy model through deep reinforcement learning training; wherein, the state includes at least weld feature error, robot joint state and contact information; the action includes joint velocity vector; the reward function includes trajectory fitting term, posture stability term, smoothing term, safety penalty term and grinding and polishing efficiency factor; wherein, the weld feature error is obtained by adaptive selection or weighting of pixel error and three-dimensional pose error according to confidence index.

[0039] Specifically, in this embodiment, this step mainly involves offline training of a hybrid visual servo control strategy based on deep reinforcement learning: modeling the weld grinding and polishing visual servo control as a Markov decision process. Among these processes, the state vector... It should include at least: the pixel error between the weld centerline and the end target in the image plane, the pose error between the 3D pose of the weld and the desired pose of the end target, and the robot joint angles. With joint velocity Contact force Or contact stability indicators, etc. Pixel error and pose error can be expressed as: ; ; In the formula, , These are the observed coordinates of the weld centerline or target feature in the image. , Let it be its expected coordinates; and These refer to the end position and attitude, respectively. This is the actual position vector of the robot's end effector. Let the desired position vector at the end be... This represents the actual attitude at the end point. The desired end-effector pose is represented by the symbol "⊖", which indicates the pose difference calculation (which can be achieved using axis angle or quaternion errors). To achieve adaptive selection of the "primary error", the confidence levels of pixel error and pose error can be calculated separately, and the one with the higher confidence level can be selected as the primary error.

[0040] In this embodiment, the joint velocity vector is preferably used as the final motion output; in another implementation, the policy network can also first output the end-effector torque, and then obtain the joint velocity through Jacobian mapping. ; In the formula, For the terminal linear velocity component, The terminal angular velocity component, for Action vector at any given moment.

[0041] If the robot controller requires joint velocity commands, these can be obtained using a Jacobian matrix. Map the end-effector velocity to the joint velocity: ; In the formula, For the generalized inverse, a damped least squares form is preferred to improve the numerical stability of near-singular configurations. Let the number of robot joints be... The policy network at time Output normalized action .

[0042] Based on the maximum permissible velocity vector of each joint Scaling is performed to obtain joint speed commands: ; In the formula, This is the element-wise product of Hadamard. To suppress jitter, further acceleration / rate of change constraints can be imposed: ).

[0043] If the system requires Cartesian space velocity for visual servo error analysis, it can be achieved by... Calculation; if the strategy in another implementation first outputs the end-velocity torque. Then, the damped least squares generalized inverse is used to calculate the joint velocity: Among them, damping factor With minimum singular value Decrease and increase to avoid near-singular amplification.

[0044] ; The reward function is used to simultaneously constrain trajectory alignment, attitude stability, motion smoothness, and safety risks. The weld seam's three-dimensional centerline is used as the basis for this constraint. For reference, the shortest distance from the end point to the center line can be defined as the trajectory error: Then the single-step reward can be expressed as: ; In the formula, As weight, To ensure safety, the quantity must be in violation of the instruction. This refers to the desired contact force (or desired contact stability index).

[0045] In this embodiment, the reward function is used to characterize objectives such as trajectory alignment, attitude stability, motion smoothness, and safety constraints: the trajectory error term is used to encourage the end path to align with the weld centerline; the attitude error term is used to maintain the tool attitude required by the grinding and polishing process; the smoothing term or penalty term is used to suppress speed / acceleration changes and motion jitter; and penalty terms are set for collision risk, joint limitation, or loss of field of view.

[0046] During the offline training phase, changes in weld morphology, lighting, and contact disturbances can be introduced into the simulation environment or offline data playback environment to improve the adaptability of the strategy to different working conditions; after training, the strategy model parameters that can be inferred online are obtained.

[0047] Step S50: Collect multi-exposure, multi-modal weld data in real time during the grinding and polishing process. Input the extracted weld visual features and robot state together to form a strategy model. Output control commands from the hybrid vision servo strategy model to drive the robot to perform weld grinding and polishing.

[0048] Specifically, in this embodiment, this step mainly involves online closed-loop execution and safety constraints. Following steps S20-S30, visual features of the weld are extracted and combined with the robot's state to form the strategy input. The strategy model outputs control commands to drive the robot to perform weld grinding and polishing, achieving adaptive adjustment of path and posture. Online visual servoing operates according to a preset control cycle. The following process is executed cyclically: collect multi-exposure multimodal weld seam data, process the data and extract weld seam visual features, collect robot status, output control commands according to the strategy model, drive the robot to perform grinding and polishing and determine whether the corresponding weld seam segment is completed.

[0049] In one implementation, the local coordinate system of the weld is obtained based on step S30. The pose can be represented as a homogeneous transformation matrix: ; In the formula, The local coordinate system of the weld relative to the robot's base coordinate system A 3×3 rotation matrix, For the corresponding position vector; Given a zero vector with 1 row and 3 columns, the desired pose of the endpoint is... It can be determined by process planning (such as along) The tangential and normal constraints are given. Using a logarithmic mapping to define the attitude error vector, the 3D pose error can be obtained as: ; ; ; In the formula, This is the actual rotation matrix at the end. For the desired rotation matrix at the end, " "This is the logarithmic mapping from rotation matrices to Lie algebras," The "vee" operator maps an antisymmetric matrix to a three-dimensional vector. Online policy inputs can include at least pose error, joint state, and contact information, for example: ; In the formula, A confidence index for the visual features of the weld. The strategy model outputs raw control commands in the form of joint velocities (or outputs end-effector torque first and then maps it): ; Furthermore, the control command is output after being limited and / or projected by the safety constraint layer to satisfy at least one of joint limiting, velocity constraints, collision distance, and field-of-view constraints. To uniformly express the "limiting / projection" process without changing the strategy structure, the safety constraint layer can be represented as the closest point projection of the original command, and the joint velocity output by the strategy can be... Project to feasible set get One implementation method is to solve the nearest point quadratic programming (QP) problem: ; In the formula, The current joint angle, , These are the upper and lower limits of joint velocity. To control the cycle, The minimum collision distance for the next cycle is predicted based on the current motion state. The safe distance threshold is used; FOV is a camera field of view / observability index. for subject to The abbreviation is used only to introduce constraints.

[0050] When QP is not feasible, a tiered degradation approach is adopted: ① Speed ​​scaling ② Maintain posture ③ Safe evacuation trajectory and shutdown alarm.

[0051] ; ; in, It is a feasible set consisting of engineering safety constraints, including at least one or more of the following: joint velocity upper and lower bound constraints, joint limit / joint velocity constraints, collision distance constraints, and field of vision constraints. To ultimately issue the joint speed command to the controller; if the strategy outputs in the form of end-effector torque, the joint speed can be obtained according to the mapping relationship in step S40. .

[0052] Confidence index The threshold can be obtained by combining the probability or confidence score output by the weld seam feature extraction network, the cross-exposure registration residual, and image quality indicators. To facilitate online threshold determination, the indicators can be normalized and then weighted and fused, with the result limited to [0, 1]. ; In the formula, The mean or median of the segmentation probability within the weld mask can be taken. The registration residual given in step S20, and These represent the saturated pixel ratio and the dark pixel ratio, respectively. Non-negative weights This is a truncation function.

[0053] Degradation or shutdown is triggered when the confidence level of the weld visual features falls below a threshold or the collision risk increases. Increased collision risk can be caused by the collision distance. The system determines whether a condition is met: the contact force is below a threshold, there is a sudden increase in contact force, the joint approaches the limit, or the field of vision constraint is violated. To unify the triggering logic, a safety trigger indicator can be defined. : ; In the formula, The confidence threshold. The minimum safe distance threshold, The maximum permissible contact force threshold, This is an indicator of field-of-view constraint failure (e.g., weld feature region leaving the effective field of view, effective pixel area below the threshold, or feature extraction failure for several consecutive cycles).

[0054] when When this occurs, a downgrade or shutdown is triggered; in downgrade mode, speed scaling can be applied to control commands while maintaining the tool posture or reverting to a safe posture.

[0055] This application embeds a safety constraint layer into the online control link, limiting and constraining the output control commands to ensure they meet at least one of joint limits, speed, collision distance, and field-of-view constraints. It also triggers degradation when the confidence level of weld visual features is insufficient or the risk increases, thus forming a fallback safety closed loop. Compared to schemes that directly output control quantities or rely solely on manual parameter tuning, this application improves control stability and engineering applicability under complex operating conditions.

[0056] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0057] Based on the same inventive concept, and corresponding to any of the above embodiments, the embodiments of this application also provide a visual servo control system for a weld grinding and polishing robot.

[0058] like Figure 2 As shown, the visual servo control system for the weld grinding and polishing robot includes: The image acquisition module 11 is configured to acquire multiple visible light weld images with different exposure times on the same weld segment according to the exposure sequence adapted to the weld material, and simultaneously acquire at least one of depth images and structured light images to obtain multi-exposure multimodal weld data. The image processing module 12 is configured to use the selected reference exposure image as a reference, perform feature matching and estimate geometric transformation on the remaining exposure images to align them to the same coordinate system, perform radiometric response compensation on the aligned multi-exposure images based on the exposure time and camera radiometric response function, and assign weights to each exposure image according to pixel saturation and brightness reliability, and fuse them to obtain a bright and uniform visible light image. The feature fusion module 13 is configured to input a brightness-equalized visible light image and a acquired depth image and / or structured light image into a dual-branch feature coding network to perform feature coding and attention fusion on data from different sources to obtain fused features, and output weld visual features and confidence index based on the fused features. Model training module 14 is configured to model the visual servo control of weld grinding and polishing as a Markov decision process, and obtain a hybrid visual servo strategy model through deep reinforcement learning. The state includes at least weld feature error, robot joint state and contact information; the action includes joint velocity vector; the reward function includes trajectory fitting term, posture stability term, smoothing term, safety penalty term and grinding and polishing efficiency factor; the weld feature error is obtained by adaptive selection or weighting of pixel error and three-dimensional pose error according to confidence index. The instruction output module 15 is configured to collect multi-exposure multimodal weld data in real time during the grinding and polishing process. The extracted weld visual features and robot state are combined to form the input of the strategy model. The hybrid vision servo strategy model outputs control instructions to drive the robot to perform weld grinding and polishing.

[0059] For ease of description, the above system is described by dividing it into various modules based on their functions. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0060] The system described in the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0061] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the above embodiments.

[0062] Figure 3 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0063] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0064] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0065] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0066] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0067] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0068] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0069] The electronic devices described above are used to implement the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0070] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0071] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0072] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0073] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0074] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the claims of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A visual servo control method for a weld grinding and polishing robot, characterized in that, include: Multiple visible light weld images with different exposure times are acquired on the same weld segment according to the exposure sequence adapted to the weld material, and at least one of depth image and structured light image is acquired simultaneously to obtain multi-exposure multi-modal weld data. Using the selected reference exposure image as a benchmark, feature matching and geometric transformation are performed on the remaining exposure images to align them to the same coordinate system. Radiometric response compensation is performed on the aligned multi-exposure images based on the exposure time and camera radiometric response function. Weights are assigned to each exposure image according to pixel saturation and brightness reliability, and the images are fused to obtain a bright and uniform visible light image. The brightness-equalized visible light image and the acquired depth image and / or structured light image are input into a dual-branch feature coding network to perform feature coding and attention fusion on data from different sources to obtain fused features. Based on the fused features, the weld visual features and confidence index are output. The visual servo control for weld grinding and polishing is modeled as a Markov decision process, and a hybrid visual servo strategy model is obtained through deep reinforcement learning training. The state includes at least weld feature error, robot joint state, and contact information; the action includes joint velocity vectors; the reward function includes trajectory fitting, posture stability, smoothing, safety penalty, and grinding and polishing efficiency factors; the weld feature error is obtained by adaptively selecting or weighting pixel error and 3D pose error according to the confidence index. The system collects multi-exposure, multi-modal weld data in real time during the grinding and polishing process. The extracted weld visual features and robot state are combined to form a strategy model input. The hybrid vision servo strategy model outputs control commands to drive the robot to perform weld grinding and polishing.

2. The method according to claim 1, characterized in that: The multi-exposure multimodal data is synchronously acquired through hardware triggering or timestamp alignment, and abnormal frames affected by dust or sparks are removed or downweighted.

3. The method according to claim 1, characterized in that: The geometric transformation is either an affine transformation or a projection transformation; the feature matching uses a robust estimation method to eliminate mismatched points, and the reprojection residual is used as the registration quality index.

4. The method according to claim 1, characterized in that: The radiometric response compensation includes inverse mapping of the camera's radiometric response function and estimation of radiance in the logarithmic domain; the weights are determined comprehensively based on the degree to which pixel grayscale approaches the median, the proportion of saturated pixels, the proportion of dark pixels, and the degree of dust interference.

5. The method according to claim 1, characterized in that: The weld visual features include weld area, weld centerline, weld width, and weld three-dimensional pose; wherein, the weld centerline is obtained by back-projecting pixels to the camera coordinate system through camera intrinsic parameters, and combining the extrinsic parameter transformation from camera to robot base coordinate system to construct a three-dimensional point set in the base coordinate system; The confidence index is obtained by comprehensively fusing the output probability of the feature extraction network, the cross-exposure registration residual, the saturated pixel ratio, the dark pixel ratio, and the image signal-to-noise ratio.

6. The method according to claim 1, characterized in that: The hybrid vision servo strategy outputs normalized joint velocity motions, which are scaled according to the maximum velocity vector of each joint to obtain joint velocity commands. After being limited or projected by the safety constraint layer, the commands are output to the robot controller for grinding and polishing. The safety constraint layer projects the joint speed command onto a feasible set consisting of joint limits, joint speed limits, collision distance thresholds, and field of view constraints. The projection method is either quadratic programming nearest point projection or equivalent constraint projection algorithm. When the feasible set is empty or the projection is infeasible, it is determined that the safety risk has increased, and a downgrade strategy is triggered.

7. The method according to claim 6, characterized in that: The degradation strategy includes at least one of the following: speed scaling, maintaining tool pose, reverting to a safe pose and re-sensing; The increased safety risk is determined by at least one of the following conditions: the collision distance is below the threshold, the contact force suddenly increases, the joint approaches the limit, or the weld feature extraction fails for several consecutive cycles.

8. A visual servo control system for a weld seam grinding and polishing robot, characterized in that, include: The image acquisition module is configured to acquire multiple visible light weld images with different exposure times on the same weld segment according to the exposure sequence adapted to the weld material, and simultaneously acquire at least one of depth images and structured light images to obtain multi-exposure multimodal weld data. The image processing module is configured to use the selected reference exposure image as a reference, perform feature matching and estimate geometric transformation on the remaining exposure images to align them to the same coordinate system, perform radiometric response compensation on the aligned multi-exposure images based on the exposure time and camera radiometric response function, and assign weights to each exposure image according to pixel saturation and brightness reliability, and fuse them to obtain a bright and uniform visible light image. The feature fusion module is configured to input the brightness-equalized visible light image and the acquired depth image and / or structured light image into a dual-branch feature coding network to perform feature coding and attention fusion on data from different sources to obtain fused features, and output weld visual features and confidence index based on the fused features. The model training module is configured to model the visual servo control of weld grinding and polishing as a Markov decision process, and obtain a hybrid visual servo strategy model through deep reinforcement learning. The state includes at least weld feature error, robot joint state, and contact information; the action includes joint velocity vectors; the reward function includes trajectory fitting, posture stability, smoothing, safety penalty, and grinding and polishing efficiency factors; and the weld feature error is obtained by adaptively selecting or weighting pixel error and 3D pose error according to the confidence index. The instruction output module is configured to collect multi-exposure, multi-modal weld data in real time during the grinding and polishing process. The extracted weld visual features and robot state are combined to form a strategy model input. The hybrid vision servo strategy model outputs control instructions to drive the robot to perform weld grinding and polishing.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that, in, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method described in any one of claims 1-7.

Images