Mechanical arm control method and system with visual sense and force sense fused perception

Abstract

The invention provides a visual sense and force sense fusion perception mechanical arm control method and system, and belongs to the field of automatic control. The method comprises the steps that mechanical arm posture data measured by a force sensor and mechanical arm posture data measured by a depth camera are fused through a data fusion method, and posture information of a current mechanical arm is obtained; according to the posture information of the current mechanical arm, an operation strategy and control parameters are generated, and a control instruction is obtained; and the mechanical arm is driven to complete operation according to the control instruction. The method is suitable for various industrial applications such as impact, grinding and roller painting. Through cooperative work of the mechanical arm, the force sensor, the execution tool, the depth camera and the mobile equipment, contact force control and dynamic adjustment of the tool posture in the operation process are achieved, and therefore the automation level of the system is improved, the manual intervention requirement is reduced, and the stability and accuracy of the operation process are improved.

Description

Technical Field

[0001] The present invention relates to the field of automatic control, and in particular to a robotic arm control method and system with vision and force fusion perception. Background Art

[0002] With the rapid development of industrial automation, robots are increasingly used in the manufacturing industry, especially in tasks that require high force and vision, such as impact, grinding, polishing, and roller coating. Robot systems are gradually replacing manual work to complete high-precision operations. However, these tasks are generally characterized by complex environments, variable object surfaces, and high difficulty in force control. Traditional robot control systems often rely on a single force or visual sensor, which makes it difficult to achieve real-time perception and dynamic adjustment during the operation process, resulting in low task quality and efficiency. For example, during impact or grinding, the robot needs to stably control the applied force to prevent damage due to excessive force or failure to achieve the expected effect due to insufficient force; during polishing and roller coating, the applied pressure and path need to be precisely controlled to ensure the uniformity and consistency of surface treatment. Summary of the invention

[0003] To solve the above problems, the present invention provides a robot arm control method and system with visual and force fusion perception, which is suitable for a variety of industrial applications such as impact, grinding and roller coating. Through the coordinated work of the robot arm, force sensor, execution tool, depth camera and mobile device, the contact force control and dynamic adjustment of the tool posture during the operation are realized, thereby improving the system automation level, reducing the need for manual intervention, and improving the stability and accuracy of the operation process, including:

[0004] A method for controlling a robotic arm by integrating vision and force perception, comprising:

[0005] S1. The robot arm posture data measured by the force sensor and the robot arm posture data measured by the depth camera are fused by a data fusion method to obtain the current robot arm posture information;

[0006] S2. Generate an operation strategy and control parameters according to the current posture information of the robot arm to obtain a control instruction;

[0007] S3. Drive the robot arm to complete the task according to the control instruction.

[0008] Optionally, in S1, the robot arm posture data measured by the force sensor and the robot arm posture data measured by the depth camera are fused by a data fusion method to obtain the current robot arm posture information formula (1):

[0009] Data=v×Data C +(1-v)×(α×DataF +β×Data A ); (1)

[0010] Among them, Data is the current posture state of the robot arm;

[0011] Data F The posture of the robot arm measured by the force sensor;

[0012] Data C The posture of the robotic arm measured by the depth camera;

[0013] Data A It is the original motion data posture information of the robot arm;

[0014] v is the depth camera occlusion measurement weight, which gradually decreases with the increase of occlusion and takes a value between [0,0.5];

[0015] α is the proportion of motion data measured by the force sensor;

[0016] β is the proportion of the motion data of the robot arm;

[0017] Optionally, the operation strategy in S2 includes: impact mode, grinding mode and roller coating mode;

[0018] In the operation strategy, the robot arm supports setting corresponding operation tools according to different modes.

[0019] Optionally, driving the robot arm to complete the operation according to the control instruction in S3 includes:

[0020] The current posture information of the robot arm is used as feedback and transmitted back to the force control stage of the robot arm for feedback adjustment control. The feedback adjustment formula is formula (2):

[0021]

[0022] Among them, x(t) is the feedback value of the robot arm posture at the current moment;

[0023] y(t) is the robot arm posture setting value at the current moment;

[0024] λ is the adjustment ratio, which controls the impact of feedback error on system update;

[0025] ω is the integral adjustment coefficient, which controls the cumulative effect of the error;

[0026] is the integral part of the error, representing the accumulation of the error from time 0 to t-1.

[0027] Optionally, driving the robot arm to complete the operation according to the control instruction in S3 further includes:

[0028] The adjustment ratio and integral adjustment coefficient are adjusted according to the current stage and the force control accuracy. The adjustment formula of the adjustment ratio and integral adjustment coefficient is formula (3):

[0029]

[0030] λ max is the maximum value of the adjustment ratio;

[0031] ω max is the maximum value of the integral adjustment coefficient;

[0032] γ is responsible for controlling the proportional coefficient, ζ is responsible for controlling the integral coefficient, and γ and ζ are adjusted between (0,10);

[0033] i = τ, τ is the intermediate process parameter;

[0034] After completing the adjustment of the proportional and integral adjustment coefficients, record the current compensation curve.

[0035] A robotic arm control system with vision and force fusion perception, applying the above method, includes:

[0036] Force sensors to monitor the interaction force between the robot's implement and the work surface;

[0037] Depth camera, used to collect three-dimensional visual information of the working area;

[0038] The host computer fuses the robot arm posture data measured by the force sensor and the robot arm posture data measured by the depth camera through a data fusion method to obtain the current posture information of the robot arm; the host computer generates an operation strategy and control parameters according to the current posture information of the robot arm to obtain a control instruction; the host computer drives the robot arm to complete the operation according to the control instruction.

[0039] Optionally, the three-dimensional visual information of the working area includes: geometric shape, texture features and positional relationship of the target surface;

[0040] The interaction force between the execution tool and the working surface includes: contact force, impact force and grinding force.

[0041] Compared with the prior art, the above technical solution has at least the following beneficial effects:

[0042] This system builds a highly adaptive robotic arm roller coating system through the close integration of three core modules: environmental perception, intelligent decision-making and action execution, which can realize automatic and uniform roller coating operation of the robotic arm on a plane.

[0043] The present invention provides a robot arm control system and method based on vision and force fusion perception. The system monitors the contact state between the tool at the end of the robot arm and the target surface in real time through force and vision perception modules, and automatically adjusts the pressure and posture of the tool according to sensor feedback information to achieve steady-state control and dynamic adaptive adjustment. The system accurately calculates and regulates the operating force of the robot arm in real time through fused vision and force signals, ensuring that the robot arm always maintains a constant and uniform pressure distribution during the operation process, thereby improving the quality and consistency of surface treatment of complex workpieces.

[0044] The above one or more technical solutions in the present invention have at least one of the following technical effects: the present invention provides a robotic arm control system and method with visual and force fusion perception, which realizes high-precision pressure control in a variety of operating environments by integrating force and visual information, saves manpower, site and equipment costs, and improves operating efficiency and resource utilization in industrial applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 This is the execution process of the method provided by the embodiment of the present invention;

[0047] Figure 2 A flow chart of a roller coating robot provided by an embodiment of the present invention;

[0048] Figure 3 A flow chart of an impact robot provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "one" or "the" do not indicate a quantitative limitation, but indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0051] It should be noted that the terms "up", "down", "left", "right", "front" and "back" used in the present invention are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0052] In order to meet the above challenges, applying the fusion of vision and force sensors to robot control is an effective solution. The vision sensor can provide global information of the environment, including target position, surface shape, and obstacles, while the force sensor can provide real-time feedback on the force value and contact status applied by the robot, thereby achieving more precise force control. By fusing vision and force information, the robot can adaptively adjust its posture and path in a dynamic environment, thereby achieving high-precision and high-efficiency operations in tasks such as impact, grinding, polishing, and roller coating.

[0053] Existing vision and force fusion systems are mostly used for simple assembly and handling tasks, and there is still a lack of efficient solutions for fine operations in complex mechanical environments. Therefore, developing a robot arm control method and system based on vision and force fusion perception can not only improve the automation level of complex tasks such as impact, grinding, polishing and roller coating, but also ensure the accuracy and safety of the operation, which is an important direction for the development of current robot control technology.

[0054] like Figure 1 As shown, the present invention provides a first aspect of a robotic arm control method with vision and force fusion perception, comprising:

[0055] S1. The robot arm posture data measured by the force sensor and the robot arm posture data measured by the depth camera are fused by a data fusion method to obtain the current robot arm posture information;

[0056] S2. Generate an operation strategy and control parameters according to the current posture information of the robot arm to obtain a control instruction;

[0057] S3. Drive the robot arm to complete the task according to the control instruction.

[0058] This method mainly involves a force sensor, a depth camera and a host computer, wherein the host computer is divided into an environment perception module, an intelligent decision-making module and an action execution module according to its functions.

[0059] The environmental perception module can collect parameters of the robotic arm and the working environment based on force sensors and depth cameras. According to needs, a position sensor can be set to sense the three-dimensional position of the robotic arm, generate comprehensive perception information, and provide certain support for subsequent decisions.

[0060] The relative viewing angle of the entire object and the dust occlusion at the scene will affect the visual data. Fusion is performed according to the changes in the visual environment at the scene. The fusion formula is shown in formula (1).

[0061] Data=v×Data C +(1-v)×(α×Data F +β×Data A ); (1)

[0062] Among them, Data is the current posture state of the robot arm;

[0063] Data F The posture of the robot arm measured by the force sensor;

[0064] Data C The posture of the robotic arm measured by the depth camera;

[0065] Data A It is the original motion data posture information of the robot arm;

[0066] v is the depth camera occlusion measurement weight, which gradually decreases with the increase of occlusion and takes a value between [0,0.5];

[0067] α is the proportion of motion data measured by the force sensor;

[0068] β is the proportion of the motion data of the robot arm;

[0069] According to formula (1), α is the proportion of force sensor measurement; β is the proportion of motion data of the robot module. The two proportions can be adjusted according to the on-site operation conditions.

[0070] The intelligent decision-making module analyzes the perceived data, and the end tool can be modularly designed according to the job requirements. When the job is ready, the required tools are automatically picked up through the front clamp of the robot arm according to the settings, and the force control strategy suitable for the current job is selected. The control parameters are adjusted in real time to meet various industrial tasks such as impact, grinding and roller coating. The intelligent decision-making module also adapts the optimized parameters of the motion control system in different job scenarios to ensure the stability and efficiency of the robot arm in the job scenario.

[0071] The action execution module includes the drive unit, the robot motor, the end effector and the hydraulic system. The execution module accurately controls the robot action and tool posture according to the decision instructions. The impact force intensity is controlled in the impact task, the constant pressure is maintained during grinding, and the roller coating task ensures uniform coating.

[0072] In a specific implementation manner, the operation strategies in S2 include: impact mode, grinding mode and roller coating mode;

[0073] In the operation strategy, the robot arm supports setting corresponding operation tools according to different modes.

[0074] In addition, the host computer in this method combines the depth camera and various required sensors to support the following data processing for the robot arm adjustment, and the relevant specific processing means are existing technologies:

[0075] The host computer supports real-time acquisition of sensor data, such as key parameters such as pressure, torque, and displacement. The host computer supports preliminary filtering and coordinate conversion of the collected data, and optimizes the raw data to ensure the validity and accuracy of the information; the host computer can be used to monitor changes in sensor data, especially when abnormal values ​​or abnormal fluctuations are detected, to ensure system stability through alarms or automatic adjustments. The host computer supports automatic adjustment of the pressure applied by the tool based on the real-time collected data to ensure operation accuracy. The three-dimensional image information of the operation area captured by the depth camera collected by the host computer mainly includes the shape and contour of the surface. The host computer calculates and processes the images captured by the depth camera to generate a three-dimensional model of the surface and operation path planning. The host computer supports the detection of abnormalities that occur in the image acquisition process of the depth camera, such as image blur, missing data, or excessive noise, and timely correction through the feedback mechanism to ensure the accuracy of subsequent image data processing and path planning. The host computer realizes the path planning and motion control of the robot arm through the calculation support of relevant algorithms, ensuring that the robot arm can operate according to the preset trajectory and speed. It uses the pressure feedback of the sensor module and the information of the depth camera module to output different force control instructions through the intelligent decision-making module according to different operation scenarios. The motion control module outputs different force control instructions through the intelligent decision-making module. According to the different working modes of the three operations of impact, grinding and roller coating, the corresponding control model is established by using the data of visual and force fusion perception, and the force control in each operation is divided into multiple force control stages, and the corresponding control strategy is designed to realize adaptive control of the robot arm.

[0076] The step S3 of driving the robot arm to complete the operation according to the control instruction includes:

[0077] S301, the posture information of the current robot arm is used as feedback and transmitted back to the force control stage of the robot arm for feedback adjustment control. The feedback adjustment formula is formula (2):

[0078]

[0079] Among them, x(t) is the feedback value of the robot arm posture at the current moment;

[0080] y(t) is the robot arm posture setting value at the current moment;

[0081] λ is the adjustment ratio, which controls the impact of feedback error on system update;

[0082] ω is the integral adjustment coefficient, which controls the cumulative effect of the error; γ and ζ are the adjustment factors of the two coefficients λ and ω, and γ and ζ are adjusted between (0,10).

[0083] It is the integral part of the error, representing the accumulation of errors from time 0 to t-1, and τ is the intermediate process parameter.

[0084] S302, adjusting the adjustment ratio and the integral adjustment coefficient according to the current stage and the force control accuracy, the adjustment formula of the adjustment ratio and the integral adjustment coefficient is formula (3):

[0085]

[0086] λ max To adjust the maximum value of the ratio;

[0087] ω max is the maximum value of the integral adjustment coefficient;

[0088] γ is responsible for controlling the proportional coefficient, and ζ is responsible for controlling the integral coefficient;

[0089] i = τ, τ is the intermediate process parameter;

[0090] S303. After the adjustment ratio and integral adjustment coefficient are completed, the current compensation curve is recorded and called in the subsequent actual production operation process to ensure the stable operation of the robot arm in a complex operating environment.

[0091] In S302, the general principle for adjusting the parameter adjustment ratio and the integral adjustment coefficient is: increase the adjustment ratio of λ and ω when fine adjustment of the robot arm state is required, reduce the adjustment ratio of λ and ω when coarse adjustment is required, and automatically adjust the two coefficients according to the error size.

[0092] In the specific implementation, the upper computer calculates the output posture coordinates so that the execution tool always contacts the working surface at the optimal angle. When the operating status of the monitoring robot arm is dangerous, especially when the robot arm makes accidental contact, the safety monitoring module will sound an alarm and automatically stop the operation to avoid equipment damage or safety accidents.

[0093] A robotic arm control system with vision and force fusion perception, applying the above method, includes:

[0094] Force sensors to monitor the interaction force between the robot's implement and the work surface;

[0095] Depth camera, used to collect three-dimensional visual information of the working area;

[0096] The host computer fuses the robot arm posture data measured by the force sensor and the robot arm posture data measured by the depth camera through a data fusion method to obtain the current posture information of the robot arm; the host computer generates an operation strategy and control parameters according to the current posture information of the robot arm to obtain a control instruction; the host computer drives the robot arm to complete the operation according to the control instruction.

[0097] In a specific implementation manner, the three-dimensional visual information of the working area includes: geometric shape, texture features and positional relationship of the target surface;

[0098] The interaction force between the execution tool and the working surface includes: contact force, impact force and grinding force.

[0099] Combining the above method with the above system, the present invention provides the following specific implementation methods when implementing the method:

[0100] First specific implementation method:

[0101] Taking roller coating operation as a typical application scenario, the system provided by the present invention can efficiently and stably complete complex operation processes. The system moves the material roller to the position where it is about to contact the target area of ​​roller coating by adjusting the mechanical arm. After it is in place, the force sensor is first calibrated to zero, and then the data is filtered to obtain the actual pressure value during roller coating. The depth camera captures the three-dimensional geometric information of the blade surface in real time, which is used for path planning and boundary judgment during roller coating. It is then adjusted according to the expected pressure difference and the unit speed, and the stepper motor speed control amount is limited. The control method and device of the roller coating pressure provided by the present invention automatically calibrates the pressure sensor to zero before each roller is pressed, and filters and coordinates the pressure sensor measurement value, and online identifies the environmental stiffness parameters, optimizes and compensates for the expected displacement, and uses the above-mentioned adaptive parameter impedance controller to control the roller pressure, so that the roller pressure of the roller coater can maintain good control accuracy from beginning to end, so as to improve the quality stability of the entire roller coating process.

[0102] Figure 2 It is a schematic diagram of the process of the roller coating robot provided by the present invention, including a description of the sequence of steps of all processes from the beginning to the end, and the main steps include step S10, step S20, step S30, step S40, and step S50.

[0103] In step S10, the system is initialized to complete tasks including zeroing the force sensor, focusing the depth camera and limiting the working range, inputting the roller coating pressure, setting the roller coating speed, and the coordinates of the roller coating start and end points.

[0104] In step S20, the depth camera captures the three-dimensional geometric information of the wind turbine blade in real time, and transmits the collected data to the host computer for roller coating path planning and boundary judgment; the robotic arm performs traditional impedance control to brake to the starting position to prevent damage to the equipment due to collision.

[0105] In step S30, the host computer uses the point cloud information obtained by the depth camera in the previous step to generate a smooth and continuous roller coating trajectory through a path planning algorithm, which is used for roller coating path planning to ensure that the coating tool moves at a uniform speed along the blade surface.

[0106] In step S40, the host computer first switches to the pressure control mode, and adjusts the posture of the roller coating tool at the end of the robot arm according to the data of the force sensor to adjust the contact pressure between the roller coating tool and the blade surface; at the same time, according to the planned roller coating path, the end painting tool is driven to move up and down along the blade surface for painting. During this process, the force sensor monitors the size of the contact force in real time, generates different impedance parameters according to different workpiece environments, and the posture of the robot arm is adjusted in real time to keep the roller and the blade vertical and the set pressure during painting.

[0107] In step S50, the boundary recognition technology uses the three-dimensional data collected by the depth camera and the contour extraction of the image processing algorithm to identify the boundary of the wind turbine blade, and compares the real-time collected position data with the preset set position to prevent the painting tool from exceeding the target area.

[0108] It should be noted that, in order to ensure coverage of the painting area, the robot arm drives the painting tool to move multiple times according to the needs of roller coating until all the reachable areas at the current position are fully roller coated.

[0109] It should be noted that the roller coating decision is generated at least based on the roller coating trajectory strategy and the pressure control strategy, that is, the roller coating decision includes the roller coating movement trajectory generated by the upper computer, the pressure control method between the roller and the roller coating target area, the working mode of the roller coating tool, etc.

[0110] Figure 2 The overall automatic roller coating control process can be summarized as follows:

[0111] (1) Start the equipment. The robot arm is first adjusted to a preset safe position, that is, above the wind turbine blades, to avoid collision with the blades or brackets before the operation starts.

[0112] (2) Initialization settings, zeroing of the force sensor, and limitation of the depth camera focus and working range.

[0113] (3) Input the roller coating pressure, set the roller coating speed, the coordinates of the roller coating start and end points, and paint the roller coating equipment.

[0114] (4) Impedance control is performed to move the robot arm to the starting position to prevent damage to the equipment due to collision.

[0115] (5) Switch to the pressure control mode and use the adaptive impedance control algorithm to adjust the posture of the roller coating tool at the end of the robot arm according to the data of the force sensor to adjust the contact pressure between the roller coating tool and the blade surface.

[0116] (6) According to the planned roller coating path, the robotic arm drives the end painting tool to move up and down along the surface of the blade to paint. During this process, the force sensor detects the magnitude of the contact force in real time, and the position of the robotic arm is adjusted in real time to keep the roller and the blade vertical and the set pressure during painting.

[0117] (7) To ensure coverage of the painting area, the robot arm drives the painting tool to move multiple times according to the needs of roller coating until all the areas reachable at the current position are fully coated.

[0118] (8) If a system failure occurs during this period, such as sensor data abnormality, camera image abnormality, robotic arm failure, etc., or the coating has reached the edge of the blade, the system will record the status information when the failure occurred and end the coating task.

[0119] (9) Determine whether the set end position has been reached. If not, return to step (5) to continue the operation; if reached, complete the current stage of coating operation and transfer to the next stage position.

[0120] During the entire roller coating process, there is almost no dust and high-gloss work on site, so the weight distribution of each item of multi-sensor data fusion in formula (1) is as follows:

[0121] In the rough large-scale roller coating stage, the system has low requirements for position control, and the depth camera occlusion measurement weight v is 0.5, the force sensor measurement weight α is 0.3, and the robot arm's motion data weight β is 0.7; when the system enters the fine roller coating stage, the depth camera occlusion measurement weight v remains unchanged at 0.5, the force sensor measurement weight α and the robot arm's motion data weight β are linearly switched to α at 0.6 and β at 0.4 within time t, where t can be modified according to the system adjustment amplitude. In this embodiment, t is 60s.

[0122] The proportional adjustment coefficients λ and ω of the feedback adjustment in formula (2) are adjusted by the integral adjustment coefficients γ and ζ in formula (3). The specific scheme is as follows:

[0123] In the rough large-scale roller coating stage, the system does not need to perform fine control of the position for efficient large-scale roller coating. The two coefficients γ and ζ are adjusted by a linear function:

[0124]

[0125] y is the output result of γ and ζ, and x is the error of the current system in mm. When the error is too large and the result of formula (4) is greater than 10, that is, the system error is greater than 200 mm, the function always outputs 10.

[0126] In the fine roller coating stage, the system needs to quickly adjust the error, so the two coefficients γ and ζ are adjusted through exponential functions:

[0127]

[0128] When the error is too large and the result of formula (5) is greater than 10, that is, the system error is greater than 50 mm, the function always outputs 10.

[0129] Second specific implementation:

[0130] Taking impact operation as a typical application scenario, the system provided by the present invention can efficiently and stably complete complex operation processes. The system adjusts the mechanical arm to move the impact robot to the position where the eye is about to be opened in the target area of ​​the furnace eye. After it is in place, the pressure sensor is first calibrated to zero, and then the data is filtered to obtain the pressure during the impact. The three-dimensional geometric information of the furnace eye and the impact platform is captured in real time using a depth camera for path planning and boundary judgment during the impact. It is then adjusted adaptively according to the expected pressure difference and the unit speed, and the stepper motor speed and hydraulic oil pressure control amount are limited. The impact pressure control method and device provided by the present invention automatically calibrates the pressure sensor once before each impact pressure, and filters and coordinates the measured value of the pressure sensor. By online identification of the environmental stiffness parameters, the expected displacement is optimized and compensated, and the above-mentioned adaptive parameter impedance controller is used to control the impact end pressure, thereby accurately controlling the pressure of the mechanical arm to improve the operation stability during the entire impact process.

[0131] Figure 3 It is a schematic diagram of the flow of the impact robot provided by the present invention, including a description of the sequence of steps of all processes from the beginning to the end, and the main steps include step S10, step S20, step S30, step S40, and step S50.

[0132] In step S10, the system is initialized to complete tasks including zeroing the force sensor, focusing the depth camera and limiting the working range, inputting the impact pressure, setting the impact speed, and the coordinates of the impact start and end points.

[0133] In step S20, the depth camera captures the three-dimensional geometric information of the furnace eye and the impact platform in real time, and transmits the collected data to the host computer for impact path planning and boundary judgment; the robotic arm performs traditional impedance control braking to the starting position to prevent damage to the equipment due to collision.

[0134] In step S30, the host computer uses the point cloud information obtained by the depth camera in the previous step to generate a smooth and continuous impact trajectory through a path planning algorithm, which is used for impact path planning to ensure that the impact platform moves at a uniform speed.

[0135] In step S40, the host computer first switches to the pressure control mode, and adjusts the posture of the impact platform at the end of the robot arm according to the data of the force sensor to adjust the contact pressure of the impact end; at the same time, according to the planned impact path, during this process, the force sensor monitors the size of the contact force in real time, generates different impedance parameters according to different workpiece environments, and the posture of the robot arm is adjusted in real time to maintain the pressure set at the impact end during impact.

[0136] In step S50, the boundary recognition technology uses the three-dimensional data collected by the depth camera and the contour extraction of the image processing algorithm to identify the impact boundary of the furnace eye, and at the same time compares the real-time collected position data with the preset set position to prevent the impact end from exceeding the target area and damaging the furnace wall.

[0137] During the entire impact process, there are more dust and high-light operations on site. The weight distribution of each item of multi-sensor data fusion in formula (1) is as follows:

[0138] At the beginning of the impact, the system has low requirements for position control, and the environment and occlusion are relatively small. The depth camera occlusion measurement weight v is 0.5, the force sensor measurement weight α is 0.3, and the robot arm's motion data weight β is 0.7; when the system enters the impact hole opening stage, affected by the environment, the depth camera occlusion measurement weight v is 0.2, the force sensor measurement weight α and the robot arm's motion data weight β are linearly switched to α of 0.7 and β of 0.3 within t time, where t can be modified according to the adjustment amplitude of the system. In this embodiment, t is 40s; when the dust or highlight on the scene is too large to make imaging impossible, the camera occlusion measurement weight v is 0, and the system only relies on the force sensor and the robot arm's motion data to distribute weights.

[0139] The proportional adjustment coefficients λ and ω of the feedback adjustment in formula (2) are adjusted by the integral adjustment coefficients γ and ζ in formula (3). The specific scheme is as follows:

[0140] In the rough control stage of impact, the system mainly reaches the predetermined position quickly, and there is no need to finely control the position. The two coefficients γ and ζ are adjusted by a linear function:

[0141]

[0142] y is the output result of γ and ζ, and x is the system error in mm. When the error is too large and the result of formula (6) is greater than 10, that is, the system error is greater than 300 mm, the function always outputs 10.

[0143] In the stage of fine impact control, the system needs to quickly adjust the error to accurately impact the furnace eye, so the two coefficients γ and ζ are adjusted through exponential functions:

[0144]

[0145] When the error is too large and the formula result is greater than 10, that is, the system error is greater than 100mm, the function always outputs 10.

[0146] This system builds a highly adaptive robotic arm roller coating system through the close integration of three core modules: environmental perception, intelligent decision-making and action execution, which can realize automatic and uniform roller coating operation of the robotic arm on a plane.

[0147] The present invention provides a robot arm control system and method based on vision and force fusion perception. The system monitors the contact state between the tool at the end of the robot arm and the target surface in real time through force and vision perception modules, and automatically adjusts the pressure and posture of the tool according to sensor feedback information to achieve steady-state control and dynamic adaptive adjustment. The system accurately calculates and regulates the operating force of the robot arm in real time through fused vision and force signals, ensuring that the robot arm always maintains a constant and uniform pressure distribution during the operation process, thereby improving the quality and consistency of surface treatment of complex workpieces.

[0148] The above one or more technical solutions in the present invention have at least one of the following technical effects: the present invention provides a robotic arm control system and method with visual and force fusion perception, which realizes high-precision pressure control in a variety of operating environments by integrating force and visual information, saves manpower, site and equipment costs, and improves operating efficiency and resource utilization in industrial applications.

[0149] The following points need to be explained:

[0150] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention, and other structures may refer to the general design.

[0151] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present invention, the thickness of the layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or there may be intermediate elements.

[0152] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to obtain new embodiments.

[0153] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A robot arm control method based on vision and force fusion perception, characterized in that: include: S1. The robot arm posture data measured by the force sensor and the robot arm posture data measured by the depth camera are fused by a data fusion method to obtain the current robot arm posture information; S2. Generate an operation strategy and control parameters according to the current posture information of the robot arm to obtain a control instruction; S3. Drive the robot arm to complete the task according to the control instruction.

2. The method for controlling a robotic arm based on the fusion of vision and force perception according to claim 1, characterized in that: In S1, the robot arm posture data measured by the force sensor and the robot arm posture data measured by the depth camera are fused by the data fusion method to obtain the current robot arm posture information formula (1): Data=v×Data C +(1-v)×(α×Data F +β×Data A ); (1) Among them, Data is the current posture state of the robot arm; Data F The posture of the robot arm measured by the force sensor; Data C The posture of the robotic arm measured by the depth camera; Data A It is the original motion data posture information of the robot arm; v is the depth camera occlusion measurement weight, which gradually decreases with the increase of occlusion and takes a value between [0,0.5]; α is the proportion of motion data measured by the force sensor; β is the proportion of the robot's motion data.

3. According to the method for controlling a robotic arm with vision and force fusion perception as described in claim 2, the operation strategy in S2 comprises: Impact mode, grinding mode and roller mode; In the operation strategy, the robot arm supports setting corresponding operation tools according to different modes.

4. The method for controlling a robotic arm based on the fusion of vision and force perception according to claim 2, characterized in that: The step S3 of driving the robot arm to complete the operation according to the control instruction includes: The current posture information of the robot arm is used as feedback and transmitted back to the force control stage of the robot arm for feedback adjustment control. The feedback adjustment formula is formula (2): Among them, x(t) is the feedback value of the robot arm posture at the current moment; y(t) is the robot arm posture setting value at the current moment; λ is the adjustment ratio, which controls the impact of feedback error on system update; ω is the integral adjustment coefficient, which controls the cumulative effect of the error; It is the integral part of the error, representing the accumulation of errors from time 0 to t-1, and τ is the intermediate process parameter.

5. The method for controlling a robotic arm based on vision and force fusion perception according to claim 4, characterized in that: The step of driving the robot arm to complete the operation according to the control instruction in S3 further includes: The adjustment ratio and integral adjustment coefficient are adjusted according to the current stage and the force control accuracy. The adjustment formula of the adjustment ratio and integral adjustment coefficient is formula (3): λ max is the maximum value of the adjustment ratio; ω max is the maximum value of the integral adjustment coefficient; γ is responsible for controlling the proportional coefficient, ζ is responsible for controlling the integral coefficient, and γ and ζ are adjusted between (0,10); i = τ, τ is the intermediate process parameter; After completing the adjustment of the proportional and integral adjustment coefficients, record the current compensation curve.

6. A robotic arm control system with vision and force fusion perception, characterized in that: The method according to any one of claims 1 to 5 comprises: Force sensors to monitor the interaction force between the robot's implement and the work surface; Depth camera, used to collect three-dimensional visual information of the working area; The host computer fuses the robot arm posture data measured by the force sensor and the robot arm posture data measured by the depth camera through a data fusion method to obtain the current posture information of the robot arm; the host computer generates an operation strategy and control parameters according to the current posture information of the robot arm to obtain a control instruction; the host computer drives the robot arm to complete the operation according to the control instruction.

7. The robot arm control system with vision and force fusion perception according to claim 6, characterized in that: The three-dimensional visual information of the working area includes: the geometric shape, texture characteristics and positional relationship of the target surface; The interaction force between the execution tool and the working surface includes: contact force, impact force and grinding force.

Images