A vision-based end-effector control method and system for collaborative robots
By acquiring visual and mechanical data, the hardness and softness of the target object are quantified, and the control data is dynamically corrected. This solves the problem of the inadequacy of traditional robot control methods in adapting to objects with complex materials and shapes, and achieves accurate perception and adaptive control of the target object, thereby improving operational safety and control precision.
Patent Information
- Application Number
- CN202611282346.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-28
- Publication Date
- 2026-11-10
AI Technical Summary
Traditional vision-based robot control methods are not adaptable enough when dealing with objects with complex materials and shapes, resulting in inaccurate gripping point positioning and improper gripping force control, which can lead to safety risks such as gripping failure, waste spillage, and object breakage.
By acquiring initial control data and continuously collecting visual and mechanical data during pre-contact, the softness and hardness of the target object are quantified using a soft-hardness recognition model based on visual and mechanical dynamic features. The control data is then dynamically corrected through a closed-loop architecture to achieve accurate perception and adaptive control of the target object.
It significantly improves the operational safety, reliability, and control precision of collaborative robots when dealing with complex materials and shapes, and solves the problem of insufficient adaptability of traditional methods.
Smart Images

Figure CN123069073A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of visual robot technology, and in particular to a vision-based collaborative robot end-effector control method and system. Background Technology
[0002] Vision-based robots possess powerful environmental perception and target recognition capabilities. They can acquire two-dimensional images and three-dimensional spatial information in a scene in real time, thereby dynamically planning motion trajectories, identifying target categories and poses, and achieving autonomous operation in unstructured or semi-structured environments. They are particularly suitable for complex task scenarios that require high-precision recognition and operation.
[0003] However, traditional vision-based robot control methods still suffer from significant limitations in adapting to complex materials and shapes when applied to certain specialized fields. Take medical waste disposal as an example: medical waste is diverse, encompassing a wide range of objects with varying degrees of hardness, including translucent materials of different textures (such as glass ampoules and plastic IV bottles) and flexible shapes (such as wads of infectious gauze and soft dressings). While traditional vision algorithms can locate such objects, robots struggle to accurately assess their shape changes under stress. This can easily lead to inaccurate gripping point positioning and improper grasping force control, resulting in safety risks such as gripping failure, waste spillage, object breakage, and liquid spillage.
[0004] Therefore, a vision-based end-effector control method is needed to address the existing challenges. Summary of the Invention
[0005] Therefore, this invention provides a vision-based collaborative robot end-effector control method to solve the problem of insufficient adaptability of existing vision control methods when dealing with objects with complex materials and shapes.
[0006] This invention provides a vision-based end-effector control method for collaborative robots, comprising: Acquire initial control data, which is used to control the collaborative robot's end effector to make pre-contact with the target object; During the pre-contact between the collaborative robot's end effector and the target object, visual data of the target object and mechanical data of the collaborative robot's end effector are continuously collected. Based on changes in visual and mechanical data, a softness-hardness index is obtained to represent the softness or hardness of the target object. Based on the soft and hard indices and the initial control data, the actual control data for controlling the collaborative robot's end effector to manipulate the target object is obtained.
[0007] In a preferred embodiment: based on changes in visual and mechanical data, a hardness index representing the softness or hardness of the target object is obtained, including: Feature extraction is performed on changes in visual data to obtain visual dynamic features; Feature extraction is performed on changes in mechanical data to obtain dynamic mechanical features; Based on visual dynamic features and mechanical dynamic features, and using a pre-defined soft and hard recognition model, a soft and hard index is obtained.
[0008] In a preferred embodiment: visual dynamic features include the point cloud deformation rate of the target object; mechanical dynamic features include the rise time, peak force, fluctuation standard deviation, number of force mutations, and force response consistency of a single robot; the soft and hard recognition model is an SVM model.
[0009] In a preferred embodiment: visual dynamic features include the point cloud deformation rate of the target object; feature extraction is performed on the changes in visual data to obtain visual dynamic features, including: Based on visual data, the rate of change of the target object's outline is obtained; Based on the mechanical data, the contact force at the end of the collaborative robot is obtained; Acquire displacement data of the collaborative robot's end effector; The point cloud deformation rate is obtained based on the contour change rate, contact force, and displacement data.
[0010] In a preferred embodiment: the point cloud deformation rate is obtained based on the profile change rate, contact force, and displacement data, including: The depth feature approximation factor is obtained based on the displacement data, the ratio of displacement data to contact force, and the ratio of profile change rate to displacement data. The point cloud deformation rate is obtained based on the contour change rate and the depth feature approximation factor.
[0011] In a preferred embodiment: based on visual dynamic features and mechanical dynamic features, and using a pre-defined soft-hard recognition model, a soft-hard index is obtained, including: Visual dynamic features and mechanical dynamic features are input into the soft and hard recognition model to obtain the initial soft and hard index output by the soft and hard recognition model. The visual dynamic features and mechanical dynamic features are input into the preset pollution identification model, and the initial soft and hard index is corrected based on the pollution identification model to obtain the corrected soft and hard index.
[0012] In a preferred embodiment: the contamination recognition model includes an LSTM neural network layer, a self-attention neural network layer, and a fully connected layer; visual dynamic features and mechanical dynamic features are input into the pre-defined contamination recognition model, and the initial soft-hard index is corrected based on the contamination recognition model to obtain the corrected soft-hard index, including: The visual dynamic features, mechanical dynamic features, and corresponding initial soft and hard indices are spliced together to obtain the input vector, and the input vectors at multiple time points are aggregated to obtain the input matrix; The input matrix is fed into the LSTM neural network layer to obtain the hidden state vector sequence output by the LSTM neural network layer; The hidden state vector sequence is input into the self-attention neural network layer to obtain the weighted aggregate vector output by the self-attention neural network layer; The weighted aggregation vector is input into the fully connected layer to obtain the corrected soft and hard exponents of the fully connected layer output.
[0013] The present invention also provides a vision-based collaborative robot end effector control system, comprising: The pre-contact module is used to acquire initial control data, which is used to control the collaborative robot's end effector to make pre-contact with the target object. The data monitoring module is used to continuously collect visual data of the target object and mechanical data of the collaborative robot's end effector when the end effector makes pre-contact with the target object. The soft-hardness recognition module is used to obtain a soft-hardness index representing the softness or hardness of a target object based on changes in visual and mechanical data. The fine-tuning module is used to obtain the actual control data for controlling the collaborative robot's end effector to manipulate the target object, based on the soft and hard indices and the initial control data.
[0014] The present invention also provides an electronic device, comprising: Memory and processor; The memory is used to store the program, and the processor is used to execute the steps in any of the vision-based collaborative robot end-effector control methods described above when the program is executed.
[0015] The present invention also provides a computer-readable storage medium for storing a computer-readable program or instruction, which, when executed by a processor, can implement the steps in any of the above-described vision-based collaborative robot end-effector control methods.
[0016] The beneficial effects of adopting the above scheme are: This invention provides a vision-based collaborative robot end-effector control method. It first acquires initial control data for controlling the collaborative robot's end-effector to make pre-contact with a target object. During this pre-contact process, it continuously collects visual data of the target object and mechanical data of the robot's end-effector. Based on changes in the visual and mechanical data, a hardness index representing the target object's softness or hardness is obtained. Finally, based on the hardness index and the initial control data, actual control data for controlling the collaborative robot's end-effector to manipulate the target object is obtained. Compared to existing technologies, this invention achieves real-time, accurate perception and adaptive control of the target object's material properties through a closed-loop architecture of "pre-contact - dynamic perception - hardness quantification - data correction." Specifically, it captures the dynamic response characteristics of the target object under force through pre-contact data fusion and multi-source data fusion, and then quantifies and outputs continuous soft and hard indices. Finally, it dynamically corrects the initial control data so that the actual control data can match the real physical characteristics of the target object. This overcomes the limitation of traditional methods that can only make qualitative judgments on complex materials, and significantly improves the operational safety, reliability and control accuracy of collaborative robots when facing complex materials and shapes. It also solves the problem of insufficient adaptability of existing vision control methods when facing objects with complex materials and shapes. Attached Figure Description
[0017] Figure 1 A flowchart of the vision-based collaborative robot end-effector control method provided by the present invention; Figure 2 for Figure 1 A detailed step diagram of step S103 is shown below; Figure 3 for Figure 2 A detailed step diagram of step S203 is shown below; Figure 4 for Figure 3 A detailed step diagram of step S302 is shown below; Figure 5 This is a schematic diagram of the model structure for software and hardware recognition in this invention; Figure 6 The system architecture diagram of the vision-based collaborative robot end effector control system provided by the present invention is shown. Detailed Implementation
[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0019] Combination Figure 1As shown, this invention discloses a vision-based collaborative robot end-effector control method, comprising: S101. Obtain initial control data. The initial control data is used to control the end effector of the collaborative robot to make pre-contact with the target object. S102. When the collaborative robot's end effector makes pre-contact with the target object, continuously collect visual data of the target object and mechanical data of the collaborative robot's end effector. S103. Based on the changes in visual and mechanical data, obtain the softness-hardness index representing the softness-hardness of the target object; S104. Based on the soft and hard indices and the initial control data, obtain the actual control data used to control the collaborative robot's end effector to operate the target object.
[0020] In the above process, initial control data refers to data obtained through any existing means, which instructs the collaborative robot to control the target object. Pre-contact refers to any form of contact between the robot and the target object, such as the robotic arm of the collaborative robot touching the target object, or the electric flip-top of the collaborative container touching the contents of the container.
[0021] The purpose of initial control data is to allow the robot to pre-contact the object before performing specific actions. During this pre-contact process, the robot judges the object's hardness or softness by sensing changes in visual data (such as images and point clouds) and mechanical data (gripping force, opening and closing torque), thus enabling more precise operations. This can be compared to a human hand gently "squeezing" an object before grasping it to feel its texture and adjust the gripping force and speed to avoid danger and improve efficiency.
[0022] Specifically, in this embodiment, the initial control data can be any type of control data. For example, it can be data obtained through any existing technology (such as machine vision or fixed instruction programs) for specific operations such as grasping and moving objects, or it can be data obtained through any existing technology specifically instructing the robot to touch the target object. Correspondingly, the specific form of pre-contact can also be manually specified. For example, when the initial control data is used to instruct the robot to perform a complete action, the robot can be controlled to operate based on the initial control data first, and the first few seconds of the robot's contact with the target object can be designated as the pre-contact state. After analyzing and obtaining the soft and hard index, the initial control data is corrected to obtain the actual control data, and the actual control data is used to replace the initial control data to continue the operation. When the initial control data is control data specifically used to contact the target object, pre-contact can be performed directly through the initial control data, and after obtaining the actual control data, the complete control action can be re-executed based on the actual control data.
[0023] In this embodiment, the hardness index is used to represent the degree of hardness or softness of an object perceived by the robot. It is understood that the specific representation of the hardness index and the specific process of obtaining actual control data can be flexibly specified based on the actual situation, and are all things that those skilled in the art can understand and conceive of. For example, the hardness or softness of an object can be represented by a value from 0 to 1, with 1 representing the hardest and 0 representing the softest, thereby adjusting the control strategy. When the hardness index is less than 0.5, the target object can be defined as a soft object, such as flexible materials or soft containers. Soft objects are easily deformed but not easily broken, exhibiting high fault tolerance. In this case, the robot can be controlled to increase speed and force within a safe range to improve efficiency, such as increasing the gripping force to increase stability, increasing the gripping speed to avoid object deformation leading to jamming and detachment, and increasing the robotic arm's movement speed to improve efficiency. When the hardness index is greater than 0.5, the target object will be defined as a hard object, such as a glass bottle or a sharp metal object. Hard objects may have problems such as being easily broken, being sharp and easily scratched, or being heavy and affecting the robot's movement trajectory. In this case, the control data needs to be corrected or recalculated with the goal of "zero damage" to achieve more precise adjustments. For example, the original gripping force can be maintained, the movement speed can be reduced, and a closed-loop real-time correction method can be used to reduce the accident rate. Another example is to recalculate the gripping point, change the gripping mode, increase the contact area to avoid local stress concentration, and improve gripping stability.
[0024] This invention overcomes the bottleneck of traditional vision methods' insufficient adaptability to complex materials through a closed-loop control system of "pre-contact - multi-source data perception - soft and hard dynamic recognition - difference strategy correction". Its core value lies in: using multi-robot collaboration to quantify soft and hard attributes, enabling collaborative robots to "perceive materials and adjust their techniques" like humans, thus ensuring the safety of collaborative robots during operations while improving efficiency.
[0025] The following section will provide a more detailed explanation of how to determine the "hardness" of a target object: It is conceivable that objects of varying softness exhibit significant differences in both deformation under force (visual level) and force perception (mechanical level). For example, when a flexible (soft) object is gripped, its shape undergoes noticeable deformation, and its volume may even expand. The mechanical data changes during gripping show a gradual increase and decrease trend, without abrupt force peaks. In contrast, the shape of a rigid object remains largely unchanged, and the gripping force perceived by the robot during gripping exhibits a sudden increase, potentially accompanied by micro-vibrations as the gripping force increases. Therefore, a comprehensive judgment can be made based on changes in both visual and mechanical data. Knowing these principles, those skilled in the art can employ any existing method for softness / hardness identification, such as threshold determination or matching based on a fixed rule base. However, this invention provides a more preferred solution: Combination Figure 2As shown, in a preferred embodiment, the above step: S103, obtaining a hardness index representing the hardness of the target object based on changes in visual and mechanical data, specifically includes: S201. Extract features from changes in visual data to obtain visual dynamic features; S202. Extract features from the changes in mechanical data to obtain dynamic mechanical features; S203. Based on visual dynamic features and mechanical dynamic features, and using a preset soft and hard recognition model, obtain the soft and hard index.
[0026] In the above process, the preset soft and hard recognition model refers to any existing artificial intelligence model (such as machine learning model or deep learning network) that can perform classification and regression. It extracts the features of dynamic deformation at the visual level and the features of dynamic response at the mechanical level. Through the preset soft and hard recognition model, the multi-dimensional dynamic features are nonlinearly fused and pattern learned. It can not only accurately capture the distinction boundary between two types of objects under extreme differences, but also effectively identify the gradual characteristics of objects with intermediate hardness (such as semi-soft plastics and elastic rubber), and output continuous soft and hard indices instead of discrete classification results, which significantly improves the adaptability to complex materials.
[0027] In a more specific embodiment, the visual dynamic features include the point cloud deformation rate of the target object; the mechanical dynamic features include the rise time, peak force, fluctuation standard deviation, number of force mutations, and force response consistency of a single robot; and the soft and hard recognition model is an SVM model.
[0028] The point cloud deformation rate can be obtained by aligning the point cloud captured by the robot to construct a target 3D model in a unified coordinate system, and then tracking the deformation of the 3D model. For dynamic mechanical features, the robot's force-time curve can be recorded (force can be any characteristic that reflects mechanics, such as clamping force, torque, or pressure, depending on the actual robot type), and the rise time, peak force, standard deviation of fluctuation, and number of force mutations can be extracted from the curve. Force response consistency is a parameter set for multi-robot scenarios. When multiple robots operate on the same object, the rise time, peak force, standard deviation of fluctuation, and number of force mutations for each robot can be statistically analyzed, and then the consistency of these features across all robots (such as mean, standard deviation, etc.) can be calculated to obtain force response consistency. The SVM model can use a lightweight computing model suitable for edge computing unit deployment, and a computationally efficient linear kernel can be used as the kernel function. By inputting the above features, the SVM model can calculate the distance (including positive and negative numbers) from the sample to the hyperplane and normalize it to a value between 0 and 1, thereby obtaining the hardness / softness index. Training the SVM model is also quite simple. By using objects with different degrees of softness, conducting pre-contact experiments with a robot or computer simulation, the corresponding sample data can be obtained.
[0029] This embodiment focuses on point cloud deformation rate in terms of vision, directly quantifying the shape changes of an object under force. Mechanically, it integrates the force-time curve characteristics of a single robot with the force response consistency of multiple robots, capturing the mechanical characterization of hardness from both individual response and group collaboration perspectives, resulting in a more comprehensive and robust description of hardness. The lightweight SVM model in this embodiment serves as the recognition core, ensuring real-time requirements and enabling continuous quantification of object hardness, improving discrimination accuracy and simplifying the connection between model output and application control.
[0030] Furthermore, it is conceivable that, among the aforementioned visual dynamic features, the calculation of the dynamic deformation rate of the point cloud depends on the processing of the three-dimensional point cloud sequence, which generally involves multiple steps such as ICP registration, PCA principal component analysis, and volume calculation. It involves a large number of matrix operations, and the processing time per frame is relatively long, which will greatly restrict the real-time control of collaborative robots in practice.
[0031] Therefore, the present invention also provides a preferred embodiment for improving the computation speed of point cloud deformation rate. In this embodiment, step S201, extracting features from the changes in visual data to obtain visual dynamic features, specifically includes: Based on visual data, the rate of change of the target object's outline is obtained; Based on the mechanical data, the contact force at the end of the collaborative robot is obtained; Acquire displacement data of the collaborative robot's end effector; The point cloud deformation rate is obtained based on the contour change rate, contact force, and displacement data.
[0032] The contact force mentioned above refers to the force in the force-time curve mentioned earlier. Images captured by a camera, using edge recognition and other algorithms to identify the contour change rate, can reflect the deformation characteristics of an object to some extent. However, the contour change rate only reflects two-dimensional deformation from a specific viewpoint, and compared to the actual deformation rate of three-dimensional point clouds, it essentially lacks depth information. Therefore, this embodiment utilizes the characteristics of contact force reflecting an object's resistance to deformation and displacement data reflecting the external manifestation of forced deformation. Contact force and displacement data are used to replace the depth information in the three-dimensional point cloud, combined with the intuitive visual deformation index of contour change rate to obtain an approximate point cloud deformation rate. The synergistic effect of these three factors allows for rapid inference of the object's deformation degree at a computationally lower cost than processing three-dimensional point clouds. This not only significantly reduces the computation time for single-frame point cloud deformation rate, meeting the stringent requirements of collaborative robots for real-time updates of control data, but also avoids complex point cloud modeling and registration processes, reducing dependence on computing hardware.
[0033] Specifically, in a preferred embodiment, the above step of obtaining the point cloud deformation rate based on the contour change rate, contact force, and displacement data specifically includes: The depth feature approximation factor is obtained based on the displacement data, the ratio of displacement data to contact force, and the ratio of profile change rate to displacement data. The point cloud deformation rate is obtained based on the contour change rate and the depth feature approximation factor.
[0034] The aforementioned depth feature approximation factor can be obtained in any way. For example, data can be collected through experiments or simulations and regression can be performed to obtain a polynomial for calculating the depth feature approximation factor based on displacement data, the ratio of displacement data to contact force, and the ratio of contour change rate to displacement data. Then, a weighted sum is performed with the contour change rate to finally obtain the point cloud deformation rate.
[0035] It is important to emphasize that the above process is essentially a trade-off between accuracy and speed. Since the ultimate goal of this invention is to quantify the hardness or softness of an object rather than to precisely calculate its deformation, the subsequent process of quantifying the hardness or softness also requires analysis of mechanical changes and comprehensive analysis through a hardness or softness identification model. Therefore, this invention allows for a certain degree of error in calculating the point cloud deformation rate. If the mechanical data acquisition is accurate, even if the point cloud deformation rate is inaccurate, the final calculation result can still meet expectations.
[0036] Furthermore, the above-described process of calculating the hardness index assumes accurate data. However, in reality, in scenarios requiring hardness identification, the target object is likely to be contaminated, thus rendering pre-contact perception ineffective. For example, medical waste is often contaminated with bodily fluids, medications, blood, and other pollutants. Visually, this contamination can lead to blurred camera lenses, errors in recognizing semi-transparent materials, and distortion in deformation rate calculations due to contaminants filling the wrinkles of flexible objects. Mechanically, contaminants can increase contact friction, masking the "gradual rise" characteristic of the force perception curve. Therefore, this invention also provides a preferred embodiment to further eliminate errors caused by contamination of the target object.
[0037] Combination Figure 3 In a preferred embodiment, step S203, obtaining the soft-hard index based on visual dynamic features and mechanical dynamic features and a preset soft-hard recognition model, specifically includes: S301. Input the visual dynamic features and mechanical dynamic features into the soft and hard recognition model to obtain the initial soft and hard index output by the soft and hard recognition model. S302. Input the visual dynamic features and mechanical dynamic features into the preset pollution identification model, and correct the initial soft and hard index based on the pollution identification model to obtain the corrected soft and hard index.
[0038] This embodiment introduces a specialized contamination recognition model to analyze contamination traces in visual and mechanical features, quantifies the degree of interference of contamination on the initial index, and makes targeted corrections to the initial soft and hard indices accordingly. The compensation effect of the contamination model offsets the systematic errors caused by contaminants, enabling the final corrected soft and hard indices to more realistically reflect the actual soft and hard characteristics of the target object in a contaminated state. This provides a reliable decision-making basis for the end-effector control of collaborative robots even in high-contamination-risk scenarios.
[0039] Specifically, the process of eliminating the impact of pollution can be viewed as reducing the weight of abnormal features affected by pollution and increasing the weight of normal features when calculating the final soft and hard index, thereby completing the correction.
[0040] For example, when there is severe obstruction by contaminants, the weight of visual features in the analysis process can be reduced. When there is viscous contaminant adhesion, which increases friction, the analysis weight of visual features can be increased. At the same time, the weight of peak force in mechanical dynamic features can be reduced, and the weight of force response consistency of multiple robots can be increased.
[0041] The present invention also provides a more specific embodiment for implementing the above-described approach: Combination Figure 4 and Figure 5 In a preferred embodiment, the contamination recognition model includes an LSTM neural network layer, a self-attention neural network layer, and a fully connected layer. Step S302, which involves inputting visual dynamic features and mechanical dynamic features into a preset contamination recognition model, and correcting the initial soft-hard index based on the contamination recognition model to obtain the corrected soft-hard index, specifically includes: S401. By splicing visual dynamic features, mechanical dynamic features, and the corresponding initial soft and hard indices, an input vector is obtained, and the input vectors at multiple time points are summarized to obtain an input matrix. S402. Input the input matrix into the LSTM neural network layer to obtain the hidden state vector sequence output by the LSTM neural network layer; S403. Input the hidden state vector sequence into the self-attention neural network layer to obtain the weighted aggregate vector output by the self-attention neural network layer; S404. Input the weighted aggregation vector into the fully connected layer to obtain the corrected soft and hard indices of the fully connected layer output.
[0042] In this embodiment, visual dynamic features, mechanical dynamic features, and an initial soft / hard index are concatenated into an input vector, and multiple time points are aggregated to form an input matrix. This allows the LSTM neural network layer to capture the temporal dependencies of various features (such as changes in visual features under pollutant coverage and the dynamic pattern of gradual frictional changes under pollutant adhesion), outputting a sequence of hidden state vectors containing temporal information. It is worth noting that this embodiment incorporates an initial soft / hard index into the input data, enabling the entire model to explicitly learn the correction magnitude for the "pollution-induced misjudgment" problem during training, thereby improving accuracy.
[0043] Subsequently, a self-attention neural network layer automatically focuses on the most critical time step and feature dimension for judging the degree of contamination (such as assigning high weights to feature anomalies caused by sudden changes in contamination or specific types of contamination), suppressing interference from irrelevant temporal information, and amplifying the correction signals of these time steps or feature data through high attention weights. Finally, a fully connected layer maps the weighted aggregation vector to the corrected soft and hard exponents, thereby achieving quantitative compensation for the contamination error in the initial exponent.
[0044] In this embodiment, LSTM is used to capture temporal dynamics, a self-attention mechanism is used to focus and filter key signals, and a fully connected layer is used to complete nonlinear mapping, forming a complete correction technology chain. This can effectively identify interference with perception features under complex pollution modes and significantly improve the accuracy of software and hardware recognition in dynamic pollution scenarios.
[0045] The overall model architecture of the above-mentioned hardware and software recognition is as follows: Figure 5 As shown. In practice, the aforementioned soft and hard identification models and contamination identification models can be combined as a whole. During training, different pre-contact data of the same object before and after contamination can be obtained through experiments or computer simulations, and used as input sample data for the soft and hard identification models and the contamination identification models, respectively. It is understood that the aforementioned LSTM neural network layers and self-attention neural network layers are existing technologies that can be learned and understood by those skilled in the art, therefore, this paper will not elaborate on the details.
[0046] Combination Figure 6 As shown, the present invention also provides a vision-based collaborative robot end effector control system, comprising: The pre-contact module 610 is used to acquire initial control data, which is used to control the end effector of the collaborative robot to make pre-contact with the target object. The data monitoring module 620 is used to continuously collect visual data of the target object and mechanical data of the collaborative robot end when the end of the robot makes pre-contact with the target object. The soft-hardness recognition module 630 is used to obtain a soft-hardness index representing the softness or hardness of a target object based on changes in visual and mechanical data. The fine-tuning module 640 is used to obtain actual control data for controlling the collaborative robot's end effector to operate on the target object, based on the soft and hard indices and the initial control data.
[0047] It should be noted that the corresponding systems provided in the above embodiments are computer program products that can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding content in the above method embodiments, and will not be repeated here.
[0048] The present invention also provides an electronic device, comprising: Memory and processor; The memory is used to store the program, and the processor is used to execute the steps in any of the vision-based collaborative robot end-effector control methods described above when the program is executed.
[0049] The present invention also provides a computer-readable storage medium for storing a computer-readable program or instruction, which, when executed by a processor, can implement the steps in any of the above-described vision-based collaborative robot end-effector control methods.
[0050] This invention provides a vision-based collaborative robot end-effector control method. It first acquires initial control data for controlling the collaborative robot's end-effector to make pre-contact with a target object. During this pre-contact process, it continuously collects visual data of the target object and mechanical data of the robot's end-effector. Based on changes in the visual and mechanical data, a hardness index representing the target object's softness or hardness is obtained. Finally, based on the hardness index and the initial control data, actual control data for controlling the collaborative robot's end-effector to manipulate the target object is obtained. Compared to existing technologies, this invention achieves real-time, accurate perception and adaptive control of the target object's material properties through a closed-loop architecture of "pre-contact - dynamic perception - hardness quantification - data correction." Specifically, it captures the dynamic response characteristics of the target object under force through pre-contact data fusion and multi-source data fusion, and then quantifies and outputs continuous soft and hard indices. Finally, it dynamically corrects the initial control data so that the actual control data can match the real physical characteristics of the target object. This overcomes the limitation of traditional methods that can only make qualitative judgments on complex materials, and significantly improves the operational safety, reliability and control accuracy of collaborative robots when facing complex materials and shapes. It also solves the problem of insufficient adaptability of existing vision control methods when facing objects with complex materials and shapes.
[0051] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0052] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A vision-based end-effector control method for collaborative robots, characterized in that, include: Acquire initial control data, which is used to control the collaborative robot's end effector to make pre-contact with the target object; During the pre-contact between the collaborative robot's end effector and the target object, visual data of the target object and mechanical data of the collaborative robot's end effector are continuously collected. Based on changes in visual and mechanical data, a softness-hardness index is obtained to represent the softness or hardness of the target object. Based on the soft and hard indices and the initial control data, the actual control data for controlling the collaborative robot's end effector to manipulate the target object is obtained.
2. The vision-based collaborative robot end-effector control method according to claim 1, characterized in that, Based on changes in visual and mechanical data, a hardness index representing the softness or hardness of a target object is obtained, including: Feature extraction is performed on changes in visual data to obtain visual dynamic features; Feature extraction is performed on changes in mechanical data to obtain dynamic mechanical features; Based on visual dynamic features and mechanical dynamic features, and using a pre-defined soft and hard recognition model, a soft and hard index is obtained.
3. The vision-based collaborative robot end-effector control method according to claim 2, characterized in that, Visual dynamic features include the point cloud deformation rate of the target object; mechanical dynamic features include the rise time, peak force, fluctuation standard deviation, number of force mutations, and force response consistency of a single robot. The soft and hard recognition model is an SVM model.
4. The vision-based collaborative robot end-effector control method according to claim 2, characterized in that, Visual dynamic features include the point cloud deformation rate of the target object; feature extraction is performed on changes in visual data to obtain visual dynamic features, including: Based on visual data, the rate of change of the target object's outline is obtained; Based on the mechanical data, the contact force at the end of the collaborative robot is obtained; Acquire displacement data of the collaborative robot's end effector; The point cloud deformation rate is obtained based on the contour change rate, contact force, and displacement data.
5. The vision-based collaborative robot end-effector control method according to claim 4, characterized in that, The point cloud deformation rate is obtained based on the contour change rate, contact force, and displacement data, including: The depth feature approximation factor is obtained based on the displacement data, the ratio of displacement data to contact force, and the ratio of profile change rate to displacement data. The point cloud deformation rate is obtained based on the contour change rate and the depth feature approximation factor.
6. The vision-based collaborative robot end-effector control method according to claim 2, characterized in that, Based on visual dynamic features and mechanical dynamic features, and using a pre-defined soft and hard recognition model, a soft and hard index is obtained, including: Visual dynamic features and mechanical dynamic features are input into the soft and hard recognition model to obtain the initial soft and hard index output by the soft and hard recognition model. The visual dynamic features and mechanical dynamic features are input into the preset pollution identification model, and the initial soft and hard index is corrected based on the pollution identification model to obtain the corrected soft and hard index.
7. The vision-based collaborative robot end-effector control method according to claim 6, characterized in that, The contamination recognition model includes an LSTM neural network layer, a self-attention neural network layer, and a fully connected layer. Visual dynamic features and mechanical dynamic features are input into the pre-defined contamination recognition model, and the initial soft / hard index is corrected based on the contamination recognition model to obtain the corrected soft / hard index, including: The visual dynamic features, mechanical dynamic features, and corresponding initial soft and hard indices are spliced together to obtain the input vector, and the input vectors at multiple time points are aggregated to obtain the input matrix; The input matrix is fed into the LSTM neural network layer to obtain the hidden state vector sequence output by the LSTM neural network layer; The hidden state vector sequence is input into the self-attention neural network layer to obtain the weighted aggregate vector output by the self-attention neural network layer; The weighted aggregation vector is input into the fully connected layer to obtain the corrected soft and hard exponents of the fully connected layer output.
8. A vision-based collaborative robot end-effector control system, characterized in that, include: The pre-contact module is used to acquire initial control data, which is used to control the collaborative robot's end effector to make pre-contact with the target object. The data monitoring module is used to continuously collect visual data of the target object and mechanical data of the collaborative robot's end effector when the end effector makes pre-contact with the target object. The soft-hardness recognition module is used to obtain a soft-hardness index representing the softness or hardness of a target object based on changes in visual and mechanical data. The fine-tuning module is used to obtain the actual control data for controlling the collaborative robot's end effector to manipulate the target object, based on the soft and hard indices and the initial control data.
9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store the program, and the processor is used to execute the steps of any one of the vision-based collaborative robot end-effector control methods of claims 1-7 when the program is executed.
10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, are capable of implementing the steps in any one of the vision-based collaborative robot end-effector control methods of claims 1-7.
