Visual servoing alignment method, apparatus, system, electronic device, and program product

Abstract

The application discloses a visual servo alignment method, device, system, electronic equipment and program product, and belongs to the technical field of industrial detection. The visual servo alignment method comprises the following steps: based on position information of a first workpiece, a target robot is controlled to drive the first workpiece to move, so that a relative position deviation between the first workpiece and a preset reference region satisfies a first preset condition; based on shape features of the first workpiece, the target robot is controlled to drive the first workpiece to rotate, so that a form of the first workpiece converges to satisfy a target form condition; based on a deviation between position information of a second workpiece and the position information of the first workpiece, the target robot is controlled to drive the first workpiece to move, so that a relative position deviation between the first workpiece and a target region of the second workpiece satisfies a second preset condition; and the target robot is controlled to drive the first workpiece to perform a contact action with the second workpiece. The visual servo alignment method improves alignment accuracy and guarantees the determinacy and stability of control.

Description

Technical Field

[0001] This application belongs to the field of industrial inspection technology, and in particular relates to a visual servo alignment method, apparatus, system, electronic device and program product. Background Technology

[0002] In automated precision assembly scenarios, it is often necessary to align flexible, slender targets with specific areas on target workpieces with high precision and complete the contact operation. Related technologies are typically automated alignment solutions based on visual feedback, such as using a camera to identify the target's position and orientation, and employing visual servo control to drive a robot to complete the alignment. However, these solutions still have many shortcomings in practical applications; their control stability, alignment accuracy, and operation success rate are insufficient to meet the increasingly stringent requirements of industrial production. Summary of the Invention

[0003] This invention aims to solve at least one of the technical problems existing in the prior art. To this end, this invention proposes a visual servo alignment method, device, system, electronic device, and program product, which realizes multi-degree-of-freedom decoupled control, solves the problem of angular feature failure when approaching the target area by utilizing shape features, and improves the final alignment accuracy by secondary fine alignment based on the second workpiece target area, thus ensuring the determinism and stability of the control.

[0004] In a first aspect, this application provides a visual servo alignment method, including: Based on the position information of the first workpiece, the target robot is controlled to move the first workpiece so that the relative position deviation between the first workpiece and the preset reference area meets the first preset condition. Based on the shape features of the first workpiece, the target robot is controlled to rotate the first workpiece so that the shape of the first workpiece converges to meet the target shape conditions. Based on the deviation between the position information of the second workpiece and the position information of the first workpiece, the target robot is controlled to drive the first workpiece to move, so that the relative positional deviation between the target areas of the first workpiece and the second workpiece meets the second preset condition. The target robot is controlled to drive the first workpiece to perform a contact action with the second workpiece.

[0005] According to the visual servo alignment method provided in the embodiments of this application, by decomposing the visual servo alignment into sequentially executed coarse position alignment (moving towards a preset reference area with the position of the first workpiece as feedback), shape convergence alignment (rotating with the shape features of the first workpiece as feedback), and fine position alignment (moving towards the target area with the deviation between the position of the second workpiece and the position of the first workpiece as feedback), multi-degree-of-freedom decoupled control is achieved. Furthermore, the problem of angle feature failure when approaching the target area is solved by utilizing shape features. The secondary fine alignment based on the target area of ​​the second workpiece improves the final alignment accuracy and ensures the determinism and stability of the control.

[0006] One embodiment of the visual servo alignment method of this application obtains the shape features based on the following steps: Obtain the binary mask image of the first workpiece; Calculate the second-order central moments of the binary mask image; A normalized shape descriptor is determined based on the second-order central moment. The normalized shape descriptor is used to characterize the degree to which the first workpiece continuously changes from a first shape to a second shape in the image.

[0007] One embodiment of the visual servo alignment method of this application, wherein the first shape is an elongated line segment shape and the second shape is a dot shape; the step of controlling the target robot to rotate the first workpiece so that the shape of the first workpiece converges to meet the target shape condition includes: By making the normalized shape descriptor approach the target value, the projected shape of the first workpiece is driven to shrink from the elongated line segment shape to the dot shape. When the projected shape of the first workpiece shrinks to the dot shape, it is determined that the shape of the first workpiece has converged.

[0008] One embodiment of the visual servo alignment method of this application, wherein controlling a target robot to move the first workpiece based on the position information of the first workpiece includes: During the process of controlling the target robot to move the first workpiece, an attitude maintenance component is superimposed, which is generated based on the deviation between the current shape features and the target shape features of the first workpiece.

[0009] One embodiment of the visual servo alignment method of this application, wherein controlling the target robot to drive the first workpiece to perform a contact action with the second workpiece includes: Based on a first control frequency, the joint target command is smoothed and filtered to control the target robot to perform a contact action. The first control frequency is higher than the second control frequency that controls the movement of the target robot based on the position information of the first workpiece.

[0010] One embodiment of the visual servo alignment method of this application, after controlling the target robot to drive the first workpiece to perform a contact action with the second workpiece, the method includes: If the first workpiece fails to make contact with the second workpiece, the target robot is controlled to move the first workpiece back and the steps of "based on the deviation between the position information of the second workpiece and the position information of the first workpiece, the target robot is controlled to move the first workpiece so that the relative position deviation between the target areas of the first workpiece and the second workpiece meets the second preset condition; the target robot is controlled to move the first workpiece to make contact with the second workpiece" are executed again until the first workpiece successfully makes contact with the second workpiece.

[0011] Secondly, this application provides a visual servo alignment device, comprising: The first processing module is used to control the target robot to move the first workpiece based on the position information of the first workpiece, so that the relative position deviation between the first workpiece and the preset reference area meets the first preset condition. The second processing module is used to control the target robot to rotate the first workpiece based on the shape features of the first workpiece, so that the shape of the first workpiece converges to meet the target shape conditions. The third processing module is used to control the target robot to move the first workpiece based on the deviation between the position information of the second workpiece and the position information of the first workpiece, so that the relative position deviation between the target areas of the first workpiece and the second workpiece meets the second preset condition. The fourth processing module is used to control the target robot to drive the first workpiece to perform a contact action with the second workpiece.

[0012] According to the visual servo alignment device provided in the embodiments of this application, by decomposing the visual servo alignment into sequentially executed coarse position alignment (moving towards a preset reference area with the position of the first workpiece as feedback), shape convergence alignment (rotating with the shape features of the first workpiece as feedback), and fine position alignment (moving towards the target area with the deviation between the position of the second workpiece and the position of the first workpiece as feedback), multi-degree-of-freedom decoupled control is realized. Furthermore, the problem of angle feature failure when approaching the target area is solved by utilizing shape features. The secondary fine alignment based on the target area of ​​the second workpiece improves the final alignment accuracy and ensures the determinism and stability of the control.

[0013] Thirdly, this application provides a visual servo alignment system, comprising: An image acquisition unit is used to acquire images of the first workpiece and the second workpiece. The feature extraction unit is used to extract the position information and shape features of the first workpiece, and the position information of the second workpiece from the image; A controller, connected to the image acquisition unit, the feature extraction unit, and the target robot, is used to execute the visual servo alignment method as described in the first aspect to control the target robot.

[0014] According to the visual servo alignment system provided in the embodiments of this application, by decomposing the visual servo alignment into sequentially executed coarse position alignment (moving towards a preset reference area with the position of the first workpiece as feedback), shape convergence alignment (rotating with the shape features of the first workpiece as feedback), and fine position alignment (moving towards the target area with the deviation between the position of the second workpiece and the position of the first workpiece as feedback), multi-degree-of-freedom decoupled control is achieved. Furthermore, the problem of angle feature failure when approaching the target area is solved by utilizing shape features. The secondary fine alignment based on the target area of ​​the second workpiece improves the final alignment accuracy and ensures the determinism and stability of the control.

[0015] Fourthly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the visual servo alignment method as described in the first aspect above.

[0016] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the visual servo alignment method as described in the first aspect above.

[0017] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects: By decomposing visual servo alignment into sequentially executed coarse position alignment (moving towards a preset reference area with the position of the first workpiece as feedback), shape convergence alignment (rotating with the shape features of the first workpiece as feedback), and fine position alignment (moving towards the target area with the deviation between the position of the second workpiece and the position of the first workpiece as feedback), multi-degree-of-freedom decoupled control is achieved. Furthermore, the problem of angle feature failure when approaching the target area is solved by utilizing shape features. The secondary fine alignment based on the target area of ​​the second workpiece improves the final alignment accuracy and ensures the determinism and stability of the control.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0019] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a schematic flowchart of the visual servo alignment method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the visual servo alignment device provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0021] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0022] The visual servo alignment method, visual servo alignment device, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0023] The visual servo alignment method can be applied to a terminal, and can be executed by hardware or software in the terminal.

[0024] The visual servo alignment method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the visual servo alignment method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The visual servo alignment method provided in this application embodiment will be described below using an electronic device as the execution subject as an example.

[0025] like Figure 1 As shown, the visual servo alignment method includes steps 110, 120, 130 and 140.

[0026] Step 110: Based on the position information of the first workpiece, control the target robot to drive the first workpiece to move, so that the relative position deviation between the first workpiece and the preset reference area meets the first preset condition. In this step, the first workpiece is the target object that is gripped and moved by the target robot. The first workpiece has the characteristics of flexibility and slenderness, and is prone to deformation during movement.

[0027] The specific types of the first workpiece include, but are not limited to: flexible cables, flexible hose ends, flexible ribbon cables, strip material ends, medical guide wires, optical fiber ends, cosmetic tools (such as lipsticks, powder puffs, and brush heads), grafting scion stems, microinjection needle tips, etc. Any flexible, slender object that needs to be clamped and moved for alignment operations can be used as the first workpiece.

[0028] Position information refers to the coordinates of the first or second workpiece in the image. This position information can be obtained by acquiring and processing images using a vision system. For example, the position information of the first workpiece can be obtained by calculating its center of gravity coordinates, and the position information of the second workpiece can be obtained by detecting the center coordinates of the target area of ​​the second workpiece. This position information is used to calculate the relative position deviation, serving as feedback for translation control.

[0029] A target robot refers to a robotic device that grips and moves a first workpiece. A target robot can be a single-arm robot, one arm of a dual-arm robot, a SCARA robot, a parallel robot, a collaborative robot, or any automated device capable of gripping a workpiece and performing translational and rotational movements. The target robot can respond to control commands to move the first workpiece in translational and rotational motion in space.

[0030] A preset reference area refers to a pre-defined fixed reference position or region used as the position control target in the first control stage. The preset reference area can be the image center, a specific coordinate point in the image, or any pre-determined reference position that does not change with the position of the second workpiece. The preset reference area is used in the first stage to move the first workpiece to near a stable reference position, preventing control divergence caused by excessive initial position deviation of the second workpiece.

[0031] The first preset condition refers to the condition for the completion of the first control stage. The first preset condition can be that the relative positional deviation between the first workpiece and the preset reference area is less than a preset threshold. For example, a horizontal deviation threshold and a vertical deviation threshold can be set. The first preset condition is determined to be met when the absolute value of the difference between the abscissa of the center of gravity of the first workpiece and the abscissa of the center point of the preset reference area is less than the horizontal deviation threshold, and the absolute value of the difference between the ordinate of the center of gravity of the first workpiece and the ordinate of the center point of the preset reference area is less than the vertical deviation threshold. The specific values ​​of the thresholds can be set according to the accuracy requirements of the actual application scenario. Once the first preset condition is met, the system switches from the first control stage to the second control stage.

[0032] Step 120: Based on the shape features of the first workpiece, control the target robot to rotate the first workpiece so that the shape of the first workpiece converges to meet the target shape conditions. In this step, shape features are feature quantities that characterize the morphology of the first workpiece in the image. Shape features can describe the degree to which the first workpiece continuously changes from a thin line segment shape to a dot shape.

[0033] The target shape condition refers to the shape condition that the first workpiece must meet when its posture is aligned. When the first workpiece is aligned with the target area in the correct posture, its projection in the image should appear as a dot (or near-dot). When a slender target is aligned with the target area along the axial direction, its projection area is minimized, and its shape is closest to a circle. At this time, the shape feature should approach the target value. The target shape condition is the control target of the second control stage.

[0034] Step 130: Based on the deviation between the position information of the second workpiece and the position information of the first workpiece, control the target robot to drive the first workpiece to move, so that the relative position deviation between the target areas of the first workpiece and the second workpiece meets the second preset condition. In this step, the second workpiece is a workpiece with a target area that needs to be aligned and in contact with the first workpiece. Specific types of the second workpiece include, but are not limited to: connectors (with sockets), circuit boards (with mounting holes), positioning fixtures (with guide grooves), sleeves (with inlets), ceramic ferrules (with inner holes), fiber optic connectors (with guide holes), biological tissue (with target injection points), and facial models (with target makeup areas). Any object with a target area that needs to be aligned with the first workpiece can be used as the second workpiece.

[0035] The second preset condition is the condition for the completion of the third control stage. The second preset condition can be that the relative positional deviation between the target areas of the first and second workpieces is less than a preset threshold. For example, a horizontal deviation threshold can be set; when the absolute value of the difference between the x-coordinate of the center of gravity of the first workpiece and the x-coordinate of the center point of the target area is less than the horizontal deviation threshold, the second preset condition is determined to be met. In scenarios where vertical deviation needs to be controlled simultaneously, a vertical deviation threshold can also be set. Once the second preset condition is met, the system enters the contact action execution stage.

[0036] Step 140: Control the target robot to drive the first workpiece to perform a contact action with the second workpiece.

[0037] In this step, the contact action is the final operational action between the first workpiece and the target area of ​​the second workpiece. The specific type of contact action depends on the application scenario. For example: in a plug-in assembly scenario, the contact action is an insertion action (inserting the first workpiece into the target area); in a makeup operation scenario, the contact action is an application action (applying makeup tools to the target area) or a pressing action (pressing a powder puff onto the skin); in a liquid spraying scenario, the contact action is a spraying action (spraying liquid onto the target surface); in a medical injection scenario, the contact action is a puncture action (inserting a needle tip into the target tissue). This application is not limited to the above action types; any operation requiring contact between the first workpiece and the target area is applicable.

[0038] In actual execution, the first stage (position control stage) uses the position information of the first workpiece as feedback to control the target robot to move the first workpiece in a translational motion. The goal of this stage is to reduce the relative positional deviation between the first workpiece and the preset reference area, so that the first workpiece moves to the vicinity of the preset reference area. It should be noted that the control target in this stage is the preset reference area (such as the image center), not the target area of ​​the second workpiece. In the initial stage, the target area of ​​the second workpiece may not have been accurately identified, or its position may be far from the preset reference area. Directly targeting the second workpiece may lead to control divergence. Therefore, the first stage adopts a conservative strategy, first moving the first workpiece to a stable area near the preset reference area. This stage does not involve attitude adjustment (or only performs minimal attitude maintenance) to avoid coupling interference between translation and rotation.

[0039] The second stage (shape control stage): After the first control stage is completed, the system switches to the second stage. This stage uses the shape features of the first workpiece as feedback to control the target robot to rotate the first workpiece. The goal of this stage is to converge the shape of the first workpiece to the target shape condition. For slender, flexible targets, when their projection in the image is a thin line segment, it means their posture is not yet aligned; when the projection shrinks to a dot, it means their axial direction is aligned with the observation direction (i.e., the orientation of the target area). Therefore, by monitoring changes in shape features, it is possible to indirectly determine whether the posture is aligned.

[0040] The third stage (precision alignment stage): After the second control stage is completed, the system switches to the third stage. In this stage, the deviation between the position information of the second workpiece and the first workpiece is used as feedback to control the target robot to move the first workpiece in translational motion. The goal of this stage is to reduce the relative positional deviation between the target areas of the first and second workpieces to a second preset condition. The control objective of the third stage is the actual position of the target area of ​​the second workpiece, which is dynamically updated as the image coordinates of the second workpiece change. This eliminates alignment errors caused by changes in camera viewpoint, calibration errors, or workpiece position offsets.

[0041] The control target in the first stage is a fixed, preset reference area (such as the image center), while the actual target area may not be at that location. If the target area of ​​the second workpiece is directly used as the target in the first stage, the control may diverge due to excessive initial deviation. However, by adopting a staged strategy of first coarse alignment (fixed reference area), then attitude adjustment, and then fine alignment (dynamic target area), both convergence stability and final accuracy can be ensured.

[0042] Fourth stage (contact action execution stage): After the third stage is completed, the target robot is controlled to drive the first workpiece to perform a contact action with the second workpiece.

[0043] According to the visual servo alignment method provided in the embodiments of this application, by decomposing the visual servo alignment into sequentially executed coarse position alignment (moving towards a preset reference area with the position of the first workpiece as feedback), shape convergence alignment (rotating with the shape features of the first workpiece as feedback), and fine position alignment (moving towards the target area with the deviation between the position of the second workpiece and the position of the first workpiece as feedback), multi-degree-of-freedom decoupled control is achieved. Furthermore, the problem of angle feature failure when approaching the target area is solved by utilizing shape features. The secondary fine alignment based on the target area of ​​the second workpiece improves the final alignment accuracy and ensures the determinism and stability of the control.

[0044] In some embodiments, shape features can be obtained based on the following steps: Obtain the binary mask image of the first workpiece; Calculate the second central moments of the binary mask image; The normalized shape descriptor is determined based on the second-order central moment. The normalized shape descriptor is used to characterize the degree to which the first workpiece changes continuously from the first shape to the second shape in the image.

[0045] In this embodiment, a binary mask image refers to a binarized image in which pixels in the region where the first workpiece is located in the original image are marked as 1 and pixels in the background region are marked as 0. Binary mask image middle, This indicates that the pixel belongs to the first workpiece. This indicates that the pixel belongs to the background.

[0046] The second-order central moment refers to the second-order statistic calculated based on a binary mask image. The specific calculation process is as follows: First, calculate the ordinary moments... The expression sums over all pixels in the image, where W is the width of the binary mask image, H is the height of the binary mask image, u is the column coordinate of the pixel, and v is the row coordinate of the pixel. For a binary mask image in coordinates The value at that location (0 or 1). The current pixel u coordinate is raised to the power of i. Let i be the j-th power of the current pixel v coordinate, where i and j are the orders of the moments (non-negative integers, such as 0, 1, 2). Then, calculate the centroid coordinates. ,in The zero-order moment is the total number of foreground pixels, and the area (number of pixels) of the first workpiece is the area of ​​the first workpiece. It is a first-order horizontal moment. It is a first-order perpendicular moment. The column coordinates (horizontal position) represent the centroid. Let be the row coordinates (vertical position) of the centroid, with the subscript 'c' being the abbreviation for centroid. Then calculate the second-order central moment: , , ,in, Let be the second-order central moment about the u-axis, used to describe the degree of dispersion in the horizontal direction. Let be the second-order central moment about the v-axis, used to describe the degree of dispersion in the vertical direction. The mixed second-order central moments are used to describe the correlation between the horizontal and vertical directions. Finally, the second-order central moments are normalized to obtain... , , , To normalize the second-order central moments at the level, To normalize the second-order vertical central moment, The normalized mixed second-order central moments are used. The second-order central moments describe the distribution characteristics of the first workpiece in the image, including its dispersion in the horizontal and vertical directions and the correlation between the two.

[0047] A normalized shape descriptor is a feature quantity constructed based on the normalized second-order central moments to describe the shape of an object. and The calculation formula is: , , The difference between horizontal and vertical distribution (a value greater than 0 indicates a bias towards horizontal length, and a value less than 0 indicates a bias towards vertical length). The correlation strength (a large absolute value indicates a significant skew). The total distributed energy is used for normalization. and It can characterize the anisotropy and orientation information of an object.

[0048] Furthermore, the degree of anisotropy can be calculated. , among which, , Time indicates isotropy. The time indicates complete anisotropy.

[0049] In some embodiments, during the second control phase (attitude alignment phase). This can be used to determine whether the first workpiece has completed attitude convergence. When When the projection of the first workpiece falls below the preset threshold, it indicates that the projection has shrunk from a thin line segment to a shape close to a dot. The system can then determine that the posture alignment is complete and trigger the switch to the third control stage (secondary alignment stage).

[0050] During the rotation, if A continuous increase instead of a decrease indicates that the rotation direction may be incorrect (the thread ends become increasingly horizontal as it rotates). This can be combined with... The error trend is corrected in direction or a protection mechanism is triggered.

[0051] And calculate the direction angle ,in, The principal direction angle indicates the direction of the workpiece's longest axis. For the four quadrant arctangent function, return , Twice the covariance term, This represents the dispersion difference term.

[0052] In some embodiments, before entering the second control phase, the system determines the current... The deviation from the target orientation angle determines the initial rotation direction, which can reduce blind searching; when the initial attitude deviation is large and When approaching 1 (the projection of the line end is a clear line segment), it can be directly... Error-driven rotation enables rapid coarse alignment. When When the projection decreases and degenerates to a near-circular point, the control strategy switches to a method based on... Precise control as the main focus, avoiding The definition of failure occurs during the dot phase.

[0053] The first form refers to the shape of the first workpiece when it is far from the target area, i.e., a slender line segment. In this form, the projection of the first workpiece has a large aspect ratio. and The absolute value is relatively large.

[0054] The second form refers to the shape of the first workpiece when it approaches the target area and is aligned, i.e., the dot shape. In this form, the projection of the first workpiece is approximately circular. and Approaching 0.

[0055] In actual execution, firstly, a binary mask image of the first workpiece is acquired to separate it from the background. Then, the second-order central moment of the binary mask image is calculated. The second-order central moment describes the distribution characteristics of the first workpiece in the image, including its dispersion in the horizontal and vertical directions and the correlation between them. Finally, a normalized shape descriptor is constructed based on the second-order central moment.

[0056] By determining the shape features of the first workpiece using a normalized shape descriptor, the evolution process of the first workpiece from a slender line segment to a dot shape can be continuously characterized, even when the projection is close to a dot. and The changes remain continuous, without any jumps or loss of physical meaning, and exhibit strong stability and robustness even when approaching the target region.

[0057] In some embodiments, the first shape is a thin line segment shape, and the second shape is a dot shape; controlling the target robot to rotate the first workpiece so that the shape of the first workpiece converges to meet the target shape condition may include: By making the normalized shape descriptor approach the target value, the projected shape of the first workpiece is driven to shrink from a thin line segment shape to a dot shape. When the projected shape of the first workpiece shrinks to a dot shape, it is determined that the shape of the first workpiece has converged.

[0058] In this embodiment, the elongated line segment shape refers to the projected shape of the first workpiece when it is far from the target area. In this shape, the first workpiece appears as an elongated line segment in the image, with a clear directionality.

[0059] The dot shape refers to the projected shape of the first workpiece when its posture is aligned with and close to the target area. In this shape, the first workpiece appears as an approximately circular dot in the image, with an aspect ratio close to 1.

[0060] The target value of the normalized shape descriptor is the value it should achieve when the projected shape of the first workpiece shrinks to a dot shape. When the projection is a perfect circle... , In practical applications, a threshold close to 0 can be set as the target value.

[0061] The projection shape shrinkage refers to the process by which, as the first workpiece rotates to the correct orientation, its projection in the image gradually changes from a long, thin line segment to a short line segment, then to an ellipse, and finally to a dot. This process corresponds to the process by which the normalized shape descriptor gradually approaches 0 from a large value.

[0062] When the first workpiece is correctly aligned with the target area, its projection in the image should be a dot; when the first workpiece is incorrectly aligned, its projection in the image should be a thin line segment. Based on this principle, the target robot is controlled to rotate the first workpiece, and the normalized shape descriptor is monitored in real time. When the rotation direction is correct, the normalized shape descriptor will gradually approach the target value (such as 0), and the projection shape will correspondingly shrink from a thin line segment to a dot; when the rotation direction is opposite to the expected direction, the error trend of the normalized shape descriptor shows divergent characteristics, and the system triggers adaptive correction of the rotation direction accordingly.

[0063] The goal of the second control stage is to make the normalized shape descriptor approach the target value through continuous rotation. When the normalized shape descriptor is close enough to the target value and stable, it can be determined that the projected shape of the first workpiece has shrunk to a dot shape, and thus the orientation of the first workpiece is determined to be aligned. At this time, the axial direction of the first workpiece is consistent with the orientation of the target area, creating good initial conditions for subsequent fine alignment and contact actions.

[0064] In this application, the attitude alignment problem is transformed into a shape feature convergence problem, which avoids the control instability caused by the discontinuity of angle features in traditional methods; it utilizes the inherent shape degradation law of flexible and slender targets, eliminating the need for additional sensors or calibration; the convergence judgment conditions are clear, facilitating automated stage switching; it is particularly suitable for scenarios where the projection of the first workpiece becomes a dot when it approaches the target area, and traditional slope angles cannot be used.

[0065] In some embodiments, controlling the target robot to move the first workpiece based on the position information of the first workpiece may include: During the process of controlling the target robot to drive the first workpiece to move, an attitude maintenance component is superimposed. The attitude maintenance component is generated based on the deviation between the current shape features of the first workpiece and the target shape features.

[0066] In this embodiment, the attitude maintenance component refers to an additional control component superimposed on the translation control command, used to suppress the coupling effect of translational motion on attitude error. Due to the kinematics of the robotic arm and the characteristics of the flexible workpiece, there is coupling between translational and rotational motions: when the target robot performs translational motion, it may cause a change in the attitude of the first workpiece, thereby disrupting the attitude that has already converged in the second control stage. The role of the attitude maintenance component is to actively maintain attitude stability during translation.

[0067] Current shape feature refers to the shape feature value of the first workpiece acquired at the current moment, such as the current shape feature value. value.

[0068] Target shape features refer to the target values ​​that shape features should achieve under target morphological conditions, for example... In some embodiments, it can be set This corresponds to the ideal state when the projection of the first workpiece is a dot shape.

[0069] Deviation refers to the difference between the current shape feature and the target shape feature, for example... .

[0070] In the first control phase (position control phase), the target robot mainly performs translational movements to reduce positional deviations. However, translational movements may interfere with the already adjusted posture, leading to increased posture errors.

[0071] An attitude maintenance component is superimposed on the translation control command. The magnitude of this component is related to the deviation between the current shape feature and the target shape feature, and can be specifically expressed as: ,in, This is the shape descriptor for the current thread end. Expectations under the target posture value, To maintain the gain coefficient (the proportional coefficient that maps attitude deviation to rotation correction), the gain coefficient is kept a positive constant. The value of the gain coefficient can be smaller than the rotation gain during the attitude alignment stage to ensure the dominance of translation control while suppressing the coupling disturbance of translation motion to attitude. The attitude maintenance rotation command (i.e., attitude maintenance component) is superimposed on the translation control.

[0072] When the attitude deviation is zero (i.e.) When the attitude maintenance component is zero, translation control is unaffected. When the attitude deviation is non-zero, the attitude maintenance component generates a small rotation command to actively compensate for the coupled disturbance of translational motion on the attitude, thus maintaining the attitude in a convergent state.

[0073] In this way, the coupling between the first control stage and the second control stage is effectively suppressed, ensuring the independence of control between stages and improving the stability of the entire system.

[0074] In some embodiments, controlling the target robot to drive the first workpiece to perform a contact action with the second workpiece may include: Based on the first control frequency, the joint target command is smoothed and filtered to control the target robot to perform contact action. The first control frequency is higher than the second control frequency that controls the movement of the target robot based on the position information of the first workpiece.

[0075] In this embodiment, the first control frequency refers to the control frequency during the execution of the contact action. In some embodiments, the first control frequency can be 1 kHz, that is, a control cycle is executed once every 0.001 seconds. High-frequency control enables more precise and smoother joint movements.

[0076] The second control frequency refers to the control frequency used during the execution of the first and second stages (position control and shape control). The second control frequency can be equal to the output frequency of the vision sensor (e.g., 30-60Hz) because control decisions in this stage primarily rely on visual feedback. The second control frequency is lower than the first control frequency.

[0077] Joint target commands refer to the desired position, velocity, or torque commands sent to each joint of the target robot. Joint target commands are obtained by performing inverse kinematics solutions on the control variables generated by the vision servoing decision layer.

[0078] Smoothing filtering refers to filtering joint target commands to eliminate abrupt changes in the commands and make joint movements smoother. In some embodiments, a progressive target update mechanism (interpolation control with amplitude limiting) and a high-frequency critical damping smoothing mechanism (second-order critical damping filtering) can be used. The damping coefficient of the critical damping filter... This ensures that the system neither overshoots nor oscillates during the response process, achieving the optimal smooth transition of joint targets.

[0079] In this application, a first control frequency higher than the conventional control frequency (e.g., 1kHz) is used during the contact action execution phase, enabling sub-millisecond real-time response to meet the requirements of precise contact actions. Simultaneously, joint target commands are smoothed and filtered to ensure continuous and abrupt joint motion trajectories. An interpolation control strategy with amplitude limiting is employed to progressively update the joint target, avoiding joint shocks caused by sudden target changes. In the high-frequency real-time control loop, a second-order critical damping filter mechanism is used to smooth the joint target, ensuring the system response is neither overshoot nor oscillating.

[0080] In some embodiments, after controlling the target robot to drive the first workpiece to perform a contact action with the second workpiece, the method may include: If the first workpiece fails to make contact with the second workpiece, the target robot is controlled to move the first workpiece back and the steps of "based on the deviation between the position information of the second workpiece and the position information of the first workpiece, the target robot is controlled to move the first workpiece so that the relative position deviation between the target areas of the first workpiece and the second workpiece meets the second preset condition; the target robot is controlled to move the first workpiece to make contact with the second workpiece" are executed again until the first workpiece makes contact with the second workpiece.

[0081] In this embodiment, "unsuccessful contact" means the contact action was not successfully completed. For contact actions of the insertion type, "unsuccessful contact" means the first workpiece failed to enter the target area of ​​the second workpiece. Reasons for unsuccessful contact may include alignment errors, deformation of the first workpiece, end-effector jitter, etc.

[0082] If the first workpiece fails to make contact with the second workpiece, the target robot can move the first workpiece in the opposite direction of the contact direction, causing it to return to its position before the contact action was performed. The purpose of this retraction is to allow the system to retry the alignment and contact actions.

[0083] After the rollback, the third control phase (precision alignment based on the second workpiece position information) and the contact action execution phase are restarted. The re-execution does not start completely from scratch, but rather performs incremental fine-tuning based on the previous alignment.

[0084] After the contact action is performed, the system needs to detect whether the contact was successful (for example, by visually determining whether the first workpiece has successfully entered the target area, or by using a force sensor to determine whether abnormal resistance has been encountered). If a failure is detected, the system will not simply report an error or stop, but will automatically execute a failure recovery procedure.

[0085] In this application, by setting a failure retry mechanism, the overall success rate of the system is greatly improved. Even if a single contact action fails, the system can automatically recover and retry. The backtrack distance is less than the stroke, which ensures the efficiency of retry. The incremental fine adjustment avoids the need for complete realignment for each retry, saving time. It is applicable to various contact action types (insertion, smearing, pressing, spraying, etc.) and has versatility.

[0086] The visual servo alignment method provided in this application can be executed by a visual servo alignment device. This application uses the example of a visual servo alignment device executing the visual servo alignment method to illustrate the visual servo alignment device provided in this application.

[0087] This application also provides a visual servo alignment device.

[0088] like Figure 2 As shown, the visual servo alignment device includes: a first processing module 210, a second processing module 220, a third processing module 230, and a fourth processing module 240.

[0089] The first processing module 210 is used to control the target robot to move the first workpiece based on the position information of the first workpiece, so that the relative position deviation between the first workpiece and the preset reference area meets the first preset condition. The second processing module 220 is used to control the target robot to rotate the first workpiece based on the shape features of the first workpiece, so that the shape of the first workpiece converges to meet the target shape conditions. The third processing module 230 is used to control the target robot to move the first workpiece based on the deviation between the position information of the second workpiece and the position information of the first workpiece, so that the relative position deviation between the target areas of the first workpiece and the second workpiece meets the second preset condition. The fourth processing module 240 is used to control the target robot to drive the first workpiece to perform a contact action with the second workpiece.

[0090] According to the visual servo alignment device provided in the embodiments of this application, by decomposing the visual servo alignment into sequentially executed coarse position alignment (moving towards a preset reference area with the position of the first workpiece as feedback), shape convergence alignment (rotating with the shape features of the first workpiece as feedback), and fine position alignment (moving towards the target area with the deviation between the position of the second workpiece and the position of the first workpiece as feedback), multi-degree-of-freedom decoupled control is realized. Furthermore, the problem of angle feature failure when approaching the target area is solved by utilizing shape features. The secondary fine alignment based on the target area of ​​the second workpiece improves the final alignment accuracy and ensures the determinism and stability of the control.

[0091] In some embodiments, the visual servo alignment device may further include a fifth processing module for acquiring shape features based on the following steps: Obtain the binary mask image of the first workpiece; Calculate the second central moments of the binary mask image; The normalized shape descriptor is determined based on the second-order central moment. The normalized shape descriptor is used to characterize the degree to which the first workpiece changes continuously from the first shape to the second shape in the image.

[0092] In some embodiments, the first form is a thin, elongated line segment, and the second form is a dot; the second processing module 220 can also be used for: By making the normalized shape descriptor approach the target value, the projected shape of the first workpiece is driven to shrink from a thin line segment shape to a dot shape. When the projected shape of the first workpiece shrinks to a dot shape, it is determined that the shape of the first workpiece has converged.

[0093] In some embodiments, the first processing module 210 may also be used for: During the process of controlling the target robot to drive the first workpiece to move, an attitude maintenance component is superimposed. The attitude maintenance component is generated based on the deviation between the current shape features of the first workpiece and the target shape features.

[0094] In some embodiments, the fourth processing module 240 can also be used for: Based on the first control frequency, the joint target command is smoothed and filtered to control the target robot to perform contact action. The first control frequency is higher than the second control frequency that controls the movement of the target robot based on the position information of the first workpiece.

[0095] In some embodiments, the visual servo alignment device may further include a sixth processing module, configured to, after controlling the target robot to drive the first workpiece to perform a contact action with the second workpiece, if the first workpiece fails to successfully contact the second workpiece, control the target robot to drive the first workpiece to retract and re-execute the steps of "based on the deviation between the position information of the second workpiece and the position information of the first workpiece, controlling the target robot to drive the first workpiece to move so that the relative position deviation between the target areas of the first workpiece and the second workpiece meets the second preset condition; controlling the target robot to drive the first workpiece to perform a contact action with the second workpiece", until the first workpiece successfully contacts the second workpiece.

[0096] The visual servo alignment device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific devices.

[0097] The visual servo alignment device in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit it.

[0098] The visual servo alignment device provided in this application embodiment can achieve... Figure 1 To avoid repetition, the various processes implemented in the method implementation examples will not be described again here.

[0099] In some embodiments, this application also provides a visual servo alignment system, including: Image acquisition unit, used to acquire images of the first workpiece and the second workpiece; The feature extraction unit is used to extract the position information and shape features of the first workpiece, as well as the position information of the second workpiece, from the image; The controller, connected to the image acquisition unit, the feature extraction unit, and the target robot, is used to execute the visual servo alignment method as described in any of the above embodiments to control the target robot.

[0100] In this embodiment, the image acquisition unit refers to a device used to acquire images of the first and second workpieces. The image acquisition unit can be a monocular camera, a binocular camera, a depth camera, or any sensor capable of acquiring images. The image acquisition unit can be mounted in a fixed position (such as on a fixed bracket outside the robotic arm) or mounted at the end of the robotic arm (with the eye on the hand).

[0101] A feature extraction unit refers to a device or module used to extract visual features from an image. A feature extraction unit can perform shape feature extraction, including binary mask image acquisition, second-order central moment calculation, and normalized shape descriptor determination. The feature extraction unit can be implemented in software on a general-purpose processor or using a dedicated image processing chip or accelerator.

[0102] The controller refers to the computing device that executes the visual servo alignment method. The controller is connected to the image acquisition unit, the feature extraction unit, and the target robot. It receives visual features output by the feature extraction unit, generates control commands according to the method described in any of the above embodiments, and sends the control commands to the target robot. The controller can be an industrial computer, an embedded system, a PLC, an FPGA, or any device with computing capabilities.

[0103] According to the visual servo alignment system provided in the embodiments of this application, by decomposing the visual servo alignment into sequentially executed coarse position alignment (moving towards a preset reference area with the position of the first workpiece as feedback), shape convergence alignment (rotating with the shape features of the first workpiece as feedback), and fine position alignment (moving towards the target area with the deviation between the position of the second workpiece and the position of the first workpiece as feedback), multi-degree-of-freedom decoupled control is achieved. Furthermore, the problem of angle feature failure when approaching the target area is solved by utilizing shape features. The secondary fine alignment based on the target area of ​​the second workpiece improves the final alignment accuracy and ensures the determinism and stability of the control.

[0104] In some embodiments, such as Figure 3 As shown, this application embodiment also provides an electronic device 300, including a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the program is executed by the processor 301, it implements the various processes of the above-described visual servo alignment method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0105] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0106] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described visual servo alignment method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0107] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0108] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described visual servo alignment method.

[0109] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0110] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described visual servo alignment method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0111] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0112] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0114] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0115] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0116] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A visual servo alignment method, characterized in that, include: Based on the position information of the first workpiece, the target robot is controlled to move the first workpiece so that the relative positional deviation between the first workpiece and the preset reference area meets the first preset condition. Based on the shape features of the first workpiece, the target robot is controlled to rotate the first workpiece so that the shape of the first workpiece converges to meet the target shape conditions. Based on the deviation between the position information of the second workpiece and the position information of the first workpiece, the target robot is controlled to drive the first workpiece to move, so that the relative positional deviation between the target areas of the first workpiece and the second workpiece meets the second preset condition. The target robot is controlled to drive the first workpiece to perform a contact action with the second workpiece; The shape features are obtained based on the following steps: Obtain the binary mask image of the first workpiece; Calculate the second-order central moments of the binary mask image; A normalized shape descriptor is determined based on the second-order central moment. The normalized shape descriptor is used to characterize the degree to which the first workpiece continuously changes from a first shape to a second shape in the image. The first form is a thin, elongated line segment, and the second form is a dot. The step of controlling the target robot to rotate the first workpiece, causing the shape of the first workpiece to converge to meet the target shape condition, includes: By making the normalized shape descriptor approach the target value, the projected shape of the first workpiece is driven to shrink from the elongated line segment shape to the dot shape. When the projected shape of the first workpiece shrinks to the dot shape, it is determined that the shape of the first workpiece has converged. The step of controlling the target robot to move the first workpiece based on the position information of the first workpiece includes: During the process of controlling the target robot to drive the first workpiece to move, an attitude maintenance component is superimposed, which is generated based on the deviation between the current shape features and the target shape features of the first workpiece. The normalized shape descriptor is used to characterize the degree of anisotropy of the first workpiece, which decreases as the projected shape of the first workpiece shrinks from the elongated line segment shape to the dot shape.

2. The visual servo alignment method according to claim 1, characterized in that, The control of the target robot to drive the first workpiece to perform a contact action with the second workpiece includes: Based on a first control frequency, the joint target command is smoothed and filtered to control the target robot to perform a contact action. The first control frequency is higher than the second control frequency that controls the movement of the target robot based on the position information of the first workpiece.

3. The visual servo alignment method according to claim 1, characterized in that, After controlling the target robot to drive the first workpiece to perform a contact action with the second workpiece, the method includes: If the first workpiece fails to make contact with the second workpiece, the target robot is controlled to move the first workpiece back and the steps of "based on the deviation between the position information of the second workpiece and the position information of the first workpiece, the target robot is controlled to move the first workpiece so that the relative position deviation between the target areas of the first workpiece and the second workpiece meets the second preset condition; the target robot is controlled to move the first workpiece to make contact with the second workpiece" are executed again until the first workpiece successfully makes contact with the second workpiece.

4. A visual servo alignment device, characterized in that, include: The first processing module is used to control the target robot to move the first workpiece based on the position information of the first workpiece, so that the relative position deviation between the first workpiece and the preset reference area meets the first preset condition. The second processing module is used to control the target robot to rotate the first workpiece based on the shape features of the first workpiece, so that the shape of the first workpiece converges to meet the target shape conditions. The third processing module is used to control the target robot to move the first workpiece based on the deviation between the position information of the second workpiece and the position information of the first workpiece, so that the relative position deviation between the target areas of the first workpiece and the second workpiece meets the second preset condition. The fourth processing module is used to control the target robot to drive the first workpiece to perform a contact action with the second workpiece; The fifth processing module is used to obtain the shape features based on the following steps: Obtain the binary mask image of the first workpiece; Calculate the second-order central moments of the binary mask image; A normalized shape descriptor is determined based on the second-order central moment. The normalized shape descriptor is used to characterize the degree to which the first workpiece continuously changes from a first shape to a second shape in the image. The first form is a thin, elongated line segment, and the second form is a dot; the second processing module is used for: By making the normalized shape descriptor approach the target value, the projected shape of the first workpiece is driven to shrink from the elongated line segment shape to the dot shape. When the projected shape of the first workpiece shrinks to the dot shape, it is determined that the shape of the first workpiece has converged. The first processing module is used for: During the process of controlling the target robot to drive the first workpiece to move, an attitude maintenance component is superimposed, which is generated based on the deviation between the current shape features and the target shape features of the first workpiece. The normalized shape descriptor is used to characterize the degree of anisotropy of the first workpiece, which decreases as the projected shape of the first workpiece shrinks from the elongated line segment shape to the dot shape.

5. A visual servo alignment system, characterized in that, include: An image acquisition unit is used to acquire images of the first workpiece and the second workpiece. The feature extraction unit is used to extract the position information and shape features of the first workpiece, and the position information of the second workpiece from the image; A controller, connected to the image acquisition unit, the feature extraction unit, and the target robot, is used to execute the visual servo alignment method as described in any one of claims 1-3 to control the target robot.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the visual servo alignment method as described in any one of claims 1-3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the visual servo alignment method as described in any one of claims 1-3.

Images