Wire harness grabbing control method and system

Abstract

The invention provides a wire harness grabbing control method and system, and belongs to the technical field of robot automatic assemblation.The wire harness grabbing control method comprises the steps that in response to a grabbing instruction, a mechanical arm is controlled to move to an observation position; scheduling a local positioning camera at the observation position to shoot an original image containing a target wire harness; detecting and analyzing the original image to obtain pose information and a grabbing point of the target wire harness; in combination with the pose information and the grabbing point, the grabbing pose of the mechanical arm is planned; the mechanical arm is controlled to execute the grabbing action based on the grabbing posture; after the grabbing action is executed, the mechanical arm is controlled to move to a secondary confirmation position, and posture rechecking is conducted on the grabbed wire harness; by fusing position recognition, rough posture recognition and rotation angle analysis, accurate and automatic recognition of the wire harness space posture and the grabbing point is achieved, the posture of compensation rotation is planned, and the grabbing accuracy and success rate are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of robotic automated assembly technology, and in particular to a control method, system, computer device, computer-readable storage medium, and computer program product for wire harness grasping. Background Technology

[0002] In the assembly process of automated automobiles, the automatic grasping and subsequent splicing of wire harnesses by robotic arms is a challenging step. Traditional automation solutions typically rely on precise feeding systems to ensure that the wire harnesses appear in designated positions with a fixed posture. However, in many real-world scenarios, such as retrieving materials from scattered bins or handling flexible wire harnesses, the initial position and posture of the wire harnesses often change randomly. This randomness makes it difficult for the robotic arm to grasp the wires accurately and stably, directly affecting production efficiency and product quality.

[0003] There is an urgent need for a control method, system, computer equipment, computer-readable storage medium, and computer program product for wire harness grasping. Summary of the Invention

[0004] This specification provides a control method, system, computer device, computer-readable storage medium, and computer program product for wire harness gripping, in order to solve the problems of inaccurate positioning caused by the variable orientation of terminals in automated wire harness gripping, which in turn leads to a low success rate of insertion after gripping.

[0005] The control method for wire harness grasping provided in this application adopts the following technical solution, including: In response to a grasping command, control the robotic arm to move to the observation position; At the observation location, a local positioning camera is deployed to capture raw images containing the target wire bundle; The original image is analyzed to obtain the pose information and grasping points of the target wire harness. Specifically, the wire harness region is segmented from the original image; the terminal region within the wire harness region is located; coordinate information is calculated based on the wire harness region; orientation information is calculated based on the terminal region; rotation analysis is performed on the terminal head region using a rotation angle detection model to obtain rotation angle information; the coordinate information, orientation information, and rotation angle information of the target wire harness are integrated with the terminal head rotation angle information to obtain the pose information; and based on the orientation information and preset grasping rules, grasping points on the target wire harness are determined. Combining the pose information with the gripping point, the gripping posture of the robotic arm is planned; wherein, the gripping posture includes: an approach posture for approaching the target wire bundle, and a final posture for clamping the target wire bundle after compensating for the rotation angle; Control the robotic arm to perform a grasping action based on the grasping posture; After performing the grasping action, the robotic arm is controlled to move to the secondary confirmation position to verify the posture of the grasped wire harness.

[0006] Optionally, calculating coordinate information based on the wire harness region includes: Calculate the centroid coordinates of the wire harness region; Obtain the height information of the target wire harness; The coordinate information of the target wire bundle is determined by combining the centroid coordinates and the height information; The calculation of orientation information based on the terminal area includes: The head key points and tail key points are determined according to the type of the terminal area; The bundle vector of the target bundle is calculated by combining the head key points and the tail key points; The harness vector is used as the orientation information.

[0007] Optionally, the step of performing rotation analysis on the terminal head region using a rotation angle detection model to obtain rotation angle information includes: Extract the current angle features of the terminal head region; The current angle feature is matched with multiple pre-stored standard angle feature representations of different rotation angles to determine the matching standard angle feature as the target angle feature; The rotation angle information is determined based on the rotation angle corresponding to the target angle feature.

[0008] Optionally, the step of combining the pose information and the grasping point to plan the grasping posture of the robotic arm includes: The approach posture of the robotic arm is determined by combining the grasping point, the coordinate information, and the current pose of the robotic arm. The final posture of the end effector is adjusted based on the orientation information and the rotation angle information.

[0009] Optionally, controlling the robotic arm to perform a grasping action based on the grasping posture includes: Drive the robotic arm to adjust to an approach posture; The end effector is controlled to rotate to the final posture at the target position and perform a grasping action to complete the grasping of the target wire harness.

[0010] Optionally, driving the robotic arm to adjust to an approach posture includes: The target location is determined based on the capture point and the location information; A proximity path is constructed by combining the current position of the end effector with the target position; A proximity command is sent to the robotic arm to guide the proximal end of the robotic arm to move along the proximity path to achieve the proximity posture.

[0011] The control system for wire harness gripping provided in this application adopts the following technical solution, including: The initial guidance module is used to respond to the grasping command and control the robotic arm to move to the observation position; An image acquisition module is used to dispatch a local positioning camera at the observation location to capture raw images containing the target wire bundle; The detection and analysis module is used to detect and analyze the original image to obtain the pose information and grasping points of the target line bundle; The posture planning module is used to combine the pose information and the gripping point to plan the gripping posture of the robotic arm; wherein, the gripping posture includes: an approach posture for approaching the target wire bundle, and a final posture for clamping the target wire bundle after compensating for the rotation angle. The grasping execution module is used to control the robotic arm to perform grasping actions based on the grasping posture; The attitude verification module is used to control the robotic arm to move to the secondary confirmation position after the grasping action is performed, and to verify the attitude of the grasped wire harness.

[0012] Optionally, the detection and analysis module includes: A region extraction submodule is used to segment the wire harness region from the original image and locate the terminal region within the wire harness region. The position calculation submodule is used to calculate coordinate information based on the wire harness area; An orientation estimation submodule is used to calculate orientation information based on the terminal area; The angle detection submodule is used to perform rotation analysis on the terminal head area through the rotation angle detection model to obtain rotation angle information. An integration submodule is used to integrate the coordinate information, orientation information and rotation angle information of the target wire harness with the terminal head to obtain the pose information; The gripping point positioning submodule is used to determine the gripping point on the target wire harness based on the orientation information and the preset gripping rules. Optionally, the location calculation submodule includes: Calculate the centroid coordinates of the wire harness region; Obtain the height information of the target wire harness; The coordinate information of the target wire bundle is determined by combining the centroid coordinates and the height information; Optionally, the direction estimation submodule includes: The head key points and tail key points are determined according to the type of the terminal area; The bundle vector of the target bundle is calculated by combining the head key points and the tail key points; The harness vector is used as the orientation information.

[0013] Optionally, the angle detection submodule includes: Extract the current angle features of the terminal head region; The current angle feature is matched with multiple pre-stored standard angle feature representations of different rotation angles to determine the matching standard angle feature as the target angle feature; The rotation angle information is determined based on the rotation angle corresponding to the target angle feature.

[0014] Optionally, the attitude planning module includes: The approach posture planning submodule is used to determine the approach posture of the robotic arm by combining the grasping point, the coordinate information and the current pose of the robotic arm. The final attitude planning submodule is used to adjust the final attitude of the end effector based on the orientation information and the rotation angle information.

[0015] Optionally, the capture execution module includes: The first execution submodule is used to drive the robotic arm to adjust to an approach posture; The second execution submodule is used to control the end effector to rotate to the final posture at the target position and perform a grasping action to complete the grasping of the target wire harness.

[0016] Optionally, the first execution submodule includes: The target location is determined based on the capture point and the location information; A proximity path is constructed by combining the current position of the end effector with the target position; A proximity command is sent to the robotic arm to guide the proximal end of the robotic arm to move along the proximity path to achieve the proximity posture.

[0017] This specification also provides a computer device, wherein the computer device includes: Processor; and, A memory that stores computer-executable instructions, which, when executed, cause the processor to perform any of the methods described above.

[0018] This specification also provides a computer-readable storage medium that stores one or more programs / instructions that, when executed by a processor, implement any of the methods described above.

[0019] This specification also provides a computer program product, wherein the computer program product includes: a computer program / instruction, which, when executed by a processor, implements any of the methods described above.

[0020] In this invention, in response to a grasping command, a robotic arm is controlled to move to an observation position; a local positioning camera is dispatched at the observation position to capture an original image containing the target wire harness; the original image is detected and analyzed to obtain the pose information and grasping point of the target wire harness; the grasping posture of the robotic arm is planned by combining the pose information and the grasping point; wherein, the grasping posture includes: an approach posture for approaching the target wire harness, and a final posture for clamping the target wire harness after compensating for rotation angle; the robotic arm is controlled to perform a grasping action based on the grasping posture; after performing the grasping action, the robotic arm is controlled to move to a secondary confirmation position to verify the posture of the grasped wire harness; this invention calculates the spatial coordinates, orientation, and terminal rotation angle of the wire harness, and integrates and plans a composite grasping posture including pre-approach and final clamping, achieving accurate and stable grasping of flexible wire harnesses. Simultaneously, the secondary posture verification mechanism after grasping effectively avoids misgrabbing and misalignment, significantly improving the success rate and reliability of the automated assembly process. Attached Figure Description

[0021] Figure 1 A schematic diagram illustrating the principle of a wire harness gripping control method provided in the embodiments of this specification; Figure 2 A schematic diagram of the structure of a wire harness grasping control system provided in the embodiments of this specification; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification; Figure 4 This is a schematic diagram of a computer-readable storage medium provided for an embodiment of this specification. Detailed Implementation

[0022] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0023] Exemplary embodiments of the invention will now be described more fully with reference to the accompanying drawings. While conforming to the inventive concept, the features, structures, characteristics, or other details described in a particular embodiment may be combined in one or more other embodiments in a suitable manner.

[0024] The terms “and / or” or “and / or” include all combinations of any one or more of the listed items.

[0025] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0026] Figure 1 A schematic diagram illustrating the principle of a wire harness gripping control method provided in this embodiment of the specification, comprising: S1 responds to the grasping command and controls the robotic arm to move to the observation position; S2 dispatches a local positioning camera at the observation location to capture a raw image containing the target wire bundle; S3 performs detection and analysis on the original image to obtain the pose information and grasping points of the target wire bundle; S4 combines the pose information with the gripping point to plan the gripping posture of the robotic arm; wherein, the gripping posture includes: an approach posture for approaching the target wire bundle, and a final posture for clamping the target wire bundle after compensating for the rotation angle. S5 controls the robotic arm to perform a grasping action based on the grasping posture; After performing the grasping action, S6 controls the robotic arm to move to the secondary confirmation position to verify the posture of the grasped wire harness.

[0027] To address the challenges of adapting robotic arms to random wire harness poses for free grasping in existing technologies, this invention provides a control method for wire harness grasping based on machine vision and artificial intelligence, involving wire harness pose and rotation angle detection and adaptive grasping by the robotic arm. This control method is applied to industrial robots handling wire harness grasping tasks. The hardware of the industrial robot includes at least: a base, a robotic arm, and an end effector.

[0028] The base, which is equivalent to the workstation of an industrial robot, is stationary and is equipped with a light source system to ensure stable imaging conditions. The robotic arm is an industrial robotic arm with sufficient degrees of freedom. The proximal end of the robotic arm is hinged to the base, and the distal end is connected to an end effector. Preferably, the end effector is equipped with a controllable flexible or customized gripper, capable of adjusting the gripping posture and force according to instructions. That is, the robotic arm can move its position based on specific instructions, and the end effector can perform rotation, gripping, and releasing actions based on specific instructions.

[0029] To facilitate the gripping action, one or more local positioning cameras are also installed; these local positioning cameras are industrial cameras; the local positioning cameras are mounted on the robotic arm to achieve "eyes on the hand"; and / or, the local positioning cameras are fixed above the workstation to achieve "eyes outside the hand". Specifically: S1 responds to the grasping command and controls the robotic arm to move to the observation position; In one embodiment of this specification, the loading station includes an identification mechanism and a loading mechanism. The identification mechanism is equipped with an identification code. The observation position is determined by scanning the identification code; then, a robotic arm is operated to move to the observation position. Preferably, the identification code is an Aruco code.

[0030] From the observation position, the field of view of the local positioning camera can cover the loading area, allowing a complete view of all the wiring harnesses. In practical applications, the observation position is a location approximately 10cm away from the loading area, positioned upwards from the normal vector of the control panel.

[0031] S2 dispatches a local positioning camera at the observation location to capture a raw image containing the target wire bundle; Terminals are provided at both ends of the wire harness. In one embodiment of this specification, the feeding mechanism is provided with a plurality of slots for orderly placement of the wire harness, and the slots are arranged at intervals. The slots include a first slot and a second slot, with the first slot located between the marking mechanism and the second slot. Each slot is used to place an independent wire harness. According to this layout rule, for any wire harness in any slot, the terminal located in the first slot (i.e., closer to the marking mechanism) is defined as the terminal head; correspondingly, the terminal located in the second slot is defined as the terminal tail. This structured arrangement makes the wire harness arrangement in the image relatively orderly, providing prior spatial constraints for subsequent image segmentation and key point localization.

[0032] Based on the acquired original image and the preset capture strategy (e.g., from top to bottom, from left to right), the target wire bundle is identified and determined from the image.

[0033] After the robotic arm moves to the observation position, a local positioning camera installed at the end of the robotic arm or fixed in a suitable position is scheduled to capture an original image containing the wire harnesses to be grasped in the current batch.

[0034] In this embodiment, a "one-time-one-material" grasping mode is adopted, that is, the robotic arm only positions and grasps one wire harness each time it moves. Therefore, in the original image captured in a single shot, the system only needs to perform subsequent detection and analysis on the target wire harness, which helps to improve the speed and accuracy of single processing.

[0035] S3 performs detection and analysis on the original image to obtain the pose information and grasping points of the target wire bundle; S31 segments the wire harness region from the original image; and locates the terminal region within the wire harness region; In this invention, the terminal area is designated as the region of interest (ROI). In machine vision and image processing, the area to be processed is delineated using rectangles, circles, or other irregular polygons. In this invention, since the wire harness appears in a fixed area of ​​the local positioning camera, the customized ROI allows the model to quickly locate the terminal area, thus improving the recognition rate.

[0036] S32 calculates coordinate information based on the wire harness area; calculates orientation information based on the terminal area; In the specific feeding or picking process, the three-dimensional spatial position (X, Y, Z coordinates) of the wire harness at the loading station and the overall orientation of the wire harness (e.g., the direction of the wire harness relative to the terminal head) are unpredictable. Due to the randomness of the wire harness's position and orientation, the robotic arm needs to accurately identify the precise position (coordinate information) and approximate orientation (orientation information) of the wire harness in order to plan an effective gripping path and ensure that the end effector can stably contact and clamp the wire harness or specific parts. Specifically: Calculate coordinate information based on the harness region, including: S321 calculates the centroid coordinates of the wire harness region; obtains the height information of the target wire harness; and determines the coordinate information of the target wire harness by combining the centroid coordinates and the height information.

[0037] In one embodiment of this specification, the centroid of the wire harness region is calculated as its principal location (X coordinate, Y coordinate). If a 3D camera is used, depth information can be directly acquired to obtain the Z coordinate. If a 2D camera is used, the Z coordinate is estimated by combining camera calibration and a preset working plane height, or by performing geometric reconstruction from multiple perspectives.

[0038] The orientation information is calculated based on the terminal area, including: S322 performs orientation estimation on the terminal area to obtain orientation information; S322-1 Determine the head key points and tail key points according to the type of the terminal area; Terminal areas include: terminal head area and terminal tail area.

[0039] In one embodiment of this specification, the wire harness is stored in the loading position; for each wire harness in the loading position, the terminal closest to the preset mark in the loading position is designated as the terminal head; and the terminal furthest from the preset mark in the loading position is designated as the terminal tail.

[0040] The terminal area containing the terminal head is called the terminal head area; key points of the head are determined based on the coordinates of the terminal head area; the terminal area containing the terminal tail is called the terminal tail area. Key points of the tail are determined based on the coordinates of the terminal tail area.

[0041] S322-2 Calculates the bundle vector of the target bundle by combining the head key points and the tail key points; In one embodiment of this specification, the harness vector is measured by triangulation. Specifically, using a binocular camera, the three-dimensional positions (i.e., Tcam_to_terminal) of the terminal head and terminal tail in the camera coordinate system are determined by triangulation, thereby calculating the vector from the terminal head to the terminal tail.

[0042] S322-3 uses the harness vector as the orientation information.

[0043] Orientation information refers to the position and orientation of the target beam in the world coordinate system.

[0044] In one embodiment of this specification, the approximate orientation of the harness is estimated by analyzing the principal axis direction of the harness region's profile, key points (such as endpoints, bends), or skeleton information.

[0045] Based on the above steps, industrial robots can quickly and robustly identify the spatial coordinates and main direction of the target wire harness in disordered or semi-disordered environments.

[0046] S33 performs rotational analysis on the terminal head area using a rotational angle detection model to obtain rotational angle information; The terminals of wire harnesses often have specific geometric shapes, and their rotation angle (rotation around their own axis) at the pick-up position is also random. This minute rotation angle directly determines whether subsequent insertion will be successful. Therefore, to ensure smooth subsequent insertion operations after gripping (requiring terminal and hole angle alignment and male-female matching), or to achieve consistent placement, this random rotation angle needs to be considered and compensated for during gripping, or the end effector needs to be controlled to grip at a specific angle so that the orientation of the wire harness terminals remains consistent after gripping.

[0047] S331 constructs a rotation angle detection model; The rotation angle detection model is a network structure with strong feature extraction capabilities and generalization ability, and attention mechanisms can be considered to make the model focus on key regions.

[0048] The preferred rotation angle detection model is a deep neural network, such as EfficientNet (extracting image features + Transformer decoding structure), ResNet, EfficientNet, Vision Transformer, etc.

[0049] S332 constructs training samples; In one embodiment of this specification, image samples of the terminal head at different rotation angles are collected. Preferably, a small number of samples can be obtained through rotational enhancement. For example, the wire harness is placed at a fixed position, with a camera at its front side, and the wire harness is rotated in 1° increments, with a set of data collected at each increment.

[0050] Training samples are constructed based on image samples of the terminal head at different rotation angles. The training samples are image pairs or triples.

[0051] S333 inputs the training samples into the rotation angle detection model for training; The training samples are input into the rotation angle detection model. Based on contrastive loss or triplet loss, the model learns a feature embedding space, so that images of the same (or similar) terminals are close in the embedding space, and images of terminals with different rotation angles or different types are far apart.

[0052] Preferably, the rotation angle detection model is trained to predict the difference in rotation angle of the input image relative to a "standard" pose (e.g., 0 degrees), or to directly regress the angle value. Contrastive learning helps the rotation angle detection model learn features that are insensitive to illumination and small displacements but sensitive to rotation.

[0053] To improve the model's generalization ability, data augmentation techniques are used extensively during training to simulate various possible disturbances, such as random rotation, translation, scaling, changes in lighting and contrast, adding noise, background replacement, and simulating wire harnesses of different thicknesses and curvatures. Parameters (such as texture, color, lighting, and physical properties) are randomized in the simulation environment so that the features learned by the model in the simulation environment can be transferred to the real world. Real image data covering various wire harness types, colors, sizes, and terminal shapes are collected.

[0054] In actual production, it may be necessary to handle various types and specifications of wire harnesses. These wire harnesses may vary significantly in length, color, thickness, terminal shape, and size. To adapt to the diversity of wire harness types, the model of this invention also possesses a certain ability to identify and process wire harness types that have not been rigorously trained. The model of this invention has good generalization ability and can stably cope with changes in the geometric and appearance characteristics of wire harnesses, thereby reducing the need for extensive data collection and model retraining for each new wire harness.

[0055] In one embodiment of this specification, for a new harness model, a model pre-trained on a large general dataset (e.g., ImageNet) can be used as a base, and then fine-tuned on the target harness dataset. For a new harness type, it may be possible to quickly adapt with only a small amount of labeled data for fine-tuning.

[0056] S334 inputs the image of the terminal head area into the rotation angle detection model to obtain rotation angle information; The rotation angle information is the rotation angle about its own axis, corresponding to the Roll angle in the world coordinate system or tool coordinate system.

[0057] S334-1 Extracts the current angle features of the terminal head region; S334-2 Matches the current angle feature with multiple pre-stored standard angle feature representations of different rotation angles, and determines the standard angle feature that matches it as the target angle feature; In one embodiment of this specification, the closest angle is found by calculating cosine similarity. Alternatively, the regression angle value output by the model can be used directly.

[0058] S334-3 determines the rotation angle information based on the rotation angle corresponding to the target angle feature.

[0059] Taking into account the randomness of terminal rotation angle, the present invention accurately and in real time detects the specific rotation angle of the terminal head of the target wire harness within the field of view through the above steps.

[0060] S35 integrates the coordinate information, orientation information and rotation angle information of the target wire harness with the terminal head to obtain the pose information; The pose information is 6D pose information, which consists of 3 translational degrees of freedom (X, Y, Z) and 3 rotational degrees of freedom (Roll, Pitch, Yaw or quaternions).

[0061] Specifically, the position (X, Y, Z) is obtained based on the coordinate information, two rotation angles (Pitch, Yaw) are obtained based on the orientation information, and the last rotation angle (Roll) is obtained based on the rotation angle information, thus obtaining the complete 6D pose information of the target harness.

[0062] S36 determines the gripping point on the target wire harness based on the orientation information and the preset gripping rules; Specifically, depending on the terminal head dimensions (such as length L and width W), the gripping point is usually selected at its geometric center or in a stable area near the end, such as at L / 3 of the distance from the end, to ensure that the clamping force is evenly distributed.

[0063] In one embodiment of this specification, the preset gripping rule includes: using a point at a preset distance from the tip of the terminal head as the gripping point. The preset length is preferably 5 cm. Specifically, through the aforementioned steps, the positions of the terminal head and terminal tail in the world coordinate system are determined using triangulation with a binocular camera. This yields a vector originating from the terminal head and pointing towards the terminal tail. The gripping point is located at the spatial coordinates corresponding to the point 5cm extending from this origin along the direction of the vector. These gripping point coordinates ultimately need to be transformed into coordinates in the robotic arm base coordinate system to guide the robotic arm's movement. The transformation process involves: first, using a hand-eye calibration matrix (… Transform the coordinates from the camera coordinate system to the robot arm end effector coordinate system, and then combine this with the known pose of the robot arm end effector in the base coordinate system. Finally, the coordinates of the grasping point in the robot arm base coordinate system (world coordinate system) are obtained, where the coordinates in the robot arm base coordinate system (world coordinate system) are... ,in This refers to the transformation from the camera coordinate system to the 5cm extended gripping point.

[0064] S4 combines the pose information with the gripping point to plan the gripping posture of the robotic arm; The grasping posture includes: an approach posture for approaching the target wire bundle, and a final posture for clamping the target wire bundle after compensating for the rotation angle. S41 combines the grasping point, the coordinate information, and the current pose of the robotic arm to determine the approach posture of the robotic arm; Approach posture means that the tool coordinate system is in position, and the clamp can be opened and closed to grasp the wire harness.

[0065] Using the gripping point (world coordinate system) and coordinate information calculated in step S3 as the primary target position, and combining it with the current pose of the robotic arm, path planning is performed. The goal of the planning is to precisely move the origin of the tool coordinate system (the center of the end effector) mounted at the end of the robotic arm to the gripping point, and to initially align one axis (approach direction) of the tool coordinate system with the direction of the harness vector determined by the orientation information. In this pose, the end effector has moved to a position where gripping can be performed, and the gripping point is located at the center of the end effector's opening. However, at this point, the terminal head still has a rotation angle, and the end effector has not yet compensated for the rotation angle (Roll angle) of the terminal head itself. Therefore, the angle still needs to be adjusted to the final pose.

[0066] S42 adjusts the final posture of the end effector based on the orientation information and the rotation angle information.

[0067] The final orientation is the position of the tool coordinate system after the aforementioned rotational compensation. In this orientation, the end effector is not only aligned with the gripping point, but its orientation also ensures that the terminal head is corrected to the desired direction when the end effector closes to clamp the wire harness. The desired orientation is a preset, consistent orientation, such as face up, to facilitate subsequent insertion or placement processes.

[0068] Specifically, the spindle direction is obtained based on the orientation information; a surface perpendicular to the spindle direction and containing the gripping point is determined as the rotation surface; the end effector is controlled to rotate around the wire harness vector on the rotation surface so that when the end effector is in its final state, the terminal head is facing upwards when the end effector grips the target wire harness.

[0069] In one embodiment of this specification, before performing the gripping action, the end effector, based on the approach posture, additionally rotates by a compensation angle about the harness vector determined by the orientation information (i.e., the approach direction in the tool coordinate system). The terminal head rotation is detected. If the degree is too high, the robotic arm needs to rotate additionally when grasping. The degree can be adjusted, or adaptively, according to the design of the end effector, to counteract random rotation of the terminal head.

[0070] This invention transforms the above pose calculation process into a spatial transformation from the tool coordinate system to the target coordinate system based on the terminal. By using the hand-eye calibration matrix and robotic arm kinematics, it can concisely and efficiently calculate the instructions required for the robotic arm end to reach the target coordinate system, thereby achieving precise positioning and orientation grasping.

[0071] S5 controls the robotic arm to perform a grasping action based on the grasping posture; S51 drives the robotic arm to adjust to an approach posture; S511 determines the target location based on the grasping point and the location information; S512 constructs an approach path by combining the current position of the end effector with the target position; S513 sends an approach command to the robotic arm, guiding the proximal end of the robotic arm to move along the approach path to achieve the approach posture.

[0072] S52 controls the end effector to rotate to the final posture at the target position and perform a grasping action to complete the grasping of the target wire harness.

[0073] As a preferred option, combining force-controlled or flexible end effectors enables adaptive clamping of wire harnesses of different thicknesses, avoiding damage.

[0074] After performing the grasping action, S6 controls the robotic arm to move to the secondary confirmation position to verify the posture of the grasped wire harness.

[0075] In one embodiment of this specification, an industrial camera is mounted on the top of the industrial robot. After the industrial robot receives the wire harness, it moves its robotic arm to the camera on its head, reuses step S3, and performs position recognition, rough orientation recognition, and fine detection of terminal rotation angle to perform secondary detection and confirmation of the target wire harness.

[0076] The originality of the core process (observation-identification-grabbing) of this invention lies in its unique solution for the specific and complex application scenario of "free grasping of wire harnesses", rather than a simple application of general workpiece pose estimation methods.

[0077] Figure 2 This is a schematic diagram of a wire harness gripping control system provided in an embodiment of this specification. The system includes: The initial guidance module 210 is used to control the robotic arm to move to the observation position in response to the grasping command; Image acquisition module 220 is used to dispatch a local positioning camera at the observation location to capture an original image containing the target wire bundle; The detection and analysis module 230 is used to detect and analyze the original image to obtain the pose information and grasping points of the target line bundle; The posture planning module 240 is used to combine the pose information and the gripping point to plan the gripping posture of the robotic arm; wherein, the gripping posture includes: an approach posture for approaching the target wire bundle, and a final posture for clamping the target wire bundle after compensating for the rotation angle. The grasping execution module 250 is used to control the robotic arm to perform grasping actions based on the grasping posture; The attitude verification module 260 is used to control the robotic arm to move to the secondary confirmation position after performing the grasping action, and to perform attitude verification on the grasped wire harness.

[0078] Optionally, the detection and analysis module 230 includes: A region extraction submodule is used to segment the wire harness region from the original image and locate the terminal region within the wire harness region. The position calculation submodule is used to calculate coordinate information based on the wire harness area; An orientation estimation submodule is used to calculate orientation information based on the terminal area; The angle detection submodule is used to perform rotation analysis on the terminal head area through the rotation angle detection model to obtain rotation angle information. An integration submodule is used to integrate the coordinate information, orientation information and rotation angle information of the target wire harness with the terminal head to obtain the pose information; The gripping point positioning submodule is used to determine the gripping point on the target wire harness based on the orientation information and the preset gripping rules. Optionally, the location calculation submodule includes: Calculate the centroid coordinates of the wire harness region; Obtain the height information of the target wire harness; The coordinate information of the target wire bundle is determined by combining the centroid coordinates and the height information; Optionally, the direction estimation submodule includes: The head key points and tail key points are determined according to the type of the terminal area; The bundle vector of the target bundle is calculated by combining the head key points and the tail key points; The harness vector is used as the orientation information.

[0079] Optionally, the angle detection submodule includes: Extract the current angle features of the terminal head region; The current angle feature is matched with multiple pre-stored standard angle feature representations of different rotation angles to determine the matching standard angle feature as the target angle feature; The rotation angle information is determined based on the rotation angle corresponding to the target angle feature.

[0080] Optionally, the attitude planning module 240 includes: The approach posture planning submodule is used to determine the approach posture of the robotic arm by combining the grasping point, the coordinate information and the current pose of the robotic arm. The final attitude planning submodule is used to adjust the final attitude of the end effector based on the orientation information and the rotation angle information.

[0081] Optionally, the capture execution module 250 includes: The first execution submodule is used to drive the robotic arm to adjust to an approach posture; The second execution submodule is used to control the end effector to rotate to the final posture at the target position and perform a grasping action to complete the grasping of the target wire harness.

[0082] Optionally, the first execution submodule includes: The target location is determined based on the capture point and the location information; A proximity path is constructed by combining the current position of the end effector with the target position; A proximity command is sent to the robotic arm to guide the proximal end of the robotic arm to move along the proximity path to achieve the proximity posture.

[0083] The functions of the system in this embodiment have been described in the above method embodiments. Therefore, for any parts not detailed in this embodiment, please refer to the relevant descriptions in the foregoing embodiments, which will not be repeated here.

[0084] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. The electronic device includes a memory 301 and a processor 302. The memory 301 is used to store computer-executable instructions. When the computer-executable instructions are executed by the processor 302, they can implement the steps of the above-described method embodiments.

[0085] Figure 4 This is a schematic diagram of the structure of a computer-readable storage medium 400 provided in an embodiment of this specification. The computer-readable storage medium 400 stores one or more computer programs, which, when executed by a processor, can implement the steps of the above-described method embodiments.

[0086] This specification also provides a computer program product, including a computer program / computer executable instructions, which, when executed by a processor, can implement the steps of the above-described method embodiments.

[0087] Those skilled in the art will understand that all or part of the processes in the above method embodiments can be implemented by a computer program instructing related hardware. When the computer program is executed, it may include the processes of the embodiments of the above methods.

[0088] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A control method for wire harness grasping, characterized in that, include: In response to a grasping command, control the robotic arm to move to the observation position; At the observation location, a local positioning camera is deployed to capture raw images containing the target wire bundle; The original image is analyzed to obtain the pose information and grasping points of the target wire harness. Specifically, the wire harness region is segmented from the original image; the terminal region within the wire harness region is located; coordinate information is calculated based on the wire harness region; orientation information is calculated based on the terminal region; rotation analysis is performed on the terminal head region using a rotation angle detection model to obtain rotation angle information; the coordinate information, orientation information, and rotation angle information of the target wire harness are integrated with the terminal head rotation angle information to obtain the pose information; and based on the orientation information and preset grasping rules, grasping points on the target wire harness are determined. Combining the pose information with the gripping point, the gripping posture of the robotic arm is planned; wherein, the gripping posture includes: an approach posture for approaching the target wire bundle, and a final posture for clamping the target wire bundle after compensating for the rotation angle; Control the robotic arm to perform a grasping action based on the grasping posture; After performing the grasping action, the robotic arm is controlled to move to the secondary confirmation position to verify the posture of the grasped wire harness.

2. The control method for wire harness gripping as described in claim 1, characterized in that, The step of calculating coordinate information based on the harness region includes: Calculate the centroid coordinates of the wire harness region; Obtain the height information of the target wire harness; The coordinate information of the target wire bundle is determined by combining the centroid coordinates and the height information; The calculation of orientation information based on the terminal area includes: The head key points and tail key points are determined according to the type of the terminal area; The bundle vector of the target bundle is calculated by combining the head key points and the tail key points; The harness vector is used as the orientation information.

3. The control method for wire harness grasping as described in claim 1, characterized in that, The step of performing rotational analysis on the terminal head area using a rotational angle detection model to obtain rotational angle information includes: Extract the current angle features of the terminal head region; The current angle feature is matched with multiple pre-stored standard angle feature representations of different rotation angles to determine the matching standard angle feature as the target angle feature; The rotation angle information is determined based on the rotation angle corresponding to the target angle feature.

4. The control method for wire harness gripping as described in claim 1, characterized in that, The step of combining the pose information and the grasping point to plan the grasping posture of the robotic arm includes: The approach posture of the robotic arm is determined by combining the grasping point, the coordinate information, and the current pose of the robotic arm. The final posture of the end effector is adjusted based on the orientation information and the rotation angle information.

5. The control method for wire harness grasping as described in claim 4, characterized in that, The control of the robotic arm to perform a grasping action based on the grasping posture includes: Drive the robotic arm to adjust to an approach posture; The end effector is controlled to rotate to the final posture at the target position and perform a grasping action to complete the grasping of the target wire harness.

6. The control method for wire harness grasping as described in claim 1, characterized in that, The process of driving the robotic arm to adjust to an approach posture includes: The target location is determined based on the capture point and the location information; A proximity path is constructed by combining the current position of the end effector with the target position; A proximity command is sent to the robotic arm to guide the proximal end of the robotic arm to move along the proximity path to achieve the proximity posture.

7. A control system for wire harness gripping, characterized in that, include: The initial guidance module is used to respond to the grasping command and control the robotic arm to move to the observation position; An image acquisition module is used to dispatch a local positioning camera at the observation location to capture raw images containing the target wire bundle; The detection and analysis module is used to detect and analyze the original image to obtain the pose information and grasping points of the target wire harness. Specifically, it segments the wire harness region from the original image and locates the terminal region within the wire harness region; calculates coordinate information based on the wire harness region; calculates orientation information based on the terminal region; performs rotation analysis on the terminal head region using a rotation angle detection model to obtain rotation angle information; integrates the coordinate information, orientation information, and rotation angle information of the target wire harness with the terminal head to obtain the pose information; and determines the grasping points on the target wire harness based on the orientation information and preset grasping rules. The posture planning module is used to combine the pose information and the gripping point to plan the gripping posture of the robotic arm; wherein, the gripping posture includes: an approach posture for approaching the target wire bundle, and a final posture for clamping the target wire bundle after compensating for the rotation angle. The grasping execution module is used to control the robotic arm to perform grasping actions based on the grasping posture; The attitude verification module is used to control the robotic arm to move to the secondary confirmation position after the grasping action is performed, and to verify the attitude of the grasped wire harness.

8. A computer device, characterized in that, The computer device includes: Processor; and, A memory storing computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs / instructions, which, when executed by a processor, implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program / instruction, which, when executed by a processor, implements the method as described in any one of claims 1-6.

Images