Method and system for aligning and inserting wire contacts into insertion holes of connectors

Abstract

The present disclosure relates to a method and system for aligning and inserting a wire contact into an insertion hole of a connector. The method may include: obtaining captured images of a wire clamp of an end effector of a robot from at least two image capture devices attached to the end effector; causing the robot to advance the end effector to move the wire contact within a predetermined distance of the connector; causing the robot to advance the end effector to move the wire contact toward the connector by a predetermined additional amount; and confirming whether the alignment is correct using force feedback at the wire clamp based on the wire contact moving the predetermined additional amount.

Description

Technical Field

[0001] According to example embodiments, a method, system, and computer program product are provided for aligning wire contacts with insertion holes of a connector, and more particularly, relate to automatically aligning wire contacts with wire contact insertion holes of a connector using a machine vision system and inserting the wire contacts into corresponding holes of the connector. Background Art

[0002] Wiring harnesses, consisting of multiple wires, are used across a wide range of industries to carry countless different types of signals. The wires of a wiring harness assembly must often be terminated with wire contacts, and the resulting wire ends inserted into wire contact insertion holes in a connector, such as a rubber grommet. Because each wire in a wiring harness is unique and can carry a different type of signal, the wire ends of the wiring harness assembly must be inserted into specific wire contact insertion holes in the connector to properly connect.

[0003] The wire ends of a wiring harness assembly can be manually inserted into corresponding wire contact insertion holes defined by the connector. Because a wiring harness assembly typically includes hundreds of wires, this manual connection process can be relatively time consuming and prone to error, and as a result, can increase the cost of the entire assembly including the wiring harness assembly. As such, automated techniques have been developed for inserting the wire ends of a wiring harness assembly into the wire contact insertion holes of a connector in an effort to reduce the time spent making the connection and correspondingly reduce the cost of the resulting assembly. However, wiring harness assembly machines typically require the connector to be in a very restricted and controlled set of positions in order to increase the likelihood of correctly inserting the wire ends of the wiring harness assembly into the wire contact insertion holes of the connector. Consequently, wiring harness assembly machines limit the flexibility that the connector can exhibit and, therefore, are not suitable for all situations. Summary of the Invention

[0004] Methods, systems, and computer program products are provided for aligning wire contacts with insertion apertures defined by a connector to facilitate automated insertion of wire ends of a wiring harness assembly into the wire contact insertion apertures of the connector. By facilitating automated insertion of wire ends of a wiring harness assembly into the wire contact insertion apertures of the connector, the time required to create such a connection, and correspondingly the cost associated with the resulting assembly, can be reduced, while also reducing the error rate associated with the connection. Example embodiments of the method, system, and computer program product provide significant flexibility with respect to the manner in which the connector can be positioned while still allowing the wire ends of the wiring harness assembly to be securely inserted into and electrically connected to the appropriate wire contact insertion apertures of the connector.

[0005] In an exemplary embodiment, a system for aligning a wire contact with an insertion hole of a connector and inserting the wire contact into a corresponding hole of the connector is provided. The system may include: a robot having an end effector, wherein the end effector includes a wire clamp and at least two image capture devices secured to the end effector; and a computing device. The computing device may be configured to: process images captured by the image capture devices to establish a correction transformation that aligns the wire contact with a target hole of the connector in a connector surface; cause the robot to translate the end effector to move the wire clamp along the correction transformation; cause the robot to advance the end effector until contact is made with the connector surface; confirm whether the wire contact is within the target hole of the connector based on force measurements; cause the robot to advance the end effector to move the wire contact toward the connector by a predetermined additional amount; and confirm whether the alignment is correct using force feedback at the wire clamp based on the wire contact moving the predetermined additional amount.

[0006] According to some embodiments, the computing device configured to confirm whether the wire contact is within the target hole of the connector based on force measurement is configured to: confirm that the wire contact is not aligned with the target hole of the connector in response to the force observed at the wire clamp being higher than a predefined value; and confirm that the wire contact is aligned with the target hole of the connector in response to the force observed at the wire clamp being lower than the predefined value. According to some embodiments, the computing device configured to process images captured by the image capture device to establish a corrective transformation to align the wire contact with the target hole of the connector may be configured to: identify, within the processed image, a tip of the wire contact and a direction along which the wire contact extends; and establish, based on the processed image, a corrective transformation to align the tip of the wire contact and the direction along which the wire contact extends with the axis of the three-dimensional coordinate system of the end effector of the robot.

[0007] The computing device of some embodiments may be configured to: determine a maximum distance that the wire clamp can advance toward the connector; cause the robot to advance the end effector to move the wire clamp toward the connector; determine that the wire contact is fully inserted into the connector in response to force feedback on the wire clamp satisfying an insertion force value as the robot is caused to advance the end effector to move the wire clamp toward the connector; and determine that the wire contact is not fully inserted into the connector in response to force feedback on the wire clamp failing to satisfy the insertion force value before reaching the maximum distance that the wire clamp can advance toward the connector as the robot is caused to advance the end effector to move the wire clamp toward the connector. In response to the computing device determining that the wire contact is fully inserted into the connector, the computing device is further configured to: cause the robot to retract the end effector to move the wire clamp away from the connector; confirm that the wire contact is incorrectly inserted in response to the wire clamp moving a pull distance without force feedback on the wire clamp exceeding a pull test value; and confirm that the wire contact is correctly inserted in response to the force feedback on the wire clamp exceeding a pull test value without the wire clamp moving the pull distance.

[0008] According to some embodiments, in response to the computing device determining that the wire contact is not fully inserted into the connector, the computing device may be further configured to: cause the wire gripper to release its grip on the wire contact; cause the robot to retract the end effector to move the wire gripper away from the connector; cause the wire gripper to re-grip the wire contact; and cause the robot to advance the end effector to move the wire gripper and the wire contact toward the connector. According to some embodiments, force feedback may be established based on a force sensor disposed between the robot and the wire gripper. A force-torque sensor may be capable of accurately detecting forces in three orthogonal axes and torques about each of these three axes, enabling a force-torque tool interface to be employed between the robot and the end effector. Another example embodiment that may be used in place of or in conjunction with a force-torque sensor is a strain gauge or strain gauge array in the wire gripper. A single strain gauge may not be capable of establishing an axis along which force is received, but in example embodiments may be used to detect force feedback of a wire contact being inserted into or extracted from a connector. If desired, a strain gauge can be used to identify the axis along which the force is received. The computing device of some embodiments can be configured to: identify the orientation of the visible portion of the wire contact in the image captured by the image capture device; and use the orientation of the wire contact to calculate a corrective movement in the coordinate system of the end effector of the robot.

[0009] Embodiments provided herein may include a method for aligning a wire contact with a target hole of a connector and inserting the wire contact into the target hole. The method may include: obtaining captured images of a wire gripper of an end effector of a robot from at least two image capture devices attached to the end effector; processing the images captured by the image capture devices to establish a correction transformation that aligns the wire contact with the target hole of the connector in a connector surface; causing the robot to translate the end effector to move the wire gripper along the correction transformation; causing the robot to advance the end effector until contact is made with the connector surface; confirming whether the wire contact is inside the target hole of the connector based on force measurements; causing the robot to advance the end effector to move the wire contact toward the connector by a predetermined additional amount; and confirming whether the alignment is correct using force feedback at the wire gripper based on the wire contact moving the predetermined additional amount.

[0010] According to some embodiments, confirming whether the wire contact is within the target aperture of the connector based on force measurement may include: confirming that the wire contact is not aligned with the target aperture of the connector in response to the force observed at the wire clamp being above a predefined value; and confirming that the wire contact is aligned with the target aperture of the connector in response to the force observed at the wire clamp being below the predefined value. Processing the image captured by the image capture device to establish a correction transformation that aligns the wire contact with the target aperture of the connector may include the steps of: identifying, within the processed image, a tip of the wire contact and a direction along which the wire contact extends; and establishing, based on the processed image, a correction transformation that aligns the tip of the wire contact and the direction along which the wire contact extends with the axis of the three-dimensional coordinate system of the end effector of the robot.

[0011] The method may include: determining a maximum distance that the wire clamp can advance toward the connector; causing the robot to advance the end effector to move the wire clamp toward the connector; determining that the wire contact is fully inserted into the connector in response to force feedback on the wire clamp satisfying an insertion force value as the robot is caused to advance the end effector to move the wire clamp toward the connector; and determining that the wire contact is not fully inserted into the connector in response to force feedback on the wire clamp failing to satisfy the insertion force value before reaching the maximum distance that the wire clamp can advance toward the connector as the robot is caused to advance the end effector to move the wire clamp toward the connector.

[0012] According to some embodiments, in response to determining that the wire contact is fully inserted into the connector, the method may include: causing the robot to retract the end effector to move the wire clamp away from the connector; confirming that the wire contact is incorrectly inserted in response to the wire clamp moving a pull distance without exceeding a pull force test value; and confirming that the wire contact is correctly inserted in response to the force feedback on the wire clamp exceeding a pull force test value without the wire clamp moving the pull distance. In response to determining that the wire contact is not fully inserted into the connector, the method may include: causing the wire clamp to release its grip on the wire contact; causing the robot to retract the end effector to move the wire clamp away from the connector; causing the wire clamp to re-grip the wire contact; and causing the robot to advance the end effector to move the wire clamp and the wire contact toward the connector. Force feedback at the wire clamp is established based on a force sensor between the robot and the wire clamp. Optionally, the method may comprise the steps of identifying, in an image captured by the image capture device, an orientation of a visible portion of the wire contact; and calculating a corrective movement in a coordinate system of the end effector of the robot using the orientation of the wire contact.

[0013] Embodiments provided herein may include a computer program product for aligning a wire contact with a target aperture defined by a connector and inserting the wire contact into the target aperture. The computer program product may include at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein. The computer-executable program code instructions include program code instructions for performing the following steps: obtaining captured images of the wire clamp of the end effector of the robot from at least two image capture devices attached to the end effector of the robot; processing the images captured by the image capture devices to establish a correction transformation that aligns the wire contact with the target hole of the connector in the connector surface; causing the robot to translate the end effector so that the wire clamp moves along the correction transformation; causing the robot to advance the end effector until contact is made with the connector surface; confirming whether the wire contact is inside the target hole of the connector based on force measurement; causing the robot to advance the end effector to move the wire contact toward the connector by a predetermined additional amount; and confirming whether the alignment is correct using force feedback at the wire clamp based on the wire contact moving the predetermined additional amount.

[0014] The program code instructions for confirming whether the wire contact is within the target aperture of the connector based on force measurement may include program code instructions for performing the following steps: confirming that the wire contact is not aligned with the target aperture of the connector in response to the force observed at the wire clamp being above a predefined value; and confirming that the wire contact is aligned with the target aperture of the connector in response to the force observed at the wire clamp being below the predefined value. The program code instructions for processing images captured by the image capture device to establish a correction transformation that aligns the wire contact with the target aperture of the connector may include program code instructions for performing the steps of: identifying, within the processed image, a tip of the wire contact and a direction in which the wire contact extends; and establishing, based on the processed image, a correction transformation that aligns the tip of the wire contact and the direction in which the wire contact extends with the axis of the three-dimensional coordinate system of the end effector of the robot.

[0015] According to some embodiments, the computer program product may include program code instructions for performing the following steps: determining a maximum distance that the wire clamp can advance toward the connector; causing the robot to advance the end effector to move the wire clamp toward the connector; determining that the wire contact is fully inserted into the connector in response to force feedback on the wire clamp satisfying an insertion force value as the robot is caused to advance the end effector to move the wire clamp toward the connector; and determining that the wire contact is not fully inserted into the connector in response to force feedback on the wire clamp failing to satisfy the insertion force value before reaching the maximum distance that the wire clamp can advance toward the connector as the robot is caused to advance the end effector to move the wire clamp toward the connector.

[0016] According to some embodiments, in response to determining that the wire contact is fully inserted into the connector, the computer program product may include program code instructions for: causing the robot to retract the end effector to move the wire clamp away from the connector; confirming that the wire contact is not properly inserted in response to the wire clamp moving a pull distance without force feedback on the wire clamp exceeding a pull test value; and confirming that the wire contact is properly inserted in response to the force feedback on the wire clamp exceeding a pull test value without the wire clamp moving the pull distance. In response to determining that the wire contact is not fully inserted into the connector, the computer program product may include program code instructions for: causing the wire clamp to release its grip on the wire contact; causing the robot to retract the end effector to move the wire clamp away from the connector; causing the wire clamp to re-grip the wire contact; and causing the robot to advance the end effector to move the wire clamp and the wire contact toward the connector.

[0017] Embodiments provided herein may include an apparatus for aligning a wire contact with a target hole of a connector and inserting the wire contact into the target hole. Example apparatus may include: means for obtaining captured images of a wire clamp of an end effector of a robot from at least two image capture devices attached to the end effector; means for processing the images captured by the image capture devices to establish a correction transformation that aligns the wire contact with the target hole of the connector in a connector surface; means for causing the robot to translate the end effector to move the wire clamp along the correction transformation; means for causing the robot to advance the end effector until contact is made with the connector surface; means for confirming whether the wire contact is inside the target hole of the connector based on force measurement; means for causing the robot to advance the end effector to move the wire contact toward the connector by a predetermined additional amount; and means for confirming whether the alignment is correct using force feedback at the wire clamp based on the wire contact being moved the predetermined additional amount.

[0018] According to some embodiments, the means for confirming whether the wire contact is within the target aperture of the connector based on force measurement may include: means for confirming that the wire contact is not aligned with the target aperture of the connector in response to the force observed at the wire clamp being above a predefined value; and means for confirming that the wire contact is aligned with the target aperture of the connector in response to the force observed at the wire clamp being below the predefined value. The means for processing images captured by the image capture device to establish a correction transformation for aligning the wire contact with the target aperture of the connector may include: means for identifying, within the processed image, a tip of the wire contact and a direction along which the wire contact extends; and means for establishing, based on the processed image, a correction transformation for aligning the tip of the wire contact and the direction along which the wire contact extends with the axis of the three-dimensional coordinate system of the end effector of the robot.

[0019] An embodiment of an apparatus may include: means for determining a maximum distance the wire clamp can advance toward the connector; means for causing the robot to advance the end effector to move the wire clamp toward the connector; means for determining that the wire contact is fully inserted into the connector in response to force feedback on the wire clamp satisfying an insertion force value as the robot is caused to advance the end effector to move the wire clamp toward the connector; and means for determining that the wire contact is not fully inserted into the connector in response to force feedback on the wire clamp failing to satisfy the insertion force value before reaching the maximum distance the wire clamp can advance toward the connector as the robot is caused to advance the end effector to move the wire clamp toward the connector.

[0020] According to some embodiments, in response to determining that the wire contact is fully inserted into the connector, the apparatus may include: means for causing the robot to retract the end effector to move the wire clamp away from the connector; means for confirming that the wire contact is incorrectly inserted in response to the wire clamp moving a pull distance without exceeding a pull force test value; and means for confirming that the wire contact is correctly inserted in response to the wire clamp exceeding a pull force test value without the wire clamp moving the pull distance. In response to determining that the wire contact is not fully inserted into the connector, the apparatus may include: means for causing the wire clamp to release its grip on the wire contact; means for causing the robot to retract the end effector to move the wire clamp away from the connector; means for causing the wire clamp to re-grip the wire contact; and means for causing the robot to advance the end effector to move the wire clamp and the wire contact toward the connector. Force feedback at the wire clamp is established based on a force sensor between the robot and the wire clamp. The example apparatus may further include means for identifying an orientation of a visible portion of the wire contact in an image captured by the image capture device; and means for calculating a corrective movement in a coordinate system of the end effector of the robot using the orientation of the wire contact. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Having thus generally described certain example embodiments of the present disclosure, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and in which:

[0022] Figure 1 is a perspective view of a connector according to an example embodiment of the present disclosure;

[0023] Figure 2 According to an example embodiment of the present disclosure Figure 1 A front view of the connector;

[0024] Figure 3 is a block diagram of a system that may be configured particularly in accordance with example embodiments of the present disclosure;

[0025] Figure 4 depicts a robotic end effector, a wire clamp, and an image capture device according to example embodiments of the present disclosure;

[0026] Figure 5 The example embodiment according to the present disclosure is illustrated by Figure 4 an image of the connector captured by an image capture device of an end effector of the robot;

[0027] Figure 6 is a flow chart of a calibration routine for calibrating an image capture device relative to a wire clamp and a robotic end effector according to an example embodiment of the present disclosure;

[0028] Figure 7 is a flow chart of a process for aligning a wire contact with a target insertion hole according to an example embodiment of the present disclosure;

[0029] Figure 8 illustrates a process of extracting wire contact orientation and tip position from an image according to an example embodiment of the present disclosure;

[0030] Figure 9 illustrates a process flow for inspecting contact holes in a connector according to an example embodiment of the present disclosure;

[0031] Figure 10 is a flow chart of a process for aligning the orientation of a wire contact with a target aperture of a connector according to an example embodiment of the present disclosure;

[0032] Figure 11 illustrates a connector having a wiring harness attached thereto using example embodiments according to the alignment techniques described herein;

[0033] Figure 12 is a flow chart of a process for aligning a wire contact with a target aperture of a connector according to an example embodiment of the present disclosure;

[0034] Figure 13 is a process flow of a method of aligning a wire contact with a target hole of a connector and inserting the wire contact into the target hole according to an example embodiment of the present disclosure; and

[0035] Figure 14 is a flow chart of a process of aligning a wire contact with a target hole of a connector and inserting the wire contact into the target hole according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, aspects are shown. Indeed, the present disclosure may be embodied in many different forms and should not be construed as limited to the aspects set forth herein. Rather, these aspects are provided so that the present disclosure will satisfy applicable legal requirements. Like reference numerals refer to like elements throughout.

[0037] According to example embodiments described herein, a method, system, and computer program product are provided for aligning wire ends / contacts with corresponding insertion holes of a connector (such as in a rubber grommet of the connector) to facilitate automated robotic wire insertion. The process described herein uses a camera mounted to a robotic end effector to simultaneously detect wire contacts and insertion holes. Utilizing simultaneous detection, embodiments of the disclosed method provide feedback to corrective movements of a robotic arm used to insert wire contacts into insertion holes of a connector. Movement of the robotic arm aligns the wire contacts with target insertion holes for successful insertion into the appropriate holes of the connector. As further described below, the simultaneous detection described herein includes a multi-step process of detecting contact holes (which may use color filtering), calculating distances to a hole template, finding extreme values ​​in a distance map, and matching a hole mask to these extreme values.

[0038] Traditionally, wiring harness assembly, which involves attaching one or more wire connectors to a wiring harness, is a labor-intensive process that is both time-consuming and introduces opportunities for error during assembly. Embodiments described herein enable automated assembly of wiring harnesses and their associated wire connectors. Specifically, embodiments provide for automated insertion of wire ends into connectors. Embodiments described herein enable the use of a robotic arm with an end effector to insert wires, thereby enabling flexible layout of connectors and wires.

[0039] According to example embodiments, a method, system, and computer program product are provided for identifying wire contacts and wire contact insertion holes defined by a connector to align and insert the wire contacts into the wire contact insertion holes. While the method, system, and computer program product can be configured to identify a variety of different types of wire contacts and wire contact insertion holes of connectors, connectors typically define multiple wire contact insertion holes within a housing, where the wire contact insertion holes are arranged in a predefined configuration. Different connectors may include different numbers of wire contact insertion holes and may include the wire contact insertion holes arranged in different configurations.

[0040] Figure 1 and Figure 2 An example of a connector is depicted in the form of a connector 10. As shown, the connector 10 includes a housing 12 and a rubber gasket 16 disposed in the housing 12. Although the housing 12 may be configured differently for other types of connectors, Figure 1 and Figure 2 The housing of the connector 10 of the embodiment is externally threaded to facilitate secure threaded engagement therewith with, for example, a wiring harness assembly or another connector. Figure 1 and Figure 2 The connector 10 in FIG. 1 also includes a radially extending flange defining a plurality of openings 14, such as for receiving screws or other fasteners for mounting the connector to an assembly. Figure 1 The connector 10 in FIG. 1 has a cylindrical shape, but connectors of other example embodiments may have different sizes and shapes. Figure 1 and Figure 2 In the example connector of FIG. 1 , a rubber gasket 16 is disposed within the housing and defines a plurality of wire contact insertion holes 18. The wire contact insertion holes 18 defined by the rubber gasket 16 are configured (e.g., sized and shaped) such that a wire end, such as a wire contact connected (e.g., crimped) to an end of a wire, is inserted and mechanically retained within the wire contact insertion holes 18. In some, but not all, embodiments, the rubber gasket may further include a plurality of wire contacts that align with corresponding wire contact insertion holes defined by the rubber gasket so that the wire end can be brought into secure electrical contact with the wire through corresponding wire contacts of the connector.

[0041] like Figure 1 and Figure 2 As shown in the example of the connector 10 of FIG. 1 , a plurality of wire contact insertion holes 18 defined by the rubber gasket 16 are arranged in a predefined pattern. In some embodiments, not all of the wire contact insertion holes of the connector 10 will be utilized. Instead, only a portion of the wire contact insertion holes will receive and electrically connect corresponding wire ends of the wiring harness assembly. Figure 2 As illustrated in FIG, by inserting the plug 20 into the corresponding wire contact insertion hole defined by the rubber gasket, the wire contact insertion hole 18 defined by the rubber gasket 16 that will not be utilized in conjunction with a specific application may be disregarded in further consideration. Figure 1 and Figure 2 A connector 10 that can be analyzed according to an example embodiment of the present disclosure is depicted and will be described below, but the methods, systems, and computer products of the example embodiments can be utilized in conjunction with a variety of other connectors, and the connectors are illustrated and described by way of example, not limitation.

[0042] Now refer to Figure 3 , depicts a system for identifying the wire contact insertion holes of a connector 10. As shown, the system 30 includes a camera 32 configured to capture an image of the connector 10. Although Figure 3, multiple cameras are indicated, but embodiments may employ a single camera, or may employ a single camera operated in conjunction with a reflector to provide various stereoscopic views of the connector 10 using a single camera. The camera described herein is an image capture device, wherein various image capture device types may be used in place of the camera. Typically, an image capture device captures an image of the field of view of the device. A camera as described herein captures an image of the field of view in the visible spectrum and processes the image accordingly. The camera 32 may be configured to capture a grayscale image of the connector 10. Alternatively, the camera 32 may be configured to capture a color image of the connector 10. In embodiments that capture a color image of the connector 10, the images associated with the different color channels of the camera 32, such as the red, green, and blue channels, may be averaged to create a composite image for subsequent analysis and viewing. Alternatively, the different color channels of the camera 32 may be analyzed separately. The camera 32 is typically configured to capture images such as Figure 2 , such that the plurality of wire contact insertion holes 18 defined by the rubber gasket 16 are clearly visible. The camera 32 may also be configured to capture images of the wire contacts during alignment of the wire contacts with the connector 10. In this manner, images captured by the camera 32 of the example embodiment may be captured at multiple angles to provide different perspective views of the connector 10 and the wire contacts.

[0043] In addition to the camera 32, Figure 3 The system 30 further includes a computing device 34 configured to analyze the image of the connector 10 captured by the camera and identify the wire contacts and the wire contact insertion holes of the connector. Figure 3 As shown in , the system 30 of the example embodiment also includes a robot 44 or communicates with the robot 44, and more specifically, includes a robot end effector that is used to insert the wire ends / contacts into corresponding candidate contact insertion holes of the connector 10 based on the wire contacts and the wire contact insertion holes of the connector identified by the computing device 34.

[0044] Computing device 34 can be configured in various ways and, therefore, can be implemented as a personal computer, tablet computer, computer workstation, mobile computing device (such as a smartphone), server, etc. Regardless of the manner in which computing device 34 is implemented, the computing device of the example embodiments includes, or is associated with, processing circuitry 36, memory 38, and optionally a user interface 40 and a communication interface 42 for performing the various functions described herein. Processing circuitry 36 can be implemented, for example, as various devices including one or more microprocessors, one or more coprocessors, one or more multi-core processors, one or more controllers, one or more computers, various other processing elements including integrated circuits such as, for example, ASICs (application-specific integrated circuits) or FPGAs (field-programmable gate arrays), or some combination thereof. In some example embodiments, processing circuitry 36 is configured to execute instructions stored in memory 38 or otherwise accessible to the processing circuitry. These instructions, when executed by processing circuitry 36, can cause computing device 34, and therefore system 30, to perform one or more of the functions described herein. Thus, computing device 34 may include an entity capable of performing operations according to example embodiments of the present disclosure while being configured accordingly. Thus, for example, when processing circuit 36 ​​is implemented as an ASIC, FPGA, or the like, the processing circuit and, accordingly, computing device 34 may include specially configured hardware for performing one or more operations described herein. Alternatively, as another example, when processing circuit 36 ​​is implemented as an executor of instructions (such as may be stored in memory 38), the instructions may specifically configure the processing circuit and, in turn, the computing device 34 to perform one or more algorithms and operations described herein.

[0045] Memory 38 may include, for example, volatile and / or non-volatile memory. Memory 38 may include, for example, a hard disk, random access memory, cache memory, flash memory, an optical disk (e.g., a compact disk read-only memory (CD-ROM), a digital versatile disk read-only memory (DVD-ROM), etc.), circuitry configured to store information, or some combination thereof. In this regard, memory 38 may include any non-transitory computer-readable storage medium. Memory 38 may be configured to store information, data, applications, instructions, etc. that enable computing device 34 to perform various functions in accordance with example embodiments of the present disclosure. For example, memory 38 may be configured to store program instructions for execution by processing circuitry 36.

[0046] The user interface 40 can communicate with the processing circuitry 36 and the memory 38 to receive user input and / or provide auditory, visual, mechanical, or other output to the user. As such, the user interface 40 can include, for example, a display for providing an image captured by the camera 32 and / or an image visually depicting the closest match between a candidate contact and a predetermined template as described below. Other examples of the user interface 40 include a keyboard, a mouse, a joystick, a microphone, and / or other input / output mechanisms.

[0047] The communication interface 42 can communicate with the processing circuit 36 ​​and the memory 38 and can be configured to receive and / or send data, such as by receiving an image from the camera 32 and sending information such as a list of candidate contact insertion holes, the contact ID number and position of the candidate contact insertion holes in the coordinate system based on the connector to the robot 44 and / or the machine end effector. Although referred to herein as candidate contact insertion holes, the contact ID number and position of the candidate contact insertion holes, and the list of candidate contact insertion holes, the contact ID number and position of the candidate contact insertion holes will be interpreted as being associated with the candidate contact insertion holes themselves and / or such wire contacts in those embodiments that include wire contacts aligned with the corresponding candidate contact insertion holes. The communication interface 42 may include, for example, one or more antennas and supporting hardware and / or software for enabling communication with a wireless communication network. Additionally or alternatively, the communication interface 42 may include circuitry for interacting with the antenna to cause signal transmission via the antenna or processing of signals received via the antenna. In some environments, the communication interface 42 may alternatively or also support wired communication.

[0048] Now refer to Figure 4 , shows an example embodiment of a system for performing the methods described herein, the system including a robotic end effector 100 including a first camera 102 and a second camera 104. The robotic end effector 100 can carry a wire 110 (the wire 110 including a wire contact at its front end) in a wire clamp 108, while the connector 112 is in a fixed position when the robotic end effector 100 is adjacent to the connector 112. The two cameras 102, 104 are mounted on the robotic end effector 100 in such a manner that they view both the wire 110 including the wire contact and the connector 112 simultaneously.

[0049] although Figure 4 The embodiment includes two cameras, but embodiments may include more cameras. Additionally, a single camera may be used in conjunction with a reflector to allow viewing of different perspectives of the wire contact and connector using a single camera. Capturing multiple perspectives using two or more cameras, for example, allows for accurate positioning of the wire contact and connector when they are coupled.

[0050] According to the example embodiments described herein, images of the wires and wire contacts along with the connector are captured from more than one angle. Using the different perspectives, a line extending in the direction of the wires and wire contacts is identified, and a hole in the connector is identified as the target hole for the wires. Figure 5 Illustrated by Figure 4 The images of the wire 110 including the wire contact and the connector (specifically, the identified hole 124 of the connector into which the wire 110 is to be inserted) captured by the cameras 102 and 104 in the image. The line identified by the multiple stereo images provides at least a stereo indication of the relationship between the wire contact and the target hole of the connector into which the wire 110 is to be inserted. Based on the line identified from the images, a movement command can be calculated to place the hole on the line in at least two images. This can initially establish a rotation of the end effector so that the top of the wire clamp 108 is perpendicular to the connector surface. In order to place the hole on the line, a movement parallel to the connector surface is established to align the wire with the appropriate target hole of the connector. The movement command is the desired displacement of the robot end effector in three-dimensional Cartesian space. Aligning the wire contact with the hole places the wire in the correct position so that the robot can move the wire along the line toward the appropriate hole of the connector for insertion.

[0051] The embodiments described herein can calibrate the camera before use to align the wire with the target hole of the connector. The purpose of calibration is to calculate the mapping of three-dimensional Cartesian coordinates to two-dimensional image coordinates. Calibration can be performed before the wire is fed into the manipulator of the end effector. Calibration does not need to be performed before each wire insertion or before each connector change, but it may be necessary when camera settings (such as focus, zoom, orientation, etc.) are changed.

[0052] Figure 6The process flow of an example calibration process is depicted. The illustrated process uses a small calibration rod, which can be, for example, a small plastic rod approximately one inch long with a clear tip (such as a red tip). According to some embodiments, the calibration rod can include a small sphere on a needle such as a computer-aided measuring machine calibration pen or simply a small dot on a piece of paper. The calibration rod is used to provide an easily identifiable point that is visible in each camera field of view from different camera perspectives. The calibration rod can be securely mounted with the tip facing upward and near the robot end effector. In preparation for calibration, the robot end effector is pushed in front of the calibration rod so that the tip of the calibration rod is positioned approximately at the desired location of the connector surface (e.g., the surface in which the connector hole is formed). The calibration process can begin with a list of end effector positions, as shown in 130. An example of such a list is the three-dimensional coordinates of nodes in a 3×3×3 cubic grid, where adjacent nodes are 1 cm apart. A constraint in generating this list can be that for each coordinate of the end effector, the tip of the calibration rod must be visible in all camera images.

[0053] At 132 , a new position for the end effector position is obtained. At 134 , the robot cycles through the list of end effector positions by moving the robot end effector to the obtained position, capturing an image of the calibration rod at 136 , finding the coordinates of the rod's tip in both camera images at 138 , and recording the end effector position at 140 . This process loops back, obtaining new positions from the list, until all end effector positions have been used for calibration, or at least a predefined number of end effector positions have been used to provide satisfactory calibration. In each captured image, the position of the rod's tip is identified. To identify the tip, the image can be color filtered (e.g., by calculating R-(G+B) / 2 for each pixel, where R, G, and B are the red, green, and blue channels, respectively). The average position of all pixels with intensity values ​​above a predetermined value can then be calculated. The predetermined value can be selected so that only the rod's tip is selected. It may be beneficial to have a light source above the rod so that the tip is well illuminated and stands out in the captured image. The result of this calibration process is the two-dimensional image coordinates of the tip of the calibration rod in each camera image.

[0054] once Figure 6The calibration routine in is completed, and the result is a list of three-dimensional end effector positions in the end effector coordinate system and the corresponding two-dimensional image coordinates for each camera. This set of corresponding coordinates is used by an algorithm to calibrate each camera. For example, a perspective N-point (PnP) algorithm can be used to calibrate each camera. A non-limiting example of a PnP algorithm can be the UPnP+Gauss-Newton algorithm. The result of this algorithm is two matrices for each camera: one that encodes intrinsic parameters (such as focal length) and one that encodes extrinsic parameters (the position and pose of the camera). These matrices can be used to map the three-dimensional position in the robot end effector coordinate system to two-dimensional image coordinates. Using this calibration process, the camera positions do not need to be known in advance.

[0055] Once the camera has been calibrated, the wire contacts held by the machine end effector can be aligned with the connector. Figure 7 The process of aligning a wire contact with a target insertion hole of a connector is illustrated. After the wire is grasped by the manipulator of the end effector, in 150, images are captured by a camera mounted on the end effector, whether the wire is placed in the manipulator or picked up by the manipulator. In these images, as shown in 152, the wire contact is detected and its orientation is obtained. This operation is further described below. Then, in 154, the robot can move the wire contact near the surface of the connector. At this position, in 156, the camera captures an image again to include the wire contact and the connector. From these images, two processes are calculated: first, in 158, the orientation of the wire contact is updated; and second, in 160, the connector hole is detected. In 162, by combining the outputs of these processes, the system calculates movement commands in robot end effector coordinates to align the contact with the target hole.

[0056] After the robot performs the first alignment step, the camera image is captured again and both the contact position and the target hole position are updated. If the update results in a correction movement command below a threshold (e.g., below 0.1 mm), the robot may not perform the correction, but instead advance to move the contact toward the connector surface. The direction of movement of the wire contact toward the connector surface matches the contact orientation in three dimensions obtained from the camera image. If the updated wire contact position results in a correction above the threshold, the robot may then perform a correction movement and capture a new image, thereby repeating the above-mentioned process until the correction is below the threshold.

[0057] Figure 7 The number of repetitions of the process may be limited, such as to three attempts. After this limit, the robot may abort the alignment process and indicate the error, for example, by an error message of the user interface. Alternatively, the robot may start again to move the contact near the connector surface as before. Figure 7There are three important elements of the alignment process.

[0058] Wire contact detection

[0059] Detection of the wire contacts is necessary to align the contacts with the target holes and to understand the direction of movement of the robotic end effector once the contacts are aligned. Figure 8 The process of extracting the orientation of a wire contact from an image is depicted. In this example embodiment, computing device 34, such as processing circuitry 36, may be configured to perform various operations to extract the orientation and tip position of a wire contact from a captured image. The first operation is to extract a window of the image in 166 where the contact is expected to be located. The image in this window may be color filtered (e.g., using a single color channel) to produce an image of only the color of interest. In an example embodiment in which the wire contact is gold, the image may be color filtered to find a gold-colored region in 167. Based on the gold-colored region extending along a linear direction, a fitted line is created in 168. The fitted line constrains the area processed for edge detection in 170, for example, by using a 30-pixel-wide corridor around the line. This corridor can cut out distracting edges from the background, such as from other wires. For edge detection, a Canny edge detection algorithm may be used. For Gaussian blurring, the unconstrained parameters of the Canny edge detector may be one or two σ, while for edge tracking, the thresholds may be 0.005 and 0.015. Second, to detect the line, a Hough transform may be performed on the edge, as shown in 172. Third, using the result array from the Hough transform, the maximum value corresponding to the longest line can be found at 174. By finding the maximum value, the angle and direction of the line, as well as its distance from one of the image corners, are identified. Near the maximum value, a nearby maximum value with the same line direction is found. An example of such a maximum value is one whose value is greater than 0.5 times the maximum value obtained using the Hough transform. These maximum values ​​may correspond to parallel lines in the direction of the contact. The center of the two extreme value lines can be estimated as the position and direction of the contact, as shown at 176.

[0060] Once the orientation of the contact is determined, such as by the processing circuitry 36 of the computing device 34, the position of the contact tip is calculated. To locate the tip, the ends of all edge lines parallel to the contact are determined. All ends are projected onto the contact line. The projection furthest from the image corner opposite the contact tip is identified as the tip location. This process of determining the orientation and tip location of the contact can be repeated for at least two camera images captured from different angles.

[0061] In the same manner as for the wire contact, although without the aforementioned gold color filter, the manipulator tip can be obtained by the processing circuitry 36 of the computing device 34. Here, images can be analyzed without the wire contact being inserted into the manipulator. These images can be captured during calibration. Because the manipulator is fixed relative to the camera, the manipulator tip position can be obtained as part of the calibration. Alternatively, the manipulator tip position can be calculated during the process before the wire is gripped by the manipulator.

[0062] Once the image coordinates of the manipulator and contact tip are known in at least two camera views, the three-dimensional coordinates of the tip can be calculated. To calculate the three-dimensional coordinates of a point, virtual lines can be formed extending from the camera position through the point in the image plane. These lines can be calculated based on the camera's extrinsic parameters obtained during calibration. The three-dimensional coordinates can be obtained as the least-squares solution that best approximates the virtual line from the at least two camera views. The contact tip's orientation in the three-dimensional end-effector coordinate system can be calculated as the vector difference between the three-dimensional position of the contact tip and the three-dimensional position of the manipulator tip.

[0063] Contact hole detection

[0064] Contact hole detection is necessary to correctly identify the correct hole of the connector into which the wire contact is to be inserted. In every camera image that includes a connector, the contact hole is detected. Figure 9 The corresponding process flow that can be executed by the processing circuit 36 ​​of the computing device 34 is illustrated. At the beginning of the process, the robot end effector is in front of the connector so that the connector surface is fully visible in at least two camera images from different angles. For at least two of the camera images from different angles, the following process is followed.

[0065] At 178, an image is captured by a camera mounted to the end effector of the robot. The image may be color filtered using a red filter, such as by calculating R-(G+B) / 2 for each pixel, where R, G, and B are the values ​​of the red, green, and blue light channels, respectively. The color filter may be selected based on the color of the connector so that it best identifies differences in the connector surface that may be associated with the holes in the connector. To crop the image, the median image coordinates of the color filtered intensity image are calculated, and a window is cut out at the median location, as shown in 180. Alternatively, the center of the window may be at the top of the wire contact. The size of the window depends on the type of connector and may be a pre-specified parameter, such as a size of 270×270 pixels, which is sufficient to clearly identify each hole in the connector. The color of the filter used should correspond to the color of the connector.

[0066] The processing circuitry 36 can then be used to calculate the squared distance between the hole template and the local image patch of the intensity image for each patch location within the image, as shown in 182. An example of a hole template may include an 18×18 pixel wide intensity gradient that approximates the shadow inside the hole, where the intensity along the gradient may follow the function f(x)=1 / (1+exp(-x / 1.7)). The intensity of the template may be scaled to match the minimum and maximum values ​​of the color-filtered intensity image. This scaling increases robustness to lighting variations. The result of this operation may include an intensity image in which low-intensity areas (black) are areas at a short distance from the hole template.

[0067] The intensity image can be used, such as by processing circuitry 36 of computing device 34, to isolate extrema identified in the image. These extrema correspond to contact holes and are local minima in the distance image. To identify local minima, an elliptical boundary around each pixel can be analyzed. The size of the boundary can depend on the distance between adjacent holes in the connector. As a non-limiting example, the elliptical boundary can include a semi-axis length of 25 pixels and a semi-axis length of 15 pixels, where the major axis approximates the elliptical shape of the hole in the camera image in the horizontal direction, as the image is not coaxial with the connector. If one or more pixels within the pixel boundary have an intensity lower than the candidate pixel intensity multiplied by a constant factor greater than 1, the pixel can be discarded as an extrema candidate. This constant factor can depend on the connector type. For example, the constant factor can be approximately 1.7. A factor greater than 1 ensures that the extrema are not more prominent and can eliminate or reduce false detections. For each remaining candidate pixel, a weighted average of all pixels within its elliptical boundary can be calculated, where the weight is the inverse of the intensity of each pixel in the distance image.

[0068] If the total weight of all pixels within the calculated ellipse boundary is above a threshold value (e.g., 2), the weighted average can be identified as a hole and added to the list. In 184, the detection of isolated peaks is shown, thereby identifying the holes of the connector. In order to avoid counting the pixels of the hole as a separate hole (double counting), all pixels within the ellipse used for weighted averaging can be marked and automatically discarded as extreme value candidates. The result of this operation is a list of contact holes. As an additional operation, outliers can be removed from the hole list. To remove outliers, the minimum distance between two holes (d min ). Second, any hole can be considered as being at a distance from its nearest neighbor greater than a constant factor times d min Outliers with a factor of 2 (e.g., a constant factor of 2) are discarded.

[0069] The hole list can be matched to the technical specifications of the connector and / or the known hole locations in the drawing. This matching can compensate for missed holes and allow hole identification numbers to be assigned to detected holes. From the technical drawing of the connector, a two-dimensional mask of the hole locations can be extracted. The mask can include a series of contacts with identifications and positions in a connector-centered coordinate system. To match the mask to the hole list, the mask can be rotated (on three axes) and translated (in three directions) in the end effector coordinate system so that it optimally overlaps with the hole list. To calculate the overlap, the mask can be projected onto each camera image using parameters from camera calibration. The cost function used for optimization can be the sum of the squared distances between the holes in the list and their nearest neighboring projected holes. Non-limiting examples of algorithms that optimize this cost function can include Powell's method.

[0070] Contact alignment

[0071] The above-mentioned process provides a line describing the position of the wire contact and the target hole for each camera image analyzed. Based on this information, the corrective movement of the robot end effector can be calculated. In 200, the target hole position in two or more camera images is identified. In 202, the three-dimensional position "p" of the target hole in the end effector coordinate system is calculated. To calculate this position, an optimization algorithm is used that minimizes the sum of the squared distances between the two-dimensional image position of the target hole and the projection of the three-dimensional position on the camera image. Non-limiting examples of optimization include Powell's method. Here, the three-dimensional position can be constrained to lie in the plane of the connector surface. This plane can be known due to the above-mentioned mask optimization process, which rotates and translates the mask to match the connector surface.

[0072] As shown in 204, the position "r" closest to the projection of the contact line in each image is calculated in the end-effector coordinate system. This position can also be constrained to lie in the plane of the connector surface. An optimization algorithm can be used to calculate "r." Based on the resulting values ​​of "p" and "r," a corrective movement can be calculated as c = pr in 208. At 210, the end-effector movement can be executed.

[0073] The identified target hole of the connector for the wire contact can then be utilized to facilitate insertion of the wire end into the corresponding wire contact insertion hole of the connector. In this regard, the wire can be identified by a wire diagram, etc., so as to be inserted into a specific wire contact insertion hole of the connector (and in some embodiments, also to make electrical contact with a corresponding wire contact aligned with the wire contact insertion hole), wherein the specific wire contact insertion hole is identified by a contact ID number, which can be identified on the connector via the diagram of the connector identifier mentioned above. Prior to insertion into the wire contact insertion hole of the connector, the wire contact is typically connected (e.g., crimped) to the exposed end of the wire to form the wire end. Based on the contact device ID number and the corresponding position of the candidate contact insertion hole for the connector 10, the wire end can be inserted into the connector at a position associated with the following contact insertion hole, which has the contact ID number of the wire contact insertion hole 18 into which the wire is to be inserted. The computing device 34, such as the processing circuit 36, can be configured to determine a candidate contact insertion hole into which the wire is to be inserted based on the contact ID number of the candidate contact insertion hole, such as a correspondence between the contact ID number of the candidate contact insertion hole and the contact ID number of the wire contact insertion hole 18 into which the wire end is to be inserted, as defined in a wire diagram, etc. The computing device 34, such as the processing circuit 36, can also be configured to determine a position of the robot 44, and more specifically, a position of the robot end effector to be used to insert the wire end into the candidate contact insertion hole, based on the position of the candidate contact insertion hole in the connector-based coordinate system and using the above-described method for efficiently and repeatably inserting the wire into the corresponding hole of the connector.

[0074] In this manner, the computing device 34, such as the processing circuit 36, can effectively drive the robot 44 (such as a robotic end effector), or otherwise provide information (such as a list of candidate contact insertion holes, contact ID numbers and corresponding positions in a connector-based coordinate system) to the robot 44 sufficient to drive the robotic end effector to insert the wire ends of the wiring harness assembly into the corresponding wire contact insertion holes 18. See, e.g. Figure 11 ,exist Figure 11 In the embodiment, a plurality of wires 90 have been inserted into corresponding wire contact insertion holes 18 of connector 10 to create a mechanical connection between the wire ends and connector 10. By facilitating automation of the connection process associated with a wire harness assembly, the system 30, method, and computer program product of the example embodiments improves the efficiency with which the wire ends of a wire harness assembly can be mechanically connected to connector 10 and correspondingly reduces the error rate and cost of the resulting assembly.

[0075] While the aforementioned process involves aligning a single wire contact for insertion, embodiments of the present disclosure can be used to align multiple wire contacts and insert them into corresponding target holes in a connector. However, the order in which the wire contacts are inserted into the target holes can be determined in a manner that does not diminish the effectiveness of the alignment method described herein. As discussed above, the wire contacts and target holes must each be visible in at least two camera images from two different angles for proper alignment. If the wire contacts are inserted into the connector in the incorrect order, the connector's target holes may be obscured from the view of one or more cameras. Thus, the assembly sequence can include starting with the target hole of the connector farthest from the camera (such as the bottom of the connector in the example configuration shown herein). In this way, the wires will be inserted into the connector from the bottom up to avoid obstructing the camera's view of the target holes. Multiple cameras viewing from multiple different angles can alleviate installation sequence requirements because, even when the camera's view of the target hole is obscured, the process described herein can be performed effectively, assuming the target hole remains visible in at least two images from at least two angles.

[0076] Figure 12 is a flow chart of a process for aligning a wire contact with a target hole of a connector according to an example embodiment of the present disclosure. As shown, in 220, images are obtained from at least two image capture devices (such as cameras 32 of device 30) attached to an end effector of a robot, wherein the images are images of the wire clamp of the end effector. In 222, within at least one image from each of the at least two image capture devices, a wire contact is detected, such as by processing circuitry 36 of computing device 34. Within at least one image from each of the at least two image capture devices, in 224, one or more insertion holes of the connector are detected, such as by processing circuitry 36 of computing device 34. In 226, a corrective movement of the robot end effector is identified, such as by processing circuitry 36 of computing device 34, which corrective movement aligns the target holes of the one or more insertion holes of the connector with the wire contact. In 228, the robot ( Figure 3 44 of the device 30) moves the end effector according to the identified correction movement.

[0077] Once the wire contact is aligned with the target connector hole, the wire contact can be inserted into the target hole to assemble the connector and its wiring harness. Initially, the wire contact can be moved to a prepared position near the target hole by a wire gripper on the robot's end effector. Alignment is performed in a plane parallel to the connector surface, while insertion is performed along an axis perpendicular to the plane of the connector surface. The wire contact can be moved a predetermined distance along the insertion axis to reach the connector's target hole, whereby the force measured on the wire gripper is used to confirm proper insertion of the wire contact. The wire contact can be moved a predetermined distance from the prepared position, where the predetermined distance is the distance that would result in contact with the connector if misaligned or a distance that would result in partial insertion into the target hole if correctly aligned. Optionally, the robot end effector can advance along the contact line until a force threshold is exceeded to establish surface contact, resulting in either misalignment and missing the hole or partial insertion into the target hole. After moving the predetermined distance, the wire gripper can measure the force on the wire gripper to confirm whether the force exceeds a surface contact threshold, which can be approximately 0.3 to 0.5 Newtons. The process can then test to confirm that the contact is inserted into the target hole by measuring the force while moving the initial predetermined distance. If the measured force is greater than a threshold, the process can be aborted, which may indicate misalignment between the wire contact and the target hole. If it is determined that the wire contact is not properly aligned with the target hole, the wire contact can be moved axially away from the connector under the action of the robot and end effector. The alignment process can be performed again to achieve correct alignment of the wire contact and correct translation of the wire contact in a plane parallel to the surface of the connector.

[0078] The initial distance and threshold force for the above-mentioned preparation position can be determined and confirmed empirically so that in the event of misalignment, the wire contact will not penetrate the rubber gasket of the connector. If the initial movement of the predetermined distance along the insertion axis from the preparation position is completed without exceeding the force threshold, the alignment is correct and the wire contact has successfully entered the target hole. If the robot advances a predetermined distance (such as 3 mm) beyond the point where the wire contact will first touch the connector, it can be determined that the wire contact has successfully entered the target hole if the force threshold (e.g., 4 Newtons) has not been exceeded during this movement. An estimate of the position of the target hole can then be calculated based on the direction vector of the contact, the commanded initial insertion depth, and an additional offset (e.g., 0 to 8 mm) determined empirically for the contact-connector pair. This additional offset may be necessary due to the detection of the surface contact passing through the actual surface plane of the connector, which depends on the hole, contact geometry, and friction.

[0079] Contact alignment and insertion in holes

[0080] Once the top of the contact has been inserted into the desired hole via the above-described operations, an initial alignment correction can be performed by aligning the direction vector of the wire contact with the connector surface normal vector. This alignment correction can be calculated by projecting the contact direction vector onto a plane parallel to the connector surface. The purpose of this alignment correction is to correct for contacts that are bent to one side at the top of the wire clamp. After the alignment correction, the distance from the wire clamp to the connector surface is calculated using the estimated position of the hole. This distance can be used to determine the allowable distance that the wire clamp can move without penetrating the connector beyond the connector boundary. Once the allowable distance is determined, the wire clamp can be commanded to insert the contact into the target hole by traveling the allowable distance along the connector surface normal vector / wire clamp direction vector, or the contact can be inserted into the target hole until an insertion force threshold (such as 12 Newtons) is reached.

[0081] If the movement is performed without exceeding the force threshold, the wire clamp is permissibly close to the connector and therefore an alternative means of pushing the contact and wire forward may be required to achieve full insertion. This may be performed via a "re-clamp" operation. If the movement of the wire clamp is stopped due to the force threshold, the distance the wire clamp has traveled is used to determine whether the wire may have been inserted. If the distance the wire clamp has traveled is less than a specified parameterized distance, it may be confirmed that the wire contact was not properly and fully inserted and seated within the connector, such that the force threshold was reached for a different reason, such as an obstruction within the connector. If the calculated distance indicates that the wire may have been inserted and seated due to exceeding the specified parameterized distance, a pull test may be performed to determine the success or failure of the insertion process.

[0082] If the distance moved by the wire clamp holding the wire contact indicates that the insertion has not been completed, the number of attempted insertions can be checked to identify any problems. If the number of attempted insertions exceeds the parameterized value, the insertion process can end and the insertion process can be considered a failure. If the number of attempted insertions is not higher than the parameterized value, a visual alignment operation can be performed to align the contact in the hole, and the insertion process can be repeated. The visual alignment method used in the example embodiment can be the same as the method used to estimate the contact vector. Differences may include: the image analysis is limited to a small image window (e.g., 100×100 pixels) near the top of the robot wire clamp, and the length of the visible portion of the contact can be determined by intersecting the contact vector with the contact surface. The resulting vector can be used to calculate a correction movement of the robot end effector in the coordinate system. The correction movement can be calculated so that the correction movement aligns the contact vector with the surface normal vector of the connector surface. This surface may be known due to the above-mentioned mask optimization process.

[0083] When the wire clamp advances the wire contact as far as the clamp is permitted to move (e.g., when the clamp is as close to the connector as permitted) and the wire contact is not yet fully inserted and seated, the clamp can perform a re-grip operation. For the re-grip and insertion operation, the clamp reduces the friction holding the wire in the clamp and moves back a parameterized distance (e.g., 4 mm). The clamp then re-grips the wire and pushes it further into the connector by returning it to its original position. If the clamp reaches its closest permissible proximity to the connector, or if an insertion force threshold is reached, the clamp stops moving.

[0084] Once the wire is fully inserted and seated, a pull test can be performed to confirm the seating of the wire contact within the connector. For the pull test, the wire clamp is withdrawn from the wire, away from the connector, until a specific distance, or force threshold, is reached. If the force threshold is reached before the specified distance, the wire contact is confirmed to be properly seated. If the specified distance is reached before the force threshold is achieved, the wire insertion has failed because it is determined that the wire is not fully seated.

[0085] Figure 13 The process flow of the method for aligning a wire contact with a target hole of a connector and inserting the wire contact into the target hole according to the above-mentioned description is illustrated. As shown, the process begins at 300, where the connector is installed and prepared to receive the wire contact in the target hole. At 302, the wire including the wire contact is loaded into the wire clamp (e.g., 303) of the end effector (e.g., 100) of the robot. Figure 4 In 304, such as using Figure 4 The cameras 102 and 104 perform initial vision processing. Then, in 306, the end effector can move the wire clamp and the wire into a ready position. In 308, alignment of the wire contact with the target hole is performed. As shown, in 310, a correction transformation is calculated using the process described in detail above. If the correction transformation is below a threshold amount, the alignment is considered complete and the process continues. If the correction transformation is above the threshold, in 312, the number of correction transformations that have been completed is checked, and if the correction transformation is below the threshold number, a correction transformation occurs in 314. If the number of correction transformations exceeds a predefined number, failure can be confirmed in 330. However, if in 314 the correction transformation is successful and results in alignment below the threshold correction transformation, then in 316 the wire clamp holding the wire contact is moved toward the connector.

[0086] Figure 13The process continues to 318, where the wire clamp is moved toward the connector to a position where misalignment will cause contact without damaging the wire contact or the connector. If the force feedback on the wire clamp exceeds a threshold, it can be determined that the wire contact is not aligned with the target hole, so the wire contact is retracted in 320 and alignment is performed again in 308. If the force feedback at the wire clamp is below the threshold, then in 322, it is confirmed that the wire contact is aligned with the target hole. Then, in 324, the wire clamp can advance the wire contact to the maximum distance allowed by the wire clamp. The maximum distance can be the distance at which the wire clamp does not contact the connector or contacts the connector without damaging the connector. In this way, the wire clamp is prevented from damaging the connector during the insertion of the wire contact.

[0087] If the maximum distance of the wire clamp is reached before the insertion force threshold is observed on the wire clamp, the wire clamp can release the wire contact while the end effector retracts and, in 322, the wire clamp re-grips the wire and advances the wire contact further into the target hole of the connector. Once the insertion force threshold is reached, it is confirmed in 326 whether the wire contact has been advanced at least the minimum distance into the connector. If the distance the wire contact has advanced is less than the threshold distance, an obstruction may have occurred within the connector and alignment may be attempted again in 322. If the insertion distance is met in 326, a pull test can be performed. The pull test uses the wire clamp to pull the wire out of the connector. If the force on the wire clamp meets the pull test threshold and the contact does not move away from the connector (e.g., above a small threshold distance), the wire contact is confirmed to be fully inserted in 334 and success is indicated in 336.

[0088] Figure 14 The present invention is a flow chart of a method for aligning a wire contact with a target hole of a connector and inserting the wire contact into the target hole. The illustrated embodiment includes: at 400, obtaining captured images, namely, images of a wire clamp of the end effector, from at least two image capture devices attached to an end effector. At 402, processing the captured images to establish a correction transformation for aligning the wire contact with the target hole of the connector. At 404, the robot translates the end effector to move the wire clamp along the correction transformation. At 406, the robot is caused to advance the end effector until contact is made with a surface of the connector. As shown in 407, based on force measurements at the end effector, it is determined whether the wire contact is within the target hole of the connector. At 408, the robot is caused to move the end effector toward the connector by a predetermined additional amount greater than a predetermined distance. At 410, force feedback at the wire clamp is used to confirm whether the alignment is correct based on the wire contact moving by the predetermined additional amount greater than the predetermined distance.

[0089] As mentioned above, Figure 6 、 Figure 7 、 Figure 10 、 Figure 12 and Figure 14 Flowcharts of systems 30, methods, and computer program products according to example embodiments of the present disclosure are illustrated. It should be understood that each block in the flowchart and the combination of blocks in the flowchart can be implemented by various means such as hardware, firmware, processors, circuits, and / or other devices associated with software execution including one or more computer program instructions. For example, one or more of the above processes can be implemented by computer program instructions. In this regard, the computer program instructions that implement the above processes can be stored by the memory 38 of the system 30 using an embodiment of the present disclosure and executed by the processing circuit 36 ​​of the system 30. As will be understood, any such computer program instructions can be loaded onto a computer or other programmable device (e.g., hardware) to produce a machine so that the resulting computer or other programmable device implements the functions specified in the flowchart blocks. These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable device to function in a specific manner so that the instructions stored in the computer-readable memory produce a product including an article that implements the functions specified in the flow chart blocks when executed. Computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operations to be performed on the computer or other programmable device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device provide operations for implementing the functions specified in the flow blocks.

[0090] Therefore, the blocks in the flowchart support combinations of means for performing the specified functions and combinations of operations for performing the specified functions. It will also be understood that one or more blocks in the flowchart and combinations of blocks in the flowchart may be implemented by a computer system based on special-purpose hardware that performs the specified functions or a combination of special-purpose hardware and computer instructions.

[0091] In some embodiments, some of the above operations may be modified or further amplified. In addition, in some embodiments, additional optional operations may be included. Modifications, additions, or amplifications to the above operations may be performed in any order or in any combination.

[0092] Item 1. A system for inserting a wire contact into a target hole of a connector, the system comprising: a robot having an end effector, wherein the end effector includes a wire clamp and at least two image capture devices fixed to the end effector; a computing device, wherein the computing device is configured to: process images captured by the image capture device to establish a correction transformation that aligns the wire contact with the target hole of the connector in the connector surface; cause the robot to translate the end effector to move the wire clamp along the correction transformation; cause the robot to advance the end effector until contact is made with the connector surface; based on force measurement, confirm whether the wire contact is inside the target hole of the connector; cause the robot to advance the end effector to move the wire contact toward the connector a predetermined additional amount; and based on the wire contact moving the predetermined additional amount, confirm whether the alignment is correct using force feedback at the wire clamp.

[0093] Clause 2. A system according to clause 1, wherein the computing device configured to confirm whether the wire contact is inside the target hole of the connector based on force measurement is configured to: confirm that the wire contact is not aligned with the target hole of the connector in response to the force observed at the wire clamp being higher than a predefined value; and confirm that the wire contact is aligned with the target hole of the connector in response to the force observed at the wire clamp being lower than the predefined value.

[0094] Item 3. A system according to Item 1, wherein the computing device configured to process the image captured by the image capture device to establish a correction transformation for aligning the wire contact with the target hole of the connector is configured to: identify the top end of the wire contact and the direction along which the wire contact extends within the processed image; and establish a correction transformation based on the processed image to align the top end of the wire contact and the direction along which the wire contact extends with the target hole on the axis of the three-dimensional coordinate system of the end effector of the robot.

[0095] Clause 4. A system according to clause 1, wherein the computing device is further configured to: determine a maximum distance that the wire clamp can advance toward the connector; cause the robot to advance the end effector to move the wire clamp toward the connector; as the robot is caused to advance the end effector to move the wire clamp toward the connector, in response to force feedback on the wire clamp satisfying an insertion force value, determine that the wire contact is fully inserted into the connector; and as the robot is caused to advance the end effector to move the wire clamp toward the connector, in response to force feedback on the wire clamp failing to satisfy the insertion force value before reaching the maximum distance that the wire clamp can advance toward the connector, determine that the wire contact is not fully inserted into the connector.

[0096] Clause 5. A system according to Clause 4, wherein, in response to the computing device determining that the wire contact is fully inserted into the connector, the computing device is further configured to: cause the robot to retract the end effector to move the wire clamp away from the connector; and in response to the wire clamp moving a pulling distance without the force feedback on the wire clamp exceeding the pulling test value, confirm that the wire contact is incorrectly inserted; and in response to the force feedback on the wire clamp exceeding the pulling test value without the wire clamp moving the pulling distance, confirm that the wire contact is correctly inserted.

[0097] Clause 6. The system of clause 4, wherein, in response to the computing device determining that the wire contact is not fully inserted into the connector, the computing device is further configured to: cause the wire clamp to release its grip on the wire contact; cause the robot to retract the end effector to move the wire clamp away from the connector; cause the wire clamp to re-grip the wire contact; and cause the robot to advance the end effector to move the wire clamp and the wire contact toward the connector.

[0098] Clause 7. The system of clause 1, wherein force feedback at the wire clamp is established based on a force sensor between the robot and the wire clamp.

[0099] Clause 8. A system according to clause 1, wherein the computing device is further configured to: identify the orientation of the visible portion of the wire contact in the image captured by the image capture device; and use the orientation of the above-mentioned wire contact to calculate the corrective movement in the coordinate system of the end effector of the robot.

[0100] Item 9. A method for aligning a wire contact with a target hole of a connector and inserting the wire contact into the target hole, the method comprising: obtaining captured images of a wire clamp of the end effector from at least two image capture devices attached to the end effector of a robot; processing the images captured by the image capture devices to establish a correction transformation that aligns the wire contact with the target hole of the connector in the connector surface; causing the robot to translate the end effector so that the wire clamp moves along the correction transformation; causing the robot to advance the end effector until contact is made with the connector surface; confirming whether the wire contact is inside the target hole of the connector based on force measurement; causing the robot to advance the end effector to move the wire contact toward the connector by a predetermined additional amount; and confirming whether the alignment is correct using force feedback at the wire clamp based on the wire contact moving the predetermined additional amount.

[0101] Clause 10. A method according to Clause 9, wherein confirming whether the wire contact is inside the target hole of the connector based on force measurement includes: confirming that the wire contact is not aligned with the target hole of the connector in response to the force observed at the wire clamp being higher than a predefined value; and confirming that the wire contact is aligned with the target hole of the connector in response to the force observed at the wire clamp being lower than the predefined value.

[0102] Item 11. A method according to Item 9, wherein processing the image captured by the image capture device to establish a correction transformation that aligns the wire contact with the target hole of the connector includes: identifying the top end of the wire contact and the direction along which the wire contact extends within the processed image; and establishing a correction transformation based on the processed image that aligns the top end of the wire contact and the direction along which the wire contact extends with the target hole on the axis of the three-dimensional coordinate system of the end effector of the robot.

[0103] Item 12. The method according to Item 9 further includes: determining a maximum distance that the wire clamp can advance toward the connector; causing the robot to advance the end effector to move the wire clamp toward the connector; as the robot is caused to advance the end effector to move the wire clamp toward the connector, in response to force feedback on the wire clamp satisfying an insertion force value, determining that the wire contact is fully inserted into the connector; and as the robot is caused to advance the end effector to move the wire clamp toward the connector, in response to force feedback on the wire clamp failing to satisfy the insertion force value before reaching the maximum distance that the wire clamp can advance toward the connector, determining that the wire contact is not fully inserted into the connector.

[0104] Clause 13. A method according to Clause 12, wherein, in response to determining that the wire contact is fully inserted into the connector, the method further includes: causing the robot to retract the end effector to move the wire clamp away from the connector; confirming that the wire contact is incorrectly inserted in response to the wire clamp moving a pulling distance without the force feedback on the wire clamp exceeding the pulling test value; and confirming that the wire contact is correctly inserted in response to the force feedback on the wire clamp exceeding the pulling test value without the wire clamp moving the pulling distance.

[0105] Clause 14. The method according to Clause 12, in response to determining that the wire contact is not fully inserted into the connector, the method further comprising: causing the wire clamp to release its grip on the wire contact; causing the robot to retract the end effector to move the wire clamp away from the connector; causing the wire clamp to re-grip the wire contact; and causing the robot to advance the end effector to move the wire clamp and the wire contact toward the connector.

[0106] Clause 15. The method of clause 9, wherein force feedback at the wire clamp is established based on a force sensor between the robot and the wire clamp.

[0107] Clause 16. The method of clause 9, further comprising: identifying, in an image captured by the image capture device, an orientation of a visible portion of the wire contact; and calculating a corrective movement in a coordinate system of the end effector of the robot using the orientation of the wire contact.

[0108] Clause 17. A computer program product for aligning a wire contact with a target hole defined by a connector and inserting the wire contact into the target hole, the computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions for: obtaining captured images of a wire clamp of an end effector of a robot from at least two image capture devices attached to the end effector; processing the images captured by the image capture devices to establish a method for inserting the wire contact into the target hole; causing the robot to translate the end effector to move the wire clamp along the correction transformation; causing the robot to advance the end effector until contact is made with the connector surface; confirming whether the wire contact is inside the target hole of the connector based on force measurement; causing the robot to advance the end effector to move the wire contact toward the connector a predetermined additional amount; and confirming whether the alignment is correct using force feedback at the wire clamp based on the wire contact moving the predetermined additional amount.

[0109] Clause 18. A computer program product according to clause 17, wherein the program code instructions for confirming whether the wire contact is inside the target hole of the connector based on force measurement include program code instructions for performing the following steps: confirming that the wire contact is not aligned with the target hole of the connector in response to the force observed at the wire clamp being higher than a predefined value; and confirming that the wire contact is aligned with the target hole of the connector in response to the force observed at the wire clamp being lower than the predefined value.

[0110] Clause 19. A computer program product according to clause 17, wherein the program code instructions for processing the image captured by the image capture device to establish a correction transformation for aligning the wire contact with the target hole of the connector include program code instructions for performing the following steps: identifying the top end of the wire contact and the direction along which the wire contact extends within the processed image; and establishing a correction transformation based on the processed image to align the top end of the wire contact and the direction along which the wire contact extends with the axis of the three-dimensional coordinate system of the end effector of the robot.

[0111] Clause 20. A computer program product according to clause 17, wherein the computer program product further comprises program code instructions for performing the following steps: determining a maximum distance that the wire clamp can advance toward the connector; causing the robot to advance the end effector to move the wire clamp toward the connector; as the robot is caused to advance the end effector to move the wire clamp toward the connector, in response to force feedback on the wire clamp satisfying an insertion force value, determining that the wire contact is fully inserted into the connector; and as the robot is caused to advance the end effector to move the wire clamp toward the connector, in response to force feedback on the wire clamp failing to satisfy the insertion force value before reaching the maximum distance that the wire clamp can advance toward the connector, determining that the wire contact is not fully inserted into the connector.

[0112] Those skilled in the art who have benefited from the teachings set forth in the above description and the associated drawings will appreciate many modifications and other embodiments of the invention set forth herein. Therefore, it is to be understood that this patent application is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. In addition, although the above description and the associated drawings describe example embodiments in the context of certain example combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, it is also contemplated that different combinations of elements and / or functions that are different from the elements and / or functions explicitly described above may be set forth in some of the appended claims. Although specific terms are employed herein, they are used only in a general descriptive sense, and not for limiting purposes.

Claims

1. A system (30) for inserting a wire contact into a target hole of a connector (10), the system comprising: A robot (44) having an end effector (100), wherein the end effector (100) includes a wire clamp (108) and at least two image capture devices (32, 102, 104) secured to the end effector (100); A computing device (34), wherein the computing device (34) is configured to: processing images captured by the image capture device to create a correction transformation that aligns the wire contact with the target aperture of the connector (10) in a connector surface; causing the robot (44) to translate the end effector (100) to move the wire clamp (108) along the correction transformation; causing the robot (44) to advance the end effector (100) until contact is made with the connector surface; confirming whether the wire contact is inside the target hole of the connector (10) based on force measurement; causing the robot (44) to advance the end effector (100) to move the wire contact toward the connector (10) a predetermined additional amount; and Based on the wire contact moving the predetermined additional amount, force feedback at the wire clamp is used to confirm whether the alignment is correct, wherein the computing device (34) configured to process images captured by the image capture device (32, 102, 104) to create a corrective transformation for aligning the wire contact with the target aperture of the connector is configured to: identifying, within the processed image, a tip of the wire contact and a direction along which the wire contact extends; and A correction transformation is established based on the processed image to align the tip of the wire contact and the direction along which the wire contact extends with the target hole on the axis of the three-dimensional coordinate system of the end effector (100) of the robot (44).

2. The system (30) according to claim 1, wherein The computing device (34) configured to confirm whether the wire contact is inside the target hole of the connector (10) based on force measurement is configured to: confirming that the wire contact is not aligned with the target aperture of the connector (10) in response to a force observed at the wire clamp (108) being above a predefined value; as well as Alignment of the wire contact with the target aperture of the connector (10) is confirmed in response to the force observed at the wire clamp (108) being below the predefined value.

3. The system (30) according to claim 1, wherein The computing device (34) is further configured to: determining a maximum distance the wire clamp (108) can advance toward the connector (10); causing the robot (44) to advance the end effector (100) to move the wire clamp (108) toward the connector (10); determining that the wire contact is fully inserted into the connector (10) in response to force feedback on the wire clamp (108) satisfying an insertion force value, as the robot (44) is caused to advance the end effector (100) to move the wire clamp (108) toward the connector (10); as well as As the robot (44) is caused to advance the end effector (100) to move the wire clamp (108) toward the connector (10), in response to force feedback on the wire clamp (108) failing to satisfy the insertion force value before reaching the maximum distance that the wire clamp (108) can advance toward the connector (10), it is determined that the wire contact is not fully inserted into the connector (10).

4. The system (30) according to claim 3, wherein In response to the computing device (34) determining that the wire contact is fully inserted into the connector (10), the computing device (34) is further configured to: causing the robot (44) to retract the end effector (100) to move the wire clamp (108) away from the connector (10); In response to the wire clamp (108) moving a pull distance without force feedback on the wire clamp (108) exceeding a pull force test value, confirming that the wire contact is not correctly inserted; and In response to force feedback on the wire clamp (108) exceeding a pull test value without the wire clamp (108) moving the pull distance, the wire contact is confirmed to be properly inserted.

5. The system (30) of claim 3, wherein, in response to the computing device (34) determining that the wire contact is not fully inserted into the connector (10), the computing device (34) is further configured to: causing the wire clamp (108) to release its grip on the wire contact; causing the robot (44) to retract the end effector (100) to move the wire clamp (108) away from the connector; causing the wire clamp (108) to re-clamp the wire contact piece; as well as The robot (44) is caused to advance the end effector (100) to move the wire clamp (108) and the wire contact toward the connector (10).

6. The system (30) according to claim 1, wherein The computing device (34) is further configured to: identifying, in an image captured by the image capture device (32, 102, 104), an orientation of a visible portion of the wire contact; and A correction movement in the coordinate system of the end effector (100) of the robot (44) is calculated using the orientation of the wire contact.

7. A method for aligning a wire contact with a target hole of a connector (10) and inserting the wire contact into the target hole, the method comprising: Obtaining captured images of a wire clamp (108) of an end effector (100) of a robot (44) from at least two image capture devices (32, 102, 104) attached to the end effector (100); processing images captured by the image capture device (32, 102, 104) to create a correction transformation that aligns the wire contact with the target aperture of the connector in a connector surface; causing the robot (44) to translate the end effector (100) to move the wire clamp (108) along the correction transformation; causing the robot (44) to advance the end effector (100) until contact is made with the connector surface; confirming whether the wire contact is inside the target hole of the connector (10) based on force measurement; causing the robot to advance the end effector (100) to move the wire contact a predetermined additional amount toward the connector (10); and Based on the wire contact being moved the predetermined additional amount, force feedback at the wire clamp (108) is used to confirm whether the alignment is correct, Wherein processing the image captured by the image capture device (32, 102, 104) to establish a correction transformation for aligning the wire contact with the target hole of the connector (10) includes: identifying, within the processed image, a tip of the wire contact and a direction along which the wire contact extends; and A correction transformation is established based on the processed image to align the tip of the wire contact and the direction in which the wire contact extends with the target hole on the axis of the three-dimensional coordinate system of the end effector (100) of the robot (44).

8. The method according to claim 7, wherein: Based on force measurement, it is confirmed that the wire contacts are Whether the interior of the target hole of the connector (10) includes: confirming that the wire contact is not aligned with the target aperture of the connector (10) in response to a force observed at the wire clamp (108) being above a predefined value; and Alignment of the wire contact with the target aperture of the connector (10) is confirmed in response to the force observed at the wire clamp (108) being below the predefined value.

9. The method according to claim 7, further comprising: determining a maximum distance the wire clamp (108) can advance toward the connector (10); causing the robot (44) to advance the end effector (100) to move the wire clamp (108) toward the connector (10); determining that the wire contact is fully inserted into the connector (10) in response to force feedback on the wire clamp (108) satisfying an insertion force value, as the robot (44) is caused to advance the end effector (100) to move the wire clamp (108) toward the connector (10); as well as As the robot (44) is caused to advance the end effector (100) to move the wire clamp (108) toward the connector (10), in response to force feedback on the wire clamp (108) failing to satisfy the insertion force value before reaching the maximum distance that the wire clamp (108) can advance toward the connector (10), it is determined that the wire contact is not fully inserted into the connector (10).

10. The method according to claim 9, wherein: In response to determining that the wire contact is fully inserted into the connector (10), the method further includes: causing the robot (44) to retract the end effector (100) to move the wire clamp (108) away from the connector (10); In response to the wire clamp (108) moving a pull distance without force feedback on the wire clamp (108) exceeding a pull force test value, confirming that the wire contact is not correctly inserted; and In response to force feedback on the wire clamp (108) exceeding a pull test value without the wire clamp (108) moving the pull distance, the wire contact is confirmed to be properly inserted.

11. The method of claim 9, in response to determining that the wire contact is not fully inserted into the connector (10), the method further comprising: causing the wire clamp (108) to release its grip on the wire contact; causing the robot (44) to retract the end effector (100) to move the wire clamp (108) away from the connector (10); causing the wire clamp (108) to re-clamp the wire contact piece; as well as The robot (44) is caused to advance the end effector (100) to move the wire clamp (108) and the wire contact toward the connector (10).

12. The method according to claim 7, further comprising: identifying an orientation of a visible portion of the wire contact in an image captured by the image capture device (32, 102, 104); as well as A correction movement in a coordinate system of the end effector (100) of the robot is calculated using the orientation of the wire contact.

Images