Robot system, method and system for inserting wire and computer program product

The camera system generates and processes connector images and creates masks for automated wire insertion, solving the problem of wire insertion accuracy and speed in aircraft manufacturing, achieving adaptability and efficient insertion of different connectors.

CN119070116APending Publication Date: 2024-12-03THE BOEING CO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410625944.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-30
Filing Date
2024-05-20
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

During aircraft manufacturing, it is difficult to ensure high accuracy and speed when automating wire insertion into connectors. The prior art relies on machine learning models, but accuracy and speed are difficult to ensure under different connectors and ambient lighting conditions.

Method used

The camera system is used to generate images of the connector, and the final mask is generated by creating an initial mask and applying thresholding techniques to guide the end effector to accurately insert wires, reducing dependence on ambient light and increasing insertion speed.

Benefits of technology

Improves accuracy and speed of wire insertion connectors, reduces computational redundancy, is suitable for connectors of different shapes and sizes, reduces manufacturing costs, and improves assembly speed and quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119070116A_ABST
    Figure CN119070116A_ABST
Patent Text Reader

Abstract

The invention relates to a robot system, a method and system for inserting a wire, and a computer program product. An image of the connector is generated using a camera system connected to the end effector. The end effector is in a connector attitude facing the connector. A region in the image surrounding the connector is selected. An initial mask is created using the image and the region. The initial mask includes a background region and a connector region. A segmentation mask is created using the initial mask and the image, wherein the segmentation mask includes an edge for the connector. The thresholding is applied to the segmentation mask to restore pixels of the connector and generate a final mask. The final mask is used to insert the wire into the connector.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates generally to manufacturing assemblies, and in particular to inserting wires into connectors using connector singulation to create masks for the connectors. Background Art

[0002] When manufacturing commercial aircraft, wires are usually installed in bundles to form wire bundles. The process involves feeding and cutting wires to the desired length by a machine. Wire contacts are attached to the wires. These wires are placed in position for assembly using a robot. These wires can then be manipulated by a robot with an end effector to form a wire bundle. The robot end effector can be or can have a wire contact insertion tool.

[0003] The end effector picks up the wire, moves the wire, and inserts the wire contacts of the wire into the electrical connector. When automating the assembly of the wire bundle, accuracy in inserting the ends of the wire into the connector with a high level of precision is required. One automated approach for assembling wire bundles involves the use of a machine learning model, such as a neural network, which is trained to move the wire and insert the contacts into the connector. The use of a machine learning model can be specific to different types of connectors and settings for wire insulation.

[0004] Therefore, it would be desirable to have methods and apparatus that take into account at least some of the above-mentioned issues as well as other possible issues.For example, it would be desirable to have methods and apparatus that overcome the problem of obtaining a desired level of accuracy when inserting wire contacts into connectors. Summary of the invention

[0005] An embodiment of the present disclosure provides a robotic system. The robotic system includes an end effector; a camera system connected to the end effector; and a controller. The controller is configured to generate an image of a connector using the camera system through the end effector in a connector posture facing the connector. The controller is configured to select an area surrounding the connector in the image. The controller is configured to generate an initial mask using the image and the area. The initial mask includes a background area and a connector area. The controller is configured to create a segmentation mask using the initial mask and the image. The segmentation mask includes an edge for the connector. The controller is configured to apply thresholding to the segmentation mask to restore pixels of the connector and generate a final mask.

[0006] Another embodiment of the present disclosure provides a method for inserting a wire. An image of a connector is generated using a camera system connected to an end effector. The end effector is in a connector posture facing the connector. A region in the image that surrounds the connector is selected. An initial mask is created using the image and the region. The initial mask includes a background region and a connector region. A segmentation mask is created using the initial mask and the image, wherein the segmentation mask includes an edge for the connector. Thresholding is applied to the segmentation mask to restore pixels of the connector and a final mask is generated. The final mask is used to insert the wire into the connector.

[0007] Another embodiment of the present disclosure provides a wire assembly system, which includes a computer system and a controller in the computer system. The controller is configured to use a camera system to generate an image of the connector through an end effector in a connector posture facing the connector. The controller is configured to select an area surrounding the connector in the image. The controller is configured to create an initial mask using the image and the area. The initial mask image includes a background area and a connector area. The controller is configured to create a segmentation mask using the initial mask and the image. The segmentation mask includes an edge for the connector. The controller is configured to apply thresholding to the segmentation mask to restore the pixels of the connector and generate a final mask. The controller is configured to insert the wire into the connector using the end effector and the final mask.

[0008] Another embodiment of the present disclosure provides a computer program product for wire assembly. The computer program product includes a computer-readable storage medium having program instructions embodied therein. The program instructions may be executed by a computer system to cause the computer system to generate an image of a connector using a camera system connected to an end effector in a connector posture facing the connector. The program instructions may be executed by a computer system to cause the computer system to select an area surrounding the connector in the image. The program instructions may be executed by a computer system to cause the computer system to create an initial mask using the image and the area. The initial mask image includes a background area and a connector area. The program instructions may be executed by a computer system to cause the computer system to create a segmentation mask using the initial mask and the image. The segmentation mask includes an edge for the connector. The program instructions may be executed by a computer system to cause the computer system to apply thresholding to the segmentation mask to restore the pixels of the connector and generate a final mask. The program instructions may be executed by a computer system to cause the computer system to insert the wire into the connector using the end effector and the final mask.

[0009] The features and functions can be achieved independently in various embodiments of the present disclosure or may be combined in yet other embodiments in which further details can be seen with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The novel features believed characteristic of the illustrative embodiments are set forth in the appended claims. However, the illustrative embodiments together with the preferred mode of use, further objects and features thereof will be best understood by reference to the following detailed description of illustrative embodiments of the present disclosure when read in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a pictorial representation of a network of data processing systems implementing the illustrative embodiments;

[0012] Figure 2 is an illustration of a block diagram of a manufacturing environment according to an illustrative embodiment;

[0013] Figure 3 is a diagram of a data flow for generating a final mask for use in inserting a wire according to an illustrative embodiment;

[0014] Figure 4 is an illustration of an end effector for a robotic arm according to an illustrative embodiment;

[0015] Figure 5 is an illustration of a flow chart of a process for inserting a set of wires into holes in a connector according to an illustrative embodiment;

[0016] Figure 6 is a flow chart of a process for inserting a wire according to an illustrative embodiment;

[0017] Figure 7 is a flow chart of a process for inserting a wire according to an illustrative embodiment;

[0018] Figure 8 is an illustration of a flow chart of a process for inserting a set of wires according to an illustrative embodiment;

[0019] Fig. 9 is an illustration of a flow chart of a process for inserting a second wire according to an illustrative embodiment;

[0020] Fig.10 is an illustration of a flow chart of a process for creating an initial mask according to an illustrative embodiment;

[0021] Fig.11 is an illustration of a flow chart of a process for creating an initial mask according to an illustrative embodiment;

[0022] Fig.12 is a flowchart of a process for selecting a region according to an illustrative embodiment;

[0023] Fig.13 is an illustration of a block diagram of a data processing system according to an illustrative embodiment;

[0024] Fig.14 is a schematic diagram of an aircraft manufacturing and maintenance method according to an illustrative embodiment;

[0025] Fig.15 is an illustration of a block diagram of an aircraft in which an illustrative embodiment may be implemented; and

[0026] Fig.16 is an illustration of a block diagram of a product management system in accordance with an illustrative embodiment. DETAILED DESCRIPTION

[0027] The illustrative embodiments recognize and take into account one or more different considerations. Increasing the accuracy and speed of inserting wires into connectors can involve many different factors. Determining the pose and position of the wire contacts is one factor in automating the process of robotic wire insertion. Another consideration involves estimating the position of the connector in the image as part of automating the process of robotic wire insertion.

[0028] One way that connectors can be located involves using a vision system to detect the color of the connector prior to wire insertion. The tuning parameters for this type of connector detection are sensitive to ambient lighting conditions. Thus, it is desirable to detect the position of the connector in a manner that is invariant to color and ambient lighting conditions.

[0029] Thus, illustrative embodiments provide methods, apparatus, systems, and computer program products for detecting connectors and, in particular, for segmenting images of connectors for use in a robotic wire insertion process. In illustrative embodiments, a connector detection system generates images from a first-person perspective. In other words, images are generated from the perspective of a robot end effector. In an illustrative embodiment, a fixed stereo camera system can be positioned near the end of an end effector on a robot arm to enable self-centered vision. In addition, a lighting system is used to illuminate the end effector. The lighting system can be selected to reduce or eliminate ambient light detected by the fixed stereo camera system.

[0030] Thus, images are generated from the perspective of the end effector and the illumination reduces issues with ambient lighting conditions. These types of images can be processed to automatically segment the connector. Additionally, removing redundant calculations can reduce the process to increase the speed of wire insertion into the connector.

[0031] In an illustrative embodiment, a robotic system is able to segment an image of a connector and generate a final connector mask that can be used to insert a wire into the connector. In an illustrative embodiment, a camera system is connected to an end effector and can generate an image for segmentation to generate a mask for inserting a wire.

[0032] Referring now to the drawings, and in particular to Figure 1, depicts a pictorial representation of a network of data processing systems in which the illustrative embodiments may be implemented. Network data processing system 100 is a network of computers in which the illustrative embodiments may be implemented. Network data processing system 100 contains network 102, which is a medium used to provide communications links between different devices and computers connected together within network data processing system 100. Network 102 may include connections such as wire, wireless communication links, or fiber optic cables.

[0033] In the depicted embodiment, server computer 104 and server computer 106 are connected to network 102 along with storage unit 108. In addition, client devices 110 are connected to network 102. As depicted, client devices 110 include client computer 112, client computer 114, and client computer 116. Client devices 110 may be, for example, computers, workstations, or network computers. In the depicted embodiment, server computer 104 provides information, such as boot files, operating system images, and applications, to client devices 110. In addition, client devices 110 may also include other types of client devices, such as robotic arm 118, tablet computer 120, and smart glasses 122. In this illustrative embodiment, server computer 104, server computer 106, storage unit 108, and client devices 110 are network devices connected to network 102, where network 102 is a communication medium for these network devices. Some or all of client devices 110 may form an Internet of Things (IoT), where these physical devices may be connected to network 102 and exchange information with each other through network 102.

[0034] In this embodiment, client device 110 is a client to server computer 104. Network data processing system 100 may include additional server computers, client computers, and other devices not shown. Client device 110 connects to network 102 using at least one of a wired connection, a fiber optic connection, or a wireless connection.

[0035] Program instructions located in network data processing system 100 may be stored on a computer recordable storage medium and downloaded to the data processing system or other devices for use. For example, program instructions may be stored on a computer recordable storage medium on server computer 104 and downloaded to client device 110 via network 102 for use on client device 110.

[0036] In the depicted embodiment, network data processing system 100 is the Internet, where network 102 represents a global collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol suite to communicate with each other. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, which consist of thousands of commercial, government, educational, and other computer systems that route data and messages. Of course, network data processing system 100 may also be implemented using many different types of networks. For example, network 102 may include at least one of the Internet, an intranet, a local area network (LAN), a metropolitan area network (MAN), or a wide area network (WAN). Figure 1 It is intended as an example, not as an architectural limitation for the different illustrative implementations.

[0037] As used herein, when "a plurality" is used with reference to an item, it refers to one or more items. For example, "a plurality of different types of networks" is one or more different types of networks.

[0038] In addition, when used with a list of items, the phrase "at least one of" means that different combinations of one or more of the listed items can be used, and only one of each item in the list may be required. In other words, "at least one of" means that any combination of items and multiple items in the list can be used, but not all items in the list are required. An item can be a specific object, thing, or category.

[0039] For example, but not limited to, "at least one of item A, item B, and item C" may include item A, item A and item B, or item B. The embodiment may also include item A, item B, and item C, or item B and item C. Of course, any combination of these items may exist. In some illustrative embodiments, "at least one" may be, for example, but not limited to, two items A; one item B; and ten items C; four items B and seven items C; or other suitable combinations.

[0040] In this illustrative embodiment, the robot controller 130 is located in the server computer 104. The robot controller 130 can control the operation of the robot arm 118. In this embodiment, the robot controller 130 controls the operation of the robot arm 118 to perform manufacturing operations. For example, the robot arm 118 can be used to assemble a wire bundle. As described, as part of the process for assembling the wire bundle, the robot arm 118 can insert wire contacts for the wires into the connector.

[0041] As depicted, the robotic controller 130 may control the robotic arm 118 to send the image 132 to the robotic controller 130 via the network 102. The robotic arm 118 has an arm 135 connected to an end effector 136. In this embodiment, the robotic arm 118 has 6 degrees of freedom.

[0042] In this embodiment, a stereo camera 134 is connected to an end effector 136 of the robotic arm 118. As depicted, the end effector 136 is gripping a wire 140 having a wire contact 138. As shown, the end effector 136 has a posture facing the connector 133. Therefore, the stereo camera 134 also has a posture facing the connector 133 to generate an image 132 of the connector 133. This configuration of the stereo camera 134 results in the camera being a first-person perspective camera system.

[0043] The robot controller 130 controls the stereo camera 134 to generate and transmit an image 132 of the connector 133, wherein the end effector 136 is in a posture facing the connector 133. In this illustrative embodiment, the robot controller 130 can analyze the image 132 to generate a mask 141 of the connector 133. In this embodiment, the analysis performed by the robot controller 130 includes receiving a selection of an area within the image 132 for generating an initial mask 142. The initial mask 142 has a background area 144 and a connector area 146. The connector area 146 in the initial mask 142 is defined by selecting the area within the image 132. In one illustrative embodiment, the robot controller 130 can also determine or calculate an inner connector area 148 within the connector area 146.

[0044] The robotic controller 130 creates a segmentation mask 150 using the initial mask 142 and the image 132. The robotic controller 130 uses a segmentation process that analyzes the initial mask 142 in the image 132 to identify pixels in the image 132 that are considered to be part of the connector 133. The result is a mask 141.

[0045] The robot controller 130 may control the movement of the arm 135 to move the end effector 136 together with the wire contact 138 , thereby inserting the wire contact 138 into the hole in the connector 133 .

[0046] In this embodiment, the movement may be controlled by the robotic controller 130 sending instructions 152 from the server computer 104 to the robotic arm 118 over the network 102. The instructions may include commands and data for operating the robotic arm 118. The robotic arm 118 may have a processor or controller that receives the instructions 152 and processes them.

[0047] In this embodiment, the robot controller 130 also recognizes or saves the posture of the end effector 136. The stereo camera 134 is used to generate the image 132. Therefore, the end effector 136 can be controlled to pick up the second wire and return to the posture of inserting the second wire into another hole in the connector 133. By saving the posture of the end effector 136, the mask 141 can be reused.

[0048] Figure 1 The illustrative embodiment in is an example of one implementation and is not meant to limit the manner in which other illustrative embodiments may be implemented. For example, the stereo camera 134 may be replaced with a different type of camera (eg, a color camera).

[0049] See now Figure 2 , depicts an illustration of a block diagram of a manufacturing environment according to an illustrative embodiment. In this illustrative embodiment, manufacturing environment 200 includes a plurality of components that can be implemented in hardware (such as Figure 1 Components implemented in the hardware shown in the network data processing system 100 in .

[0050] As described, wire assembly system 202 in manufacturing environment 200 can perform operations to manufacture aircraft 204. In this embodiment, aircraft 204 can take many different forms. For example, aircraft 204 can be a commercial aircraft, an airplane, a rotorcraft, a tilt-rotor aircraft, a tilt-wing aircraft, a vertical take-off and landing aircraft, an electric vertical take-off and landing vehicle, a personal aircraft, and other suitable types of aircraft.

[0051] Operations may be performed to manufacture other platforms and objects in addition to aircraft 204. For example, the operations may be used to manufacture, for example, mobile platforms, fixed platforms, land-based structures, water-based structures, and space-based structures. More specifically, the platform may be a surface ship, a tank, a personnel carrier, a train, a spacecraft, a space station, a satellite, a submarine, a car, a power plant, a bridge, a dam, a house, a manufacturing facility, a building, an engine, a piece of equipment, and other suitable platforms.

[0052] In this illustrative embodiment, wire assembly system 202 includes a number of different components. As shown, wire assembly system 202 includes computer system 212, controller 214, and robot 206. As depicted, controller 214 is located in computer system 212.

[0053] The controller 214 may be implemented in software, hardware, firmware, or a combination thereof. When software is used, the operations performed by the controller 214 may be implemented in program instructions configured to run on hardware such as a processor unit. When firmware is used, the operations performed by the controller 214 may be implemented in program instructions and data and stored in a persistent memory to run on a processor unit. When hardware is used, the hardware may include circuits that operate to perform the operations in the controller 214.

[0054] In an illustrative embodiment, the hardware may take the form of at least one selected from the following: a circuit system, an integrated circuit, an application specific integrated circuit (ASIC), a programmable logic device, or some other suitable type of hardware configured to perform multiple operations. Using a programmable logic device, the device can be configured to perform multiple operations. The device can be reconfigured later or can be permanently configured to perform multiple operations. Programmable logic devices include, for example, programmable logic arrays, programmable array logic, field programmable logic arrays, field programmable gate arrays, and other suitable hardware devices. In addition, the process can be implemented in organic components integrated with inorganic components, and can be composed entirely of organic components other than humans. For example, these methods can be implemented as circuits in organic semiconductors.

[0055] Computer system 212 is a physical hardware system and includes one or more data processing systems. When more than one data processing system is present in computer system 212, those data processing systems communicate with each other using a communication medium. The communication medium may be a network. The data processing system may be selected from at least one of a computer, a server computer, a tablet computer, or some other suitable data processing system.

[0056] As depicted, computer system 212 includes multiple processor units 216 capable of executing program instructions 218 that implement the processes in the illustrative embodiments. In other words, program instructions 218 are computer-readable program instructions.

[0057] As used herein, a processor unit in the plurality of processor units 216 is a hardware device and includes hardware circuits, such as those on an integrated circuit, that respond to and process instructions and program codes for operating a computer. When the plurality of processor units 216 execute program instructions 218 for a process, the plurality of processor units 216 may be one or more processor units in the same computer or in different computers. In other words, the process may be distributed among processor units 216 on the same or different computers in the computer system 212.

[0058] In addition, the number of processor units 216 can be the same type or different types of processor units. For example, the plurality of processor units 216 can be selected from at least one of a single-core processor, a dual-core processor, a multi-processor core, a general-purpose central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and some other types of processor units.

[0059] Thus, the implemented connector segmentation system and process reduces the need for manual tuning of visual parameters. As a result, the robotic system can be more easily established and more adaptable to new connector types. In other words, connectors of different shapes and sizes can be handled by the robotic system. Furthermore, the robotic system is configured to operate even with varying ambient lighting conditions through the lighting system. As a result, wire insertion reliability is improved.

[0060] In addition, the robotic system and process reduce computational redundancy. In the illustrative embodiment, the calculated connector segmentation and hole positions do not need to be recalculated before each wire insertion in the illustrative embodiment. The state of each connector is saved and can be used at a later time. This optimization reduces computational redundancy and improves assembly time in large-scale manufacturing processes. Therefore, the illustrative embodiment reduces manufacturing costs, improves wire bundle assembly speed and quality, and increases automation for inserting wires into connectors.

[0061] In this illustrative embodiment, robot 206 includes platform 207, end effector 208, and camera system 209. In the embodiment depicted here, end effector 208 is connected to platform 207, and camera system 209 is connected to end effector 208. Camera system 209 may include one or more cameras.

[0062] When one component is "connected" to another component, the connection is a physical connection. For example, a first component (camera system 209) can be considered to be physically connected to a second component (end effector 208) by being fixed to the second component, bonded to the second component, mounted to the second component, welded to the second component, fastened to the second component, or connected to the second component in some other suitable manner. The first component can also be connected to the second component using a third component. The first component can also be considered to be physically connected to the second component by being formed as a part of the second component, an extension of the second component, or both. For example, the camera system 209 can be connected to the end effector 208 via a robotic arm. The robotic arm is part of the platform 207. The camera system 209 can be connected to the robotic arm, which in turn is connected to the end effector 208.

[0063] Platform 207 may take many different forms. For example, platform 207 may be selected from the group consisting of a robotic arm, a crawler, an unmanned aerial vehicle, and other suitable types of platforms that may be coupled to end effector 208. In other illustrative embodiments, platform 207 may be coupled to another component or platform. For example, end effector 208 may be coupled to a robotic arm that is coupled to a crawler or a mobile base.

[0064] In this illustrative example, robot 206 may perform operations on object 220 as part of a process of manufacturing aircraft 204. In this illustrative example, object 220 is connector 222 having hole 224. For example, end effector 208 may insert set of wires 225.

[0065] The controller 214 can use the connector pose 226 to determine or calculate the connector pose 226 of the end effector 208 relative to the connector 222. In the illustrative embodiment, the connector pose 226 is the position and orientation of the end effector 208. In this embodiment, three-dimensional coordinates are used to describe the position. The orientation can be relative to one or more specific features of the end effector 208.

[0066] In this illustrative embodiment, controller 214 is configured to perform a number of different operations when inserting set of wires 225 into connector 222. Prior to manipulating set of wires 225 for insertion, controller 214 is configured to perform operations to make identifying and inserting set of wires 225 more accurate, more efficient, or some combination thereof as compared to current techniques.

[0067] In the illustrative embodiment, controller 214 moves end effector 208 to connector pose 226 facing connector 222. In this embodiment, connector pose 226 may be such that hole 224 is visible in image 232. In one embodiment, the surface of connector 222 having hole 224 is planar or substantially perpendicular to an axis from end effector 208 and camera system 209 to the surface of connector 222.

[0068] In this illustrative example, controller 214 controls camera system 209 to generate image 232 of connector 222, where end effector 208 is facing connector 222 in connector pose 226. In this illustrative example, camera system 209 is stereo camera 211.

[0069] As depicted, camera system 209 sends image 232 to controller 214 for processing. Additionally, controller 214 stores connector pose 226 from which image 232 was generated for later use.

[0070] In this illustrative embodiment, controller 214 selects region 233 in image 232 that contains connector 222. In other words, connector 222 is located within region 233 in image 232.

[0071] In this illustrative example, region 233 may be selected by controller 214 using selection 250 in user input 258. In this example, operator 252 may operate human-machine interface 254 to generate user input 258 including selection 250.

[0072] As depicted, human-machine interface (HMI) 254 includes display system 255 and input system 256. Display system 255 is a physical hardware system and includes one or more display devices on which graphical user interface 247 can be displayed. The display device may include at least one of a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a computer monitor, a projector, a flat panel display, a head-up display (HUD), a head-mounted display (HMD), smart glasses, augmented reality glasses, or some other suitable device that can output information for visual presentation of the information.

[0073] Operator 252 is a person who can interact with graphical user interface 247 through user input 258 generated by input system 256. Input system 256 is a physical hardware system and can be selected from at least one of the following: a mouse, a keyboard, a touchpad, a trackball, a touch screen, a stylus, a motion sensing input device, a gesture detection device, a data glove, a cyber glove, a tactile feedback device, a neural interface, a biometric interface, or some other suitable type of input device.

[0074] Using the human-machine interface 254, the controller 214 can display the image 232 received from the camera system 209 on a display system 255 in the human-machine interface 254. The controller 214 can receive a user input 258 from an input system 256. In this embodiment, the user input 258 includes a selection 250 of the area 233.

[0075] The controller 214 creates an initial mask 240 using the image 232 and the region 233 in the image 232. In this embodiment, the initial mask 240 is an image and includes a background region 241 and a connector region 242.

[0076] When creating the initial mask 240, the controller 214 creates a background region 241 as a first portion of the image 232 outside of the region 233. The controller 214 creates a connector region 242 as a second portion of the image 232 defined by the region 233.

[0077] In one illustrative embodiment, controller 214 may also create inner connector region 243. For example, controller 214 may create inner connector region 243 inside connector region 242. In this embodiment, inner connector region 243 is selected to have a higher probability of surrounding a connector than the connector region.

[0078] In this embodiment, the initial mask 240 may have a background region 241, a connector region 242 defined by a region 233 in the image 232, and an inner connector region 243 that is an inner portion of the connector region 242. In other embodiments, the inner connector region 243 may be omitted, and the segmentation mask 244 may be created using the initial mask 240 having the background region 241 and the connector region 242.

[0079] Controller 214 creates a segmentation mask 244 that is also an image. Segmentation mask 244 is created by controller 214 using initial mask 240 and image 232. In this embodiment, segmentation mask 244 includes background region 241, connector region 242, and inner connector region 243. In addition, segmentation mask 244 also includes edges 245 for connector 222. In this illustrative embodiment, image segmentation 249 may be used to generate segmentation mask 244 to determine edges 245 of connector 222.

[0080] Image segmentation 249 can be implemented using different segmentation techniques. For example, image segmentation 249 can be performed using the GrabCut algorithm. The GrabCut algorithm is an image segmentation based on graph cuts. The algorithm can use iterative graph cuts to use the image 232 and the connector 222 identified by the initial mask 240. Using image segmentation 249, pixels in the image 232 can be reclassified into two different regions based on pixel similarity (such as being background or being a connector) using the initial mask 240.

[0081] In the depicted embodiment, the controller 214 applies thresholding 246 to the segmentation mask 244 to restore the pixels of the connector 222 to generate a final mask 248. For example, the segmentation mask 244 can be multiplied with the image 232. The thresholding 246 can be used to separate the connector 222 in the image 232 from the background region 241 in the segmentation mask 244.

[0082] In this embodiment, every pixel in the image 232 that is within the edge 245 defining the connector 222 is retained. The remaining portion of the image is removed or otherwise modified to form a final mask 248 that depicts the connector area as delineated from the background. The final mask may be composed of white pixels corresponding to the connector area and black pixels corresponding to the background.

[0083] In this embodiment, the final mask 248 is an image that can be used by the controller 214 to insert the set of wires 225 into the connector 222. Specifically, the final mask 248 can be used to insert the set of wires 225 into the holes 224 in the connector 222.

[0084] For example, when inserting the set of wires 225, the controller 214 positions the end effector 208 clamping the wire 227 in the set of wires 225 in the connector posture 226. As used herein, a "set" item is one or more items. For example, a set of wires 225 is one or more wires 225.

[0085] Controller 214 moves end effector 208 in connector pose 226 from connector pose 226 to insert wire 227 into connector 222 using final mask 248 .

[0086] In this embodiment, the controller 214 uses the end effector 208 to pick up the second wire 229 in the group of wires 225. In this embodiment, the second wire 229 is picked up from a position 231, in which the wire 225 is placed or stored for insertion into the connector 222. The controller 214 moves the end effector 208 back to the connector posture 226 from the position 231. The controller 214 moves the end effector 208 in the connector posture 226 to insert the second wire 229 into the connector 222 using the final mask 248. In this embodiment, by moving the end effector 208 back to the connector posture 226 (wherein the image 232 is taken to generate the final mask 248), the same mask can be used without generating a new mask. In this way, the robot 206 automatically increases the speed of inserting the wire 225 into the connector 222 compared to generating a new mask each time the wire is inserted into the connector 222.

[0087] In one illustrative embodiment, there are one or more technical solutions that overcome the technical problems associated with automatically inserting wires into connectors. Therefore, one or more technical solutions can provide at least one of the ability to increase the accuracy and speed at which wires can be inserted into connectors.

[0088] The computer system 212 may be configured to perform at least one of the steps, operations, and actions described in the different illustrative embodiments using software, hardware, firmware, or a combination thereof. Thus, the computer system 212 operates as a special-purpose computer system, wherein the controller 214 in the computer system 212 is capable of generating a mask for inserting wires, which increases the speed and accuracy with which the wires can be inserted into the connector. Specifically, the controller 214 converts the computer system 212 into a special-purpose computer system compared to a currently available general-purpose computer system without the controller 214.

[0089] In the illustrative embodiment, the controller 214 is used in the computer system 212 to integrate the process into the actual application for inserting the wire into the connector, which increases the performance of the computer system 212. In other words, the controller 214 in the computer system 212 involves the actual application of the process integrated into the controller 214 in the computer system 212, which generates an image of the connector, selects the area containing the connector in the image, and uses the image and the area to create an initial mask. Further, this actual application includes reading the segmentation mask using the initial mask image and applying thresholding to generate a final mask. The final mask is then used in the actual application to insert the wire into the connector using the final mask. In these embodiments, the end effector is moved to the camera system on the end effector to capture the gesture of the image used to generate the mask.

[0090] Figure 2 The illustration of manufacturing environment 200 in the embodiment is not meant to imply physical or structural limitations on the manner in which the illustrative embodiments may be implemented. Other components in addition to or in place of the components described may be used. Some components may be unnecessary. In addition, square blocks are presented to illustrate some functional components. When implemented in the illustrative embodiments, one or more of these square blocks may be combined, divided, or combined and divided into different square blocks.

[0091] For example, selection 250 of region 233 may be performed using other techniques besides user input 258. For example, an object recognition process may be used to identify connector 222 in image 232 and create region 233 based on the recognition. Thus, the automation of wire insertion may be further automated using robot 206. In another embodiment, object 220 may be any object in addition to or in place of connector 222, wherein a set of wires or other objects may be inserted into a hole in object 220 or otherwise interface with object 220.

[0092] Next go to Figure 3 , depicts a diagram of a data flow for generating a final mask for inserting a wire in accordance with an illustrative embodiment. The data flow may be used Figure 2 The wire assembly system 202 is used to perform.

[0093] Image 300 is generated by a camera system on the end effector of the robot. In this embodiment, image 300 is an image of connector 302. Image 300 is Figure 2 232. As depicted, region 303 has been selected to surround connector 302.

[0094] An initial mask 304 is also shown. The initial mask 304 is an image generated by selecting the region 303 from the image 300. The initial mask 304 is Figure 2 2. As shown, the initial mask 304 includes a background region 306 and a connector region 308. In this embodiment, an inner connector region 310 is also depicted.

[0095] In this illustrative embodiment, regions in initial mask 304 have different confidence levels regarding what is located in a particular region. For example, background region 306 has 100% confidence that pixels in image 300 are part of the background and do not contain connector 302. Connector region 308 has 50% confidence that pixels from image 300 are for connector 302. Inner connector region 310 has 100% confidence that pixels within image 300 corresponding to this region are for connector 302.

[0096] In this illustrative embodiment, segmentation mask 312 is generated using image 300 and initial mask 304. In this embodiment, the segmentation process uses image 300 and initial mask 304 is an input for generating segmentation mask 312.

[0097] As depicted, segmentation mask 312 includes background region 306, connector region 308, and segmentation is performed on segmentation mask 312. Additionally, segmentation mask 312 includes edge 314 for connector 302 in image 300. As shown, edge 314 defines region 316 surrounding connector 302.

[0098] The segmentation mask 312 is processed to obtain a final mask 318. The final mask 318 is Figure 2 302. In this embodiment, the segmentation mask 312 is processed using the image 300 to obtain the final mask 318. Thresholding is performed to identify pixels from the image 300 that are within the edge 314 that defines the area 316 surrounding the connector 302. In this manner, the final mask 318 can be generated for use in inserting a wire into the connector 302.

[0099] Now turn to Figure 4 , depicts a diagram of an end effector for a robotic arm according to an illustrative embodiment. In this illustrative embodiment, the end effector 400 is Figure 2 4. In this illustrative embodiment, end effector 400 is configured for wire insertion. As shown, end effector 400 holds wire 401. As depicted, stereo camera 404 is connected to end effector 400. Stereo camera 404 includes first camera 406 and second camera 408. In this illustrative embodiment, stereo camera 404 also includes lighting in the form of light emitting diode (LED) system 410. Stereo camera 404 is Figure 24. Additional illumination is provided by an overhead light emitting diode (LED) array 412. The use of this illumination system can reduce or eliminate ambient light in the images generated by the stereo camera 404. As a result, more consistency in the images can be generated for identifying the connector 402.

[0100] As depicted in this embodiment, the placement of stereo camera 404, LED system 410, and overhead LED array 412 can direct light at connector 402 and reduce the effects of ambient light when generating an image of connector 402. As a result, stereo camera 404 can generate an image of a mask that more accurately generates connector 402.

[0101] In this illustrative embodiment, end effector 400 has a connector posture facing connector 402. In other words, end effector 400 faces surface 405. More specifically, end effector 400 has a connector posture facing surface 405 of connector 402. In this embodiment, surface 405 is a surface where there is a hole (not shown) in connector 402 for wire insertion of a wire, such as wire 401 having wire contact 403.

[0102] With this configuration, the first camera 406, the second camera 408, the light emitting diode (LED) system 410, the overhead light emitting diode (LED) array 412, and the end effector 400 in the connector posture can generate an image of the connector 402 with the desired quality, which increases the ability to generate a final mask for performing wire insertion. In this embodiment, the connector posture of the end effector 400 includes the position of the tip or other part of the end effector 400 and the orientation of the direction form of the end effector 400, such as the axis 411. Although this embodiment illustrates the use of a stereo camera 404, other types of cameras can be used to segment the connector. For example, a monochrome camera can be used instead of the stereo camera 404.

[0103] Next go to Figure 5 , depicts an illustration of a flowchart of a process for inserting a set of wires into holes in a connector in accordance with an illustrative embodiment. Figure 5 The processes in may be implemented in hardware, software, or both. When implemented in software, the processes may take the form of program instructions executed by one or more processor units in one or more hardware devices in one or more computer systems. For example, the processes may be implemented in Figure 2 The invention is implemented in the controller 214 in the computer system 212 in the embodiment.

[0104] The process begins by moving the end effector to a connector pose facing the connector (operation 500). The process collects an image of the connector (operation 502). The process segments the image to generate a final mask for wire insertion (operation 504).

[0105] The process uses the final mask to identify the location of the holes in the connector (operation 506). In operation 506, the holes can be identified using a circular Hough transform (CHT). This transform is used to detect circles in digital images and can be used to identify holes in the final mask, which is an image of the connector.

[0106] The process selects a target hole in the connector for wire insertion (operation 508). The process aligns the wire with the target hole in the connector (operation 510). The process inserts the wire into the hole in the connector (operation 512).

[0107] The process determines whether there is another target hole for wire insertion in the connector (operation 514). The number of target holes in the connector may depend on the number of connections required. In other words, the wire may not be inserted into all holes in the connector. If there is another target hole, the process uses the end effector to pick up the wire for insertion (operation 516). Then, the process returns the end effector to the connector posture (operation 518). The process returns to operation 506. The same final mask is used because the new wire is inserted by the same end effector in the connector posture. As a result, since there is no need to generate a new mask, the insertion of the wire can be performed more quickly.

[0108] Referring again to operation 514, if there is not another target hole, the process then terminates.

[0109] Next go to Figure 6 , an illustration of a flowchart of a process for inserting a wire is depicted in accordance with an illustrative embodiment. Figure 6 The processes in may be implemented in hardware, software, or both. When implemented in software, the processes may take the form of program instructions executed by one or more processor units in one or more hardware devices in one or more computer systems. For example, the processes may be implemented in Figure 2 The invention is implemented in the controller 214 in the computer system 212 in the embodiment.

[0110] The process begins by generating an image of a connector using a camera system connected to an end effector (operation 600). In operation 600, the end effector is in a connector pose facing the connector. The process selects a region in the image that surrounds the connector (operation 602).

[0111] The process creates an initial mask using the image and the region (operation 604). In operation 604, the initial mask includes a background region and a connector region.

[0112] The process creates a segmentation mask using the initial mask and the image (operation 606). In operation 606, the segmentation mask includes edges for connectors.

[0113] The process applies thresholding to the segmentation mask to restore the pixels of the connector and generates a final mask (operation 608). The process terminates thereafter. In operation 608, the wire is inserted into the connector using the final mask.

[0114] Next reference Figure 7 , depicts a flowchart of a process for inserting a wire in accordance with an illustrative embodiment. Figure 7 The process shown in Figure 6 An embodiment of the additional operations performed by the operations in the example.

[0115] The process inserts a set of wires into the connector using the final mask (operation 700). The process terminates thereafter.

[0116] exist Figure 8 , a flow chart of a process for inserting a set of wires is depicted in accordance with an illustrative embodiment. Figure 8 The process shown in Figure 7 Operation 700 and Figure 6 An embodiment of a method for implementing the operations in .

[0117] The process positions the end effector holding the wire in a connector posture (operation 800). The process moves the end effector in the connector posture to insert the wire into the connector using the final mask (operation 802). Thereafter the process terminates.

[0118] Steering Fig. 9 , an illustration of a flowchart of a process for inserting a second wire is depicted in accordance with an illustrative embodiment. Fig. 9 The process shown is useful for implementing Figure 7 An embodiment of additional steps of operation 700 in.

[0119] The process picks up the second wire using the end effector (operation 900 ). The process moves the end effector back to the connector pose (operation 902 ).

[0120] The process moves the end effector in the connector posture to insert the second wire into the connector using the final mask (operation 904). The process terminates thereafter.

[0121] Next in Fig.10In FIG. 1 , an illustration of a flow chart of a process for creating an initial mask according to an illustrative embodiment is depicted. The process in the figure is Figure 6 An embodiment of an implementation of operation 604 in .

[0122] The process creates a background region as a first portion of the image outside the region (operation 1000). The process creates a connector region as a second portion of the image defined by the region (operation 1002). The process terminates thereafter.

[0123] Now turn to Fig.11 , depicts an illustration of a flow chart of a process for creating an initial mask in accordance with an illustrative embodiment. The process in this figure is similar to Fig.10 The operations in are used together to create an additional step embodiment of the initial mask.

[0124] The process creates an inner connector region inside a connector region, wherein the inner connector region is selected to have a higher probability of surrounding the connector than the connector region (operation 1100). The process terminates thereafter.

[0125] Next reference Fig.12 , depicts an illustration of a flowchart of a process for selecting a region in accordance with an illustrative embodiment. The process in this figure is Figure 6 An embodiment of an implementation of operation 602 in .

[0126] The process displays an image on a display system (operation 1200). The process receives user input from an input system (operation 1202). The process terminates thereafter. In operation 1202, the user input includes a selection of a region.

[0127] The flow charts and block diagrams in the different described embodiments show the architecture, function and operation of some possible implementations of the device and method in the illustrative embodiments. In this regard, each frame in the flow chart or block diagram can represent at least one of a module, section, function and operation or a part of a step. For example, one or more frames can be implemented as a combination of program instructions, hardware or program instructions and hardware. When implemented in hardware, the hardware can, for example, take the form of an integrated circuit that is manufactured or configured to perform one or more operations in the flow chart or block diagram. When implemented as a combination of program instructions and hardware, the implementation can take the form of firmware. Each frame in the flow chart or block diagram can be implemented using a combination of a dedicated hardware system or dedicated hardware that performs different operations and a program instruction run by dedicated hardware.

[0128] In some alternative implementations of the illustrative embodiments, one or more functions indicated in the blocks may not occur in the order indicated in the figure. For example, in some cases, two blocks shown in succession may be performed substantially simultaneously, or the blocks may sometimes be performed in reverse order, depending on the functions involved. In addition, other blocks may be added in addition to the blocks shown in the flowchart or block diagram.

[0129] Now turn to Fig.13 , depicts a block diagram of a data processing system according to an illustrative embodiment. Data processing system 1300 may be used to implement Figure 1 The server computer 104, the server computer 106, and the client device 110 in the data processing system 1300 can also be used to implement Figure 2 In this illustrative example, data processing system 1300 includes communications framework 1302, which provides communications between processor unit 1304, memory 1306, persistent storage device 1308, communications unit 1310, input / output (I / O) unit 1312, and display 1314. In this example, communications framework 1302 takes the form of a bus system.

[0130] The processor unit 1304 is used to execute instructions of the software that may be loaded into the memory 1306. The processor unit 1304 includes one or more processors. For example, the processor unit 1304 may be selected from at least one of a multi-core processor, a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a network processor, or some other suitable type of processor. In addition, the processor unit 1304 may be implemented using one or more heterogeneous processor systems in which a main processor and a secondary processor are present on a single chip. As another illustrative embodiment, the processor unit 1304 may be a symmetric multi-processor system that includes multiple processors of the same type on a single chip.

[0131] Memory 1306 and permanent storage 1308 are embodiments of storage 1316. A storage device is any hardware capable of storing information such as, for example, but not limited to, data, program instructions in functional form, or at least one of other suitable information on a temporary basis, on a permanent basis, or on both a temporary basis and a permanent basis. Storage 1316 may also be referred to as computer-readable storage in these illustrative embodiments. In these embodiments, memory 1306 may be, for example, a random access memory or any other suitable volatile or non-volatile storage device. Permanent storage 1308 may take various forms, depending on the particular implementation.

[0132] For example, permanent storage device 1308 may include one or more components or devices. For example, permanent storage device 1308 may be a hard drive, a solid state drive (SSD), flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination thereof. The media used by permanent storage device 1308 may also be removable. For example, a removable hard drive may be used for permanent storage device 1308.

[0133] In these illustrative examples, communications unit 1310 provides for communications with other data processing systems or devices. In these exemplary embodiments, communications unit 1310 is a network interface card.

[0134] Input / output unit 1312 allows for input and output of data with other devices that may be connected to data processing system 1300. For example, input / output unit 1312 may provide a connection for user input through at least one of a keyboard, a mouse, or some other suitable input device. Additionally, input / output unit 1312 may send output to a printer. Display 1314 provides a mechanism for displaying information to a user.

[0135] Instructions for at least one of an operating system, applications, or programs may be located in storage devices 1316 that are in communication with processor unit 1304 through communications framework 1302. Processor unit 1304 may perform the processes of the different embodiments using computer-implemented instructions, which may be located in a memory, such as memory 1306.

[0136] These instructions are referred to as program instructions, computer usable program instructions, or computer readable program instructions that can be read and executed by a processor in processor unit 1304. The program instructions in different embodiments may be embodied on different physical or computer readable storage media, such as memory 1306 or persistent storage device 1308.

[0137] Program instructions 1318 are located in functional form on computer readable media 1320, which is selectively removable and can be loaded or transferred to data processing system 1300 for execution by processor unit 1304. In these illustrative embodiments, program instructions 1318 and computer readable media 1320 form computer program product 1322. In the illustrative embodiment, computer readable media 1320 is computer readable storage media 1324.

[0138] Computer readable storage medium 1324 is a physical or tangible storage device for storing program instructions 1318, rather than a medium for propagating or transmitting program instructions 1318. Computer readable storage medium 1324 may be at least one of an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or other physical storage medium. Some known types of storage devices that include these media include: a magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or pits / lands formed in a major surface of a disk), or any suitable combination thereof.

[0139] As the term is used in this disclosure, computer-readable storage medium 1324 should not be construed as storing in the form of transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, light pulses passing through fiber optic cables, electrical signals transmitted through wires, or at least one of other transmission media.

[0140] In addition, data may be moved at some occasional points in time during the normal operation of the storage device. These normal operations include access, defragmentation, or garbage collection. However, these operations do not render the storage device as transient because the data is not transient when it is stored in the storage device.

[0141] Alternatively, program instructions 1318 may be transferred to data processing system 1300 using a computer readable signal medium. A computer readable signal medium is a signal, and may be, for example, a propagated data signal containing program instructions 1318. For example, a computer readable signal medium may be at least one of an electromagnetic signal, an optical signal, or any other suitable type of signal. These signals may be transmitted via a connection, such as a wireless connection, a fiber optic cable, a coaxial cable, an electrical wire, or any other suitable type of connection.

[0142] Furthermore, as used herein, "computer-readable medium 1320" may be singular or plural. For example, program instructions 1318 may be located in computer-readable medium 1320 in the form of a single storage device or system. In another embodiment, program instructions 1318 may be located in computer-readable medium 1320 distributed in multiple data processing systems. In other words, some instructions in program instructions 1318 may be located in one data processing system, while other instructions in program instructions 1318 may be located in another data processing system. For example, a portion of program instructions 1318 may be located in computer-readable medium 1320 in a server computer, while another portion of program instructions 1318 may be located in computer-readable medium 1320 in a group of client computers.

[0143] The different components shown for data processing system 1300 are not meant to provide architectural limitations to the manner in which the different embodiments may be implemented. In some illustrative embodiments, one or more components may be incorporated into or otherwise form a part of another component. For example, in some illustrative embodiments, memory 1306, or portions thereof, may be incorporated into processor unit 1304. The different illustrative embodiments may be implemented in a data processing system including components in addition to or in place of those shown for data processing system 1300. Fig.13 Other components shown in the can be varied from the illustrative embodiment shown. The different embodiments can be implemented using any hardware device or system capable of running program instructions 1318.

[0144] You can Fig.14 Aircraft manufacturing and service method 1400 and the like are shown Fig.15 Illustrative embodiments of the present disclosure are described in the context of aircraft 1500 as shown. Fig.14 , an illustration of an aircraft manufacturing and service method is depicted in accordance with an illustrative embodiment. During pre-production, aircraft manufacturing and service method 1400 may include Fig.15 Specification and design 1402 of aircraft 1500 and procurement of materials 1404.

[0145] During production, Fig.15 Component and subassembly manufacturing 1406 and system integration 1408 of aircraft 1500. Thereafter, Fig.15 The aircraft 1500 in may undergo certification and delivery 1410 for entry into service 1412. Upon entry into service 1412 by a customer, Fig.15 Aircraft 1500 is scheduled for routine maintenance and service 1414, which may include modification, reconfiguration, refurbishment, and other maintenance or service.

[0146] Each process of the aircraft manufacturing and maintenance method 1400 may be performed or carried out by a system integrator, a third party, an operator, or some combination thereof. In these embodiments, the operator may be a customer. For the purposes of this specification, a system integrator may include, but is not limited to, any number of aircraft manufacturers and main system subcontractors; a third party may include, but is not limited to, any number of vendors, subcontractors, and suppliers; and an operator may be an airline, a leasing company, a military entity, a service organization, etc.

[0147] Reference now Fig.15 , depicts a diagram of an aircraft in which an illustrative embodiment may be implemented. In this embodiment, aircraft 1500 is Fig.14 14. Aircraft manufacturing and service method 1400 in the context of the present invention may be produced and may include airframe 1502 and interior 1506 having multiple systems 1504. Embodiments of system 1504 include one or more of propulsion system 1508, electrical system 1510, hydraulic system 1512, and environmental system 1514. Any number of other systems may be included. Although an aerospace embodiment is shown, different illustrative embodiments may be applied to other industries, such as the automotive industry.

[0148] Can be Fig.14 Apparatus and methods embodied herein are employed during at least one of the stages of aircraft manufacturing and service method 1400 .

[0149] In one illustrative embodiment, Fig.14 The parts or subassemblies produced in parts and subassembly manufacturing 1406 may be used in conjunction with aircraft 1500. Fig.14 As another example, the components or subassemblies produced when the components are put into use 1412 can be made or manufactured in a similar manner. Fig.14 One or more apparatus embodiments, method embodiments, or a combination thereof are utilized during component and subassembly manufacturing 1406 and system integration 1408 in the aircraft 1500. When the aircraft 1500 is placed into service 1412, Fig.14 One or more apparatus embodiments, method embodiments, or a combination thereof may be utilized during maintenance 1414, or both. Use of a number of different illustrative embodiments may significantly speed up assembly of aircraft 1500, reduce the cost of aircraft 1500, or both speed up assembly of aircraft 1500 and reduce the cost of aircraft 1500.

[0150] For example, wire assembly system 202 may be used during component and subassembly manufacturing 1406 to assemble wire bundles for wire harnesses used in aircraft 1500. As another example, wire assembly system 202 may also be used to assemble wire bundles for components in maintenance and service 1414. Furthermore, wire assembly system 202 may be used to manipulate other objects in addition to or in lieu of wire contacts used to insert wires into connectors.

[0151] Now turn to Fig.16 , depicts an illustration of a block diagram of a product management system in accordance with an illustrative embodiment. Product management system 1600 is a physical hardware system. In this illustrative embodiment, product management system 1600 includes at least one of a manufacturing system 1602 and a maintenance system 1604.

[0152] Manufacturing system 1602 is configured to manufacture products, such as Fig.15 As depicted, manufacturing system 1602 includes manufacturing device 1606. Manufacturing device 1606 includes at least one of assembly device 1608 and assembly device 1610.

[0153] Assembly device 1608 is used to make Fig.15 1500. For example, assembly apparatus 1608 may include machines and tools. These machines and tools may be at least one of a drill, a hydraulic press, a furnace, an autoclave, a mold, a composite tape laying machine, an automated fiber placement (AFP) machine, a vacuum system, a robotic pick and place system, a flatbed cutter, a laser cutter, a computer numerical control (CNC) cutting machine, a lathe, or other suitable types of apparatus. Assembly apparatus 1608 may be used to manufacture at least one of a metal part, a composite part, a semiconductor, a circuit, a fastener, a rib, a skin panel, a wing spar, an antenna, or other suitable types of parts.

[0154] The assembly device 1610 is used to assemble components to form Fig.15 Specifically, the assembly device 1610 is used to assemble components and parts to form Fig.15 The assembly apparatus 1610 may also include machines and tools. These machines and tools may be at least one of a robotic arm, a crawler, a fastener installation system, a track-based drilling system, and a robot. The assembly apparatus 1610 may be used to assemble parts such as seats, horizontal stabilizers, wings, engines, engine casings, landing gear systems, and Fig.15 1500 other parts of the aircraft.

[0155] In the exemplary embodiment, maintenance system 1604 includes maintenance device 1612. Maintenance device 1612 may include Fig.15 Maintenance equipment 1612 may include any equipment required to perform maintenance on aircraft 1500. Fig.15 The operations may include disassembling a part, refurbishing a part, inspecting a part, reconditioning a part, manufacturing a replacement part, or performing various operations on a part of an aircraft 1500. Fig.15 At least one of the other operations of performing maintenance on the aircraft 1500. These operations can be for routine maintenance, inspection, upgrade, refurbishment or other types of maintenance operations.

[0156] In an illustrative embodiment, maintenance device 1612 may include an ultrasonic inspection device, an x-ray imaging system, a vision system, a drill, a crawler, and other suitable devices. In some cases, maintenance device 1612 may include assembly device 1608, assembly device 1610, or both to produce and assemble parts required for maintenance.

[0157] Product management system 1600 also includes control system 1614. Control system 1614 is a hardware system and may also include software or other types of components. Control system 1614 is configured to control the operation of at least one of manufacturing system 1602 or maintenance system 1604. Specifically, control system 1614 may control the operation of at least one of assembly device 1608, assembly device 1610, or maintenance device 1612.

[0158] The hardware in control system 1614 may be implemented using hardware that may include computers, circuits, networks, and other types of devices. Control may take the form of direct control of manufacturing devices 1606. For example, robots, computer-controlled machines, and other devices may be controlled by control system 1614. In other illustrative embodiments, control system 1614 may manage operations performed by human operators 1616 when manufacturing or performing maintenance on aircraft 1500. For example, control system 1614 may assign tasks, provide instructions, display models, or perform other operations to manage operations performed by operators 1616. In these illustrative embodiments, Figure 2 The controller 214 may be implemented in the control system 1614 to Fig.15 In other words, controller 214 may be used to control a robot (such as a robot arm having an end effector in at least one of assembly device 1608 or assembly device 1610) to insert a wire into a connector or other object with a hole added for wire insertion.

[0159] In different illustrative embodiments, human operator 1616 may operate or interact with at least one of manufacturing device 1606, maintenance device 1612, or control system 1614. The interaction may occur to manufacture Fig.15 There are 1500 aircraft in the market.

[0160] Of course, product management system 1600 may be configured to manage products other than Fig.15 Although product management system 1600 has been described with respect to manufacturing in the aerospace industry, product management system 1600 can be configured to manage products for other industries. For example, product management system 1600 can be configured to manufacture products for the automotive industry and any other suitable industry.

[0161] Therefore, illustrative embodiments provide methods, devices, systems, and computer program products for wire insertion. In one illustrative embodiment, a method for inserting a wire into a connector is provided. An image of the connector is generated using a camera system connected to an end effector. The end effector is in a connector posture facing the connector. An area in the image that surrounds the connector is selected. An initial mask is created using the image and the area. The initial mask includes a background area and a connector area. A segmentation mask is created using the initial mask and the image, wherein the segmentation mask includes an edge for the connector. Thresholding is applied to the segmentation mask to restore pixels of the connector and a final mask is generated. The final mask is used to insert the wire into the connector.

[0162] In an illustrative embodiment, a wire can be inserted into a connector without generating a new final mask for each wire insertion. In an illustrative embodiment, the end effector is moved to a connector pose at which an image is generated for creating the final mask. Each time a wire is picked up for insertion into a connector, the end effector returns to the pose. In this way, the same final mask can be used to insert wires from the connector pose.

[0163] Descriptions of different illustrative embodiments have been presented for purposes of illustration and description, and are not intended to be exhaustive or limited to embodiments of the disclosed forms. Different illustrative embodiments describe components that perform actions or operations. In illustrative embodiments, components may be configured to perform the described actions or operations. For example, components may have a configuration or design of a structure that provides components with the ability to perform the actions or operations described as being performed by components in the illustrative embodiments. In addition, in the context of using the terms "comprises," "comprising," "having," "containing," and variations thereof, these terms are intended to be inclusive in a manner similar to the term "comprises" as an open transition word, without excluding any additional or other elements.

[0164] Many modifications and variations will be apparent to those of ordinary skill in the art. In addition, different illustrative embodiments may provide different features than other desired embodiments. The selected one or more embodiments are selected and described in order to best explain the principles of the embodiments, the practical application, and to enable others of ordinary skill in the art to understand the disclosure of various embodiments with various modifications suitable for the particular use contemplated.

Claims

1. A robot system, comprising: End effector (136, 208, 400); a camera system (209) connected to the end effector (136, 208, 400); and A controller (214), wherein the controller (214) is configured to: generating an image (132, 232, 300) of the connector (133, 222, 302, 402) using the camera system (209) with the end effector (136, 208, 400) in a connector posture (226) facing the connector (133, 222, 302, 402); selecting a region (233, 303) in the image (132, 232, 300) surrounding the connector (133, 222, 302, 402); Creating an initial mask (142, 240, 304) using the image (132, 232, 300) and the region (233, 303), wherein the initial mask (142, 240, 304) includes a background region (144, 241, 306) and a connector region (146, 242, 308); creating a segmentation mask (150, 244, 312) using the initial mask (142, 240, 304) and the image (132, 232, 300), wherein the segmentation mask (150, 244, 312) includes edges (245, 314) for the connectors (133, 222, 302, 402); and Thresholding (246) is applied to the segmentation mask (150, 244, 312) to restore the pixels of the connectors (133, 222, 302, 402) and generate a final mask (248, 318).

2. The robot system according to claim 1, wherein: The controller (214) is also configured to: A set of wires (225) are inserted into the connector (133, 222, 302, 402) using the final mask (248, 318).

3. The robot system according to claim 2, wherein: Upon inserting the set of wires (225), the controller (214) is configured to: Positioning the end effector (136, 208, 400) that clamps the wire (140, 227, 401) in the connector posture (226); as well as Moving the end effector (136, 208, 400), to use the final mask (248, 318) to separate the wires (140, 227, 401) is inserted into the connector (133, 222, 302, 402).

4. The robot system according to claim 3, wherein: The controller (214) is also configured to: Using the end effector (136, 208, 400) to pick up the second wire (140, 227,401); moving the end effector (136, 208, 400) back to the connector posture (226); as well as Moving the end effector (136, 208, 400), so as to use the final mask (248, 318) to form the second wire (140, 227, 401) is inserted into the connector (133, 222, 302, 402).

5. The robot system according to claim 1, wherein: When creating the initial mask (142, 240, 304), the controller (214) is configured to: The background area (144, 241, 306) is created as the image (132, 232, 300) outside the region (233, 303); and The connector region (146, 242, 308) is created as a second portion of the image (132, 232, 300) defined by the region (233, 303).

6. The robot system according to claim 1, wherein: When creating the initial mask (142, 240, 304), the controller (214) is further configured to: An inner connector region (146, 242, 308) is created that is located inside the connector region (146, 242, 308), wherein the inner connector region (146, 242, 308) is selected to have a higher probability of surrounding the connector (133, 222, 302, 402) than the connector region (146, 242, 308).

7. The robot system according to claim 1, wherein: The connector region (146, 242, 308) is defined by the region (233, 303) and by an inner connector region (146, 242, 308) which is an inner portion of the connector region (146, 242, 308).

8. The robot system according to claim 1, wherein: When the area (233, 303) is selected, the controller (214) is configured to: displaying the image (132, 232, 300) on a display system (255); and A user input (258) is received from an input system (256), wherein the user input (258) includes a selection (250) of the area (233, 303).

9. The robotic system according to claim 1, further comprising: A robot, wherein the robot comprises the end effector (136, 208, 400), the camera system (209) is connected to the end effector (136, 208, 400), and the end effector (136, 208, 400) is connected to a platform.

10. A method for inserting a wire (140, 227, 401), the method comprising: An image (132, 232, 300) of a connector (133, 222, 302, 402) is generated (600) using a camera system (209) connected to an end effector (136, 208, 400), wherein the end effector (136, 208, 400) is in a position facing the connector (133, 222, 302, 402) of the connector posture (226); Selecting (602) a region (233, 303) in the image (132, 232, 300) surrounding the connector (133, 222, 302, 402); creating (604) an initial mask (142, 240, 304) using the image (132, 232, 300) and the region (233, 303), wherein the initial mask (142, 240, 304) includes a background region (144, 241, 306) and a connector region (146, 242, 308); creating (606) a segmentation mask (150, 244, 312) using the initial mask (142, 240, 304) and the image (132, 232, 300), wherein the segmentation mask (150, 244, 312) includes edges (245, 314) for the connectors (133, 222, 302, 402); and Thresholding (246) is applied to the segmentation mask (150, 244, 312) to restore pixels of the connector (133, 222, 302, 402) and generate a final mask (248, 318), wherein the final mask (248, 318) is used to insert the wire (140, 227, 401) into the connector (133, 222, 302, 402).

11. The method according to claim 10, further comprising: A set of wires (225) is inserted (700) into the connector (133, 222, 302, 402) using the final mask (248, 318).

12. The method according to claim 11, wherein: Inserting the set of wires (225) comprises: Positioning (800) the end effector (136, 208, 400) clamping the wire (140, 227, 401) in the connector posture (226); and The end effector (136, 208, 400) is moved (802) in the connector posture (226) to insert the wire (140, 227, 401) into the connector (133, 222, 302, 402) using the final mask (248, 318).

13. The method according to claim 12, further comprising: Picking up (900) a second wire (140, 227, 401) using the end effector (136, 208, 400); moving (902) the end effector (136, 208, 400) back to the connector posture (226); as well as The end effector (136, 208, 400) is moved (904) in the connector posture (226) to insert the second wire (140, 227, 401) into the connector (133, 222, 302, 402) using the final mask (248, 318).

14. The method according to claim 10, wherein: Creating an initial mask (142, 240, 304) includes: creating (1000) the background region (144, 241, 306) as a first portion of the image (132, 232, 300) outside of the region (233, 303); and The connector region (146, 242, 308) is created (1002) as a second portion of the image (132, 232, 300) defined by the region (233, 303).

15. The method according to claim 10, wherein: Creating an initial mask (142, 240, 304) includes: An inner connector region (146, 242, 308) is created (1100) inside the connector region (146, 242, 308), wherein the inner connector region (146, 242, 308) is selected to have a higher probability of surrounding the connector (133, 222, 302, 402) than the connector region (146, 242, 308).

16. The method according to claim 10, wherein: The connector region (146, 242, 308) is defined by the region (233, 303) and by an inner connector region (146, 242, 308) which is an inner portion of the connector region (146, 242, 308).

17. The method according to claim 10, wherein: Selecting the region (233, 303) includes: displaying (1200) the image (132, 232, 300) on a display system (255); and A user input (258) is received (1202) from an input system (256), wherein the user input (258) includes a selection (250) of the area (233, 303).

18. The method according to claim 10, wherein: The robot includes the end effector (136, 208, 400), the camera system (209) is connected to the end effector (136, 208, 400), and the end effector (136, 208, 400) is connected to the platform.

19. A wire (140, 227, 401) assembly system, comprising: Computer systems (212); as well as A controller (214) in the computer system (212), wherein the controller (214) is configured to: generating an image (132, 232, 300) of the connector (133, 222, 302, 402) using a camera system (209) coupled to an end effector (136, 208, 400), the end effector being in a connector pose (226) facing the connector (133, 222, 302, 402); selecting a region (233, 303) in the image (132, 232, 300) surrounding the connector (133, 222, 302, 402); Creating an initial mask (142, 240, 304) using the image (132, 232, 300) and the region (233, 303), wherein the initial mask (142, 240, 304) includes a background region (144, 241, 306) and a connector region (146, 242, 308); creating a segmentation mask (150, 244, 312) using the initial mask (142, 240, 304) and the image (132, 232, 300), wherein the segmentation mask (150, 244, 312) includes edges (245, 314) for the connectors (133, 222, 302, 402); applying thresholding (246) to the segmentation mask (150, 244, 312) to restore pixels of the connector (133, 222, 302, 402) and generate a final mask (248, 318); and A wire (140, 227, 401) is inserted into the connector (133, 222, 302, 402) using the end effector (136, 208, 400) and the final mask (248, 318).

20. A computer program product (1322) for wire (140, 227, 401) assembly, the computer program product (1322) comprising a computer readable storage medium (1320), the computer readable storage medium (1320) having program instructions (1318) embodied therein, the program instructions (1318) being executable by a computer system (212) to cause the computer system (212) to: generating an image (132, 232, 300) of a connector (133, 222, 302, 402) using a camera system (209) coupled to an end effector (136, 208, 400), the end effector being in a connector pose (226) facing the connector (133, 222, 302, 402); selecting a region (233, 303) in the image (132, 232, 300) surrounding the connector (133, 222, 302, 402); An initial mask (142, 240, 304) is created using the image (132, 232, 300) and the region (233, 303), wherein The initial mask (142, 240, 304) includes a background area (144, 241, 306) and a connector area (146, 242, 308); creating a segmentation mask (150, 244, 312) using the initial mask (142, 240, 304) and the image (132, 232, 300), wherein the segmentation mask (150, 244, 312) includes edges (245, 314) for the connectors (133, 222, 302, 402); applying thresholding (246) to the segmentation mask (150, 244, 312) to restore pixels of the connector (133, 222, 302, 402) and generate a final mask (248, 318); and A wire (140, 227, 401) is inserted into the connector (133, 222, 302, 402) using the end effector (136, 208, 400) and the final mask (248, 318).