Man-machine collaborative precise hole making system

Abstract

The invention relates to a man-machine collaborative precise drilling system, and belongs to the field of aeronautical manufacturing engineering / aircraft assembly. The problems that automatic hole making is expensive and poor in accessibility, and traditional manual hole making is unstable in precision and low in efficiency are solved, machine vision and operators are combined to form a man-machine cooperation precise hole making system, the manual lineation link is avoided, and the manual hole making efficiency is greatly improved. And the accuracy of the drilling position is guaranteed through the digital measurement technology, the aircraft assembly quality is improved, the whole operation process is controllable in precision and efficient, and the intelligent level of aircraft manufacturing is improved.

Description

Technical Field

[0001] This invention relates to a human-machine collaborative precision drilling system, belonging to the field of aerospace manufacturing engineering / aircraft assembly. Background Technology

[0002] Human-computer collaborative systems, also known as human-computer associative systems, are generally considered to be systems jointly composed of humans and computers. The computer primarily handles large amounts of data computation and some reasoning tasks, while humans are responsible for tasks where computing or execution power is insufficient, such as selection, decision-making, execution, and evaluation. This fully leverages human flexibility and creativity. Humans and computers collaborate closely, enabling more efficient handling of various complex problems. Aircraft assembly can be divided into frame assembly and panel assembly, involving the positioning and connection of numerous parts, groups, and components. These connections are almost entirely fastener connections, thus requiring the creation of numerous fastener mounting holes at different stages of assembly. Hole-making methods are divided into automatic and manual hole-making. Automatic hole-making offers high efficiency and precision, but the equipment is expensive and only suitable for hole-making in open areas. Manual hole making is the most traditional method, generally using pneumatic or electric drills. Quality is ensured by the operator's skill level. Its advantage is less restriction of operating space, but its disadvantages include inconsistent quality and low efficiency. This is because the manual hole making process involves marking to determine the hole position and then drilling with hand tools. Both the marking and drilling processes are susceptible to human error and misjudgment. Furthermore, the marking process is time-consuming, and the limited resolution of steel rulers leads to significant marking errors, which are even more pronounced on curved parts. In mass production, over 70% of fastener holes require repeated marking and drilling, inevitably leading to quality problems. Moreover, hole position deviations are difficult to inspect, and excessive quantities can reduce aircraft fatigue life and potentially cause flight accidents.

[0003] To address the aforementioned issues, a human-machine collaborative precision hole-making system needs to be designed, which combines digital measurement technology with the operator's execution to form a digital interaction link between reality and virtuality, thereby realizing the digital transmission of information throughout the hole-making process. Summary of the Invention

[0004] This project has invented a human-machine collaborative precision drilling system, including its operation process and a trackable drilling tool.

[0005] The technical solution of the present invention is as follows:

[0006] A human-machine collaborative precision hole-making system combines the advantages of automated hole-making equipment and traditional manual hole-making, integrating machine vision, augmented reality, and flexible assembly concepts to create a novel human-machine collaborative processing system. The system mainly consists of eight parts: assembly fixture, reference target, product, binocular tracking camera, computer, trackable hole-making tool, operator, AR glasses, etc. Figure 1 As shown.

[0007] The assembly fixture is used for product positioning and clamping, ensuring that the assembled product is in the correct position in the aircraft coordinate system. The assembly fixture is welded from steel plates, and the product locator is screwed to the surface of the plate. The target is installed at an open area on the surface of the jig.

[0008] The aforementioned reference target is used for coordinate system transformation in the binocular tracking camera measurement system. The target needs to be securely mounted on the assembly fixture, typically using a magnetic attachment method. The reference target is a hemisphere with a diameter of 0.5 inches, and a highly reflective photogrammetric sticker with an outer diameter of 10 mm and an inner diameter of 6 mm is affixed to its center.

[0009] The product is composed of various parts and is installed on an assembly fixture, where it is positioned and clamped by a locator.

[0010] The binocular tracking camera is used to track the real-time spatial position of a trackable drilling tool in the aircraft coordinate system.

[0011] The computer is equipped with data analysis software for processing data collected by the binocular tracking camera, analyzing the deviation between the position of the trackable drilling tool and the theoretical drilling position, and transmitting the deviation information to the AR glasses.

[0012] The trackable drilling tool can be tracked by a binocular tracking camera to determine the coordinates of the drill bit tip in the aircraft coordinate system in real time, and then used for drilling the product. Figure 3 As shown.

[0013] The operator holds a trackable hole-making tool, wears AR glasses, and adjusts the hole-making position according to the positional deviation information provided by the computer to complete the hole-making work.

[0014] The AR glasses receive information from the computer and are used to display the positional deviation information of the trackable drilling tool in real time. The AR glasses are worn on the operator's head.

[0015] Operation process of human-machine collaborative precision hole making system

[0016] The human-machine collaborative precision drilling system represents a novel process implementation route. It enables virtual-to-real mapping of tool positions and point models, visualized in a visual manner, and allows for low-latency hand-eye coordination between the operator, scientifically integrating humans and equipment within the system. This process consists of 16 steps, as follows: Figure 2 As shown.

[0017] Step 1: Product placement. Position the product on the assembly fixture, using the fixture to position the product at its theoretical location in the aircraft coordinate system.

[0018] Step 2: Set up the binocular tracking camera. Set up the binocular tracking camera according to the position of the assembly fixture and the drilling area of ​​the product, ensuring that the camera's measurement field of view covers the assembly fixture and the drilling area of ​​the product.

[0019] Step 3: Connect to the computer and open the software. Install data processing and analysis software on the computer. The software can communicate with the stereo tracking camera, obtain the coordinates of the target points captured by the camera in real time, and highlight the target points in the camera image.

[0020] Step 4: Import the theoretical coordinates of the assembly tooling reference target point. The theoretical coordinates of the target point are the coordinate values ​​of the tooling in the aircraft coordinate system. These are used for subsequent coordinate system transformations in measurements.

[0021] Step 5: Import the hole location model data. Extract the center coordinates of all holes from the aircraft theoretical model, number them, and import them into the software in .txt format as theoretical target points. These coordinates are in the aircraft coordinate system.

[0022] Step 6: Import the coordinates of the hole-making tool frame points. For example... Figure 9 As shown, the frame points include the frame points of the reamer / spot drill body and the drill bit frame points. The frame points are obtained by capturing images of the target points attached to the surface of the target frame using a binocular tracking camera.

[0023] Step 7: Binocular tracking camera coordinate system transformation. The camera captures images of four reference targets on the assembly fixture, obtaining their coordinate values ​​in the camera's own measurement coordinate system. Using the coordinate values ​​of the reference targets imported in Step 4 as a reference, a coordinate system transformation is performed, converting the measurement coordinate system to the aircraft coordinate system. After the transformation, the target point coordinate values ​​captured by the camera are all in the aircraft coordinate system.

[0024] Step 8: The operator wears AR glasses. The operator wears AR glasses and communicates with the computer via the network, ensuring that the software analysis results displayed on the computer screen can be seen in real time through the glasses.

[0025] Step 9: The operator holds the trackable drilling tool. The operator attaches the drill bit to the drilling tool, connects the tool's power source, forming a complete trackable drilling tool. The operator moves the tool within the camera's field of view and checks whether the software can display the tool's real-time pose information.

[0026] Step 10: The operator locates the drilling point on the product. The operator holds the trackable drilling tool and moves it to the area of ​​the product to be drilled, bringing the drill tip to the product surface to check the pose deviation information of the AR glasses.

[0027] Step 11: Tracking the spatial pose of the drilling tool. When the operator holds the drilling tool and it enters the camera's field of view, the camera can track the spatial pose of the drill bit tip based on the drilling tool frame points imported in Step 6.

[0028] Step 12: Analyze the positional deviation of the drilling tool tip. Based on the drilling point model data imported in Step 5, analyze the deviation of the current tip position. The analysis strategy is to calculate the distance between the current tip coordinates and the coordinates of all point models in the software. Find the value with the smallest distance and display it on the software interface. The interface will display the ID, X, Y, and Z axis deviations, and 3D distance of that theoretical point in real time.

[0029] Step 13: Transmit the information to the AR glasses. Transmit the deviation information displayed in real time by the software in Step 12 to the AR glasses, allowing the operator to view the current deviation of the drill bit tip from the theoretical hole position in real time.

[0030] Step 14: The operator adjusts the tool's pose based on the information. The operator adjusts the tool's pose according to the deviation information displayed in the AR glasses until the deviation is within the acceptable range required by the process specifications, at which point the adjustment is stopped.

[0031] Step 15: Drilling. Activate the trackable drilling tool to drill the hole.

[0032] Step 16: Locate the next hole. Repeat the above steps until all holes have been machined.

[0033] Trackable hole-making tool

[0034] A hole-making tool has been invented that allows the position of the installed drill bit tip to be tracked in real time by a binocular tracking camera. This eliminates the need for manual marking, and the operator can hold the tool and determine the hole position according to the instructions in the software to make the hole.

[0035] The principle behind the tracking of drilling tools is as follows: After assembling the pneumatic drill / electric drill, the target frame, the target, and the process marker, a binocular tracking camera is used to photograph the target on the tool, obtaining data such as... Figure 9The frame points in the right-hand view include the body frame points and the drill bit frame points. This set of points is output as a TXT file and imported into the binocular tracking camera software. The software defines this set of points as the frame points of a rigid component. The process benchmark is then replaced with the drill bit, with the drill bit tip coinciding with the center of the process benchmark target. When a trackable drilling tool enters the binocular tracking camera's field of view, and at least four target point coordinates correspond to the four points in the TXT file, the binocular tracking camera recognizes the target as the rigid component corresponding to the TXT file. At this point, the software interface displays the data for the entire frame points, such as... Figure 9 Regardless of whether other target points are currently visible, because these target points are a whole, obtaining the positions of a portion of the target points allows us to obtain a 4×4 coordinate transformation matrix using the coordinates of the corresponding points in the txt file. Since every point in rigid body motion has the same coordinate transformation matrix, the software can obtain the spatial pose of the entire rigid body from the currently visible points, and thus obtain the coordinates of the drill bit tip. Because the current measurement is in the aircraft coordinate system, the drilling position can be determined by calculating the deviation between the drill bit tip coordinates and the theoretical drilling point.

[0036] The tool consists of five parts: pneumatic drill / electric drill, target frame, target, process guide, and drill bit. Figure 3 .

[0037] Pneumatic drills / electric drills: These are standard hole-making tools, bolted together with the main target frame to form a single unit. Process markers and drill bits can be mounted separately on the chuck of the pneumatic drill / electric drill. Figure 4 .

[0038] The target frame is used for target placement and maintaining the relative positions of the target points. It is bolted to the pneumatic drill / electric drill. The frame consists of multiple aluminum target mounts and carbon fiber connecting rods, forming a rigid polyhedron. Its internal dimensions are large enough to enclose most of the pneumatic drill / electric drill body. Figure 5 .

[0039] Targets: High-reflectivity photogrammetry targets with an outer diameter of 20mm and an inner diameter of 10mm are selected. The targets are attached to the target holders on both the main target frame and the process marker. The center of the target attached to the process marker should coincide with the center of the target holder. Targets attached to the main target frame only need to be secure and not overlap. Figure 6 .

[0040] Process marker: A process marker is used to replace the drill bit and serves to mark the position of the drill bit tip. It is mounted on the chuck of a pneumatic / electric drill and is made of aluminum alloy, consisting of a shank and a target seat. The shank diameter is 6mm, and the target seat diameter is 20mm with a thickness of 6mm. The overall length of the process marker should not exceed 30mm and should match the length of the drill bit. Figure 7 .

[0041] The effects of the invention

[0042] This invention solves the problems of high cost and poor accessibility of automated hole making, and unstable accuracy and low efficiency of traditional manual hole making. It combines machine vision with operator skills to form a human-machine collaborative precision hole making system, avoiding the manual marking process and significantly improving the efficiency of manual hole making. Utilizing digital measurement technology ensures the accuracy of hole positioning, improving aircraft assembly quality, making the entire operation process precise, controllable, and efficient, and enhancing the level of intelligence in aircraft manufacturing. Attached Figure Description

[0043] Figure 1 A schematic diagram of the components of a human-machine collaborative precision hole-making system;

[0044] Figure 2 A flowchart illustrating the operation of a human-machine collaborative precision hole-making system;

[0045] Figure 3 This is a schematic diagram of a traceable hole-making tool.

[0046] Figure 4 This is a schematic diagram of a pneumatic drill / electric drill;

[0047] Figure 5 This is a schematic diagram of the ontological target framework;

[0048] Figure 6 This is a schematic diagram of the target.

[0049] Figure 7 This is a schematic diagram of the process benchmark.

[0050] Figure 8 This is a schematic diagram of a drill bit;

[0051] Figure 9 This is a schematic diagram of the frame points for the hole-making tool.

[0052] In the diagram: 1-Assembly fixture, 2-Benchmark target, 3-Product, 4-Binocular tracking camera, 5-Computer, 6-Trackable hole-making tool, 7-Operator, 8-AR glasses, 51-Pneumatic drill / electric drill, 52-Body target frame, 53-Target, 54-Process benchmark, 55-Drill bit, 11-Body frame point, 12-Drill bit frame point. Detailed Implementation

[0053] A method for using a human-machine collaborative precision hole-making system, comprising the following steps:

[0054] Step 1: Product placement. Position the product on the assembly fixture, using the fixture to position the product at its theoretical location in the aircraft coordinate system.

[0055] Step 2: Set up the binocular tracking camera. Set up the binocular tracking camera according to the position of the assembly fixture and the drilling area of ​​the product, ensuring that the camera's measurement field of view covers the assembly fixture and the drilling area of ​​the product.

[0056] Step 3: Connect to the computer and open the software. Install data processing and analysis software on the computer. The software can communicate with the stereo tracking camera to obtain the coordinates of the target points captured by the camera in real time (the coordinates at this time are the coordinate values ​​in the camera's own measurement coordinate system), and highlight the target points in the camera image.

[0057] Step 4: Import the theoretical coordinates of the assembly tooling reference target point. The theoretical coordinates of the target point are the coordinate values ​​of the tooling in the aircraft coordinate system. These are used for subsequent coordinate system transformations in measurements.

[0058] Step 5: Import the hole location model data. Extract the center coordinates of all holes from the aircraft theoretical model, number them, and import them into the software in .txt format as theoretical target points. These coordinates are in the aircraft coordinate system.

[0059] Step 6: Import the coordinates of the hole-making tool frame points. For example... Figure 9 As shown, the frame points include the frame points of the drilling tool body and the drill bit frame points. The frame points are obtained by using a binocular tracking camera to photograph the target points bonded to the surface of the trackable drilling tool.

[0060] Step 7: Binocular tracking camera coordinate system transformation. The camera captures images of four reference targets on the assembly fixture, obtaining the coordinate values ​​in the camera's own measurement coordinate system. Using the coordinate values ​​of the reference targets imported in Step 4 as a reference, a coordinate system transformation is performed to convert the measurement coordinate system to the aircraft coordinate system. After the transformation, the target point coordinate values ​​captured by the camera are all coordinate values ​​in the aircraft coordinate system.

[0061] Step 8: The operator wears AR glasses. The operator wears AR glasses and communicates with the computer via the network, ensuring that the software analysis results displayed on the computer screen can be seen in real time through the glasses.

[0062] Step 9: The operator holds the trackable drilling tool. The operator attaches the drill bit to the drilling tool, connects the tool's power source, forming a complete trackable drilling tool. The operator moves the tool within the camera's field of view and checks whether the software can display the tool's real-time position information.

[0063] Step 10: The operator locates the drilling point on the product. The operator holds the drilling tool and moves it to the area of ​​the product to be drilled, bringing the drill bit tip into contact with the product surface, and checks the pose deviation information of the AR glasses.

[0064] Step 11: Tracking the spatial pose of the drilling tool. When the operator holds the drilling tool and it enters the camera's field of view, the camera can track the spatial pose of the drill bit tip based on the drilling tool frame points imported in Step 6.

[0065] Step 12: Analyze the positional deviation of the drilling tool tip. Based on the drilling point model data imported in Step 5, analyze the deviation of the current tip position. The strategy is to calculate the distance between the current tip coordinates and the coordinates of all point models in the software. Find the value with the smallest distance and display it on the software interface. The interface will display the ID of the theoretical point, the deviations in the X, Y, and Z directions, and the 3D distance (the 3D distance is the length of the line connecting two points in space) in real time.

[0066] Step 13: Transmit the information to the AR glasses. Transmit the deviation information displayed in real time by the software in Step 12 to the AR glasses, allowing the operator to view the current deviation of the drill bit tip from the theoretical hole position in real time.

[0067] Step 14: The operator adjusts the tool's pose based on the information. The operator adjusts the tool's pose according to the deviation information displayed in the AR glasses until the deviation is within the acceptable range required by the process specifications, at which point the adjustment is stopped.

[0068] Step 15: Drilling. Activate the trackable drilling tool to drill the hole.

[0069] Step 16: Locate the next hole. Repeat the above steps until all holes have been machined.

Claims

1. A human-machine collaborative precision hole making system, characterized in that: It includes an assembly tool (1), a reference target (2), a product (3), a binocular tracking camera (4), a computer (5), a trackable hole-making tool (6), and AR glasses (8); The assembly tool (1) is used for positioning and clamping the product (3) to ensure that the assembled product is in the correct position in the aircraft coordinate system; the assembly tool is welded from steel plates, and the product locator is screwed to the surface of the plate; the reference target is installed in an open area on the surface of the jig; The product (3) is composed of various parts, which are mounted on an assembly tool (1) and positioned and clamped by a positioner on the tool; The binocular tracking camera (4) is used to track the real-time spatial position of the trackable hole-making tool (6) in the aircraft coordinate system; The computer (5) is installed with data analysis software for processing data collected by the binocular tracking camera (4), analyzing the deviation between the position of the trackable hole-making tool (6) and the theoretical hole-making position, and transmitting the deviation information to the AR glasses (8); The trackable hole-making tool (6) can be tracked by a binocular tracking camera (4), and the coordinate value of the drill tip of the hole-making tool in the aircraft coordinate system can be determined in real time, and used for hole-making processing of the product (3), as shown in FIG3 ; The operator (7) holds the trackable hole-making tool (6), wears AR glasses (8), adjusts the hole-making position according to the position deviation information provided by the computer (5), and completes the hole-making work; The AR glasses (8) receive information from the computer (5) and are used to display the position deviation information of the trackable hole-making tool (6) in real time. The AR glasses are worn on the head by the operator (7).

2. A human-machine collaborative precision hole making system as claimed in claim 1, characterized in that: The reference target (2) is used for the conversion of the measurement coordinate system of the binocular tracking camera (4). The target needs to be firmly mounted on the assembly tool (1) by magnetic attraction. The reference target (2) is a hemisphere with a diameter of 0.5 inches. A high-reflective photogrammetry sticker with an outer diameter of 10 mm and an inner diameter of 6 mm is pasted at the center of the sphere.

3. A human-machine collaborative precision hole making system as claimed in claim 1 or 2, characterized in that: Also included is a tracked hole-making tool, including a pneumatic drill / electric drill (51), a body target frame (52), a target (53), a process benchmark (54), and a drill bit (55); The pneumatic drill / electric drill (51) is bolted to the main body target frame (52) to form a whole, and the chuck of the pneumatic drill / electric drill can be respectively installed with a process benchmark (54) and a drill bit (55); The main body target frame (52) is used for arranging target points and maintaining the relative positions between target points, and is connected to the pneumatic drill / electric drill (51) by bolts; the frame is composed of a plurality of aluminum target seats and carbon fiber connecting rods to form a polyhedron with good rigidity, and the internal space size can ensure that most of the body of the pneumatic drill / electric drill (51) is enclosed.

4. A human-machine collaborative precision hole making system as claimed in claim 3, characterized in that: The target (53) is a high-reflective photogrammetry target with an outer diameter of 20 mm and an inner diameter of 10 mm. The target is respectively pasted on the target seat of the main target frame (52) and the target seat of the process benchmark (54); the center of the target pasted on the target seat of the process benchmark (54) should coincide with the center of the target seat.

5. A human-machine collaborative precision hole making system as claimed in claim 3 or 4, characterized in that: The process benchmark (54) is used to replace the drill bit (55) and plays a role in calibrating the position of the drill bit tip; the process benchmark is installed on the chuck of the pneumatic drill / electric drill (51) and consists of a rod handle and a target seat.

6. The human-machine collaborative precision hole making system according to claim 5, characterized in that: The process benchmark (54) is made of aluminum alloy.

7. The human-machine collaborative precision hole making system according to claim 5, characterized in that: The handle diameter of the process benchmark (54) is 6 mm, the target base diameter is 20 mm, and the thickness is 6 mm. The overall length of the process benchmark does not exceed 30 mm, and the length is consistent with that of the drill bit (55).

8. A working method of a human-machine collaborative precision hole making system, characterized in that: Here are the steps: Step 1: Put the product on the shelf; position the product on the assembly tooling, and use the tooling to position the product at the theoretical position of the aircraft coordinate system; Step 2: Set up a binocular tracking camera; according to the position of the assembly tooling and the product hole-making area, set up a binocular tracking camera to ensure that the camera's measurement field of view covers the assembly tooling and the product hole-making area; Step 3: Connect the computer and open the software; Data processing and analysis software is installed on the computer. The software can communicate with the binocular tracking camera, obtain the coordinates of the target points that the camera can capture in real time, and highlight the target points in the camera image; Step 4: Import the theoretical coordinates of the assembly tooling benchmark target point; the theoretical coordinates of the target point are the coordinate values ​​of the tooling in the aircraft coordinate system; they are used for subsequent measurement coordinate system conversion; Step 5: Import the hole-making point model data; extract the center coordinate values ​​of all holes from the aircraft theoretical model, number them, and import them into the software in txt format as theoretical target points; the coordinates are the coordinate values ​​in the aircraft coordinate system; Step 6: Import the coordinates of the hole-making tool frame points; the frame points include the frame points of the air drill / spot drill body and the frame points of the drill bit; The frame points are obtained by using a binocular tracking camera to shoot the target points attached to the surface of the target frame; Step 7: Coordinate system conversion of binocular tracking camera; the camera shoots the four reference targets on the assembly tooling, obtains the coordinate values ​​in the camera's own measurement coordinate system, and uses the coordinate values ​​of the tooling reference targets imported in the fourth step as a reference to perform coordinate system conversion, converting the measurement coordinate system to the aircraft coordinate system; after the conversion, the coordinate values ​​of the target points shot by the camera are all coordinate values ​​in the aircraft coordinate system; Step 8: The operator wears AR glasses; the operator wears AR glasses and communicates with the computer through the network to ensure that the glasses can display the software analysis results displayed on the computer screen in real time; Step 9: The operator holds the trackable hole-making tool; the operator installs the drill bit on the hole-making tool, connects the tool power source, forms a complete trackable hole-making tool, moves the tool in the camera field of view, and checks whether the real-time posture information of the tool can be displayed in the software; Step 10: The operator searches for the hole-making point on the product; the operator holds the trackable hole-making tool and moves it to the area of ​​the product to be holed, touches the drill tip to the surface of the product, and checks the posture deviation information of the AR glasses; Step 11: Tracking the spatial position of the drilling tool; when the operator holds the drilling tool and enters the camera's field of view, the camera can track the spatial position of the drill tip based on the drilling tool frame points imported in step 6; Step 12: Analyze the position deviation of the tool tip of the hole-making tool; According to the hole-making point model data imported in step 5, perform deviation analysis on the current tool tip position; The analysis strategy is to calculate the distance between the coordinate value of the current tool tip and the coordinate value of all point models in the software; Find the value with the smallest distance and display it in the software interface, which will display the ID of the theoretical point, the deviation in the X, Y, Z directions and the 3D distance in real time; Step 13: Transmit the information to the AR glasses; transmit the deviation information displayed in real time by the software in step 12 to the AR glasses, and the operator can view the deviation of the current drill tip from the theoretical hole position in real time; Step 14: The operator adjusts the tool posture according to the information; the operator adjusts the tool posture according to the deviation information in the AR glasses until the deviation is within the qualified range of the process requirements and then stops the adjustment action; Step 15: Drilling: Start the trackable drilling tool to drill holes; Step 16: Find the next hole position; repeat the above steps until all holes are processed.

Images