A human-machine collaborative drilling and riveting system and method based on augmented reality and visual assistance
Through augmented reality and vision-assisted human-machine collaborative drilling and riveting system, combined with industrial robots and operators, the drilling and riveting hole position sequence and coordination gap are optimized in real time, and the problem of low quality and efficiency of drilling and riveting in small spaces in complex structures is solved, and the problem of low quality and efficiency of drilling and riveting in small spaces is realized. High-quality and efficient drilling and riveting effect.
Patent Information
- Application Number
- CN202210884257.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-07-26
AI Technical Summary
The existing aircraft drilling and riveting technology is difficult to achieve high-quality and efficient drilling and riveting technology in a small space with complex structures. Manual drilling and riveting efficiency is low and automated drilling and riveting equipment is difficult to flexibly adjust the drilling and riveting sequence, resulting in poor quality and low efficiency. .
Adopt a human-machine collaborative drilling and riveting system based on augmented reality and vision assistance, combining industrial robots, drilling and riveting actuators, operators, augmented reality equipment, drilling and riveting tools and depth cameras to collect and identify operators' working postures and tops in real time. The tightening state is optimized, and the order of drilling and rivet holes and the coordination gap control are coordinated to realize the human-machine cooperation throughout the drilling and riveting process.
It significantly improves the quality and efficiency of drilling and rivets in the narrow space of complex structures, ensures the consistency and safety of the drilling and rivet process, and meets the high-quality and high-efficiency drilling and rivets in the complex structure of aircraft.
Smart Images

Figure CN115139314B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a human-machine collaborative drilling and riveting system and method based on augmented reality and visual assistance, and belongs to the fields of mechanical engineering and aircraft assembly. Background Art
[0002] Drilling and riveting are the main connection methods for aircraft assembly, which mainly include two processes: drilling and riveting. The process quality directly affects the assembly quality and service reliability of the aircraft structure. At present, the main drilling and riveting methods for aircraft are manual drilling and riveting and automated drilling and riveting. Manual drilling and riveting refers to two operators holding drilling and riveting tools to perform drilling and riveting operations through mutual cooperation. It has the advantages of strong flexibility and good spatial adaptability, but its efficiency is low, and it is difficult to effectively control the matching gap between the structures to be assembled by applying the top tightening force using existing manual drilling and riveting tools, resulting in poor drilling and riveting process quality. Automatic drilling and riveting is divided into two forms: machine tool drilling and riveting and industrial robot drilling and riveting. It has good process quality and high efficiency, but is limited by the size of the equipment itself. It is only suitable for parts with simple structure and good openness, and it is difficult to flexibly adjust the drilling and riveting sequence according to the actual process status to optimize the process quality. It is easy to aggravate the cumulative transmission of drilling and riveting deviations, resulting in excessive overall deformation of the structure. The new generation of aircraft continues to pursue faster cruising speeds, stronger stealth performance and higher fuel efficiency, which has led to increasingly complex structures, resulting in narrow assembly work space and poor openness, which directly restricts the application of automated drilling and riveting technology. Currently, the assembly of complex structures in small spaces and poor openness still uses manual drilling and riveting, resulting in poor quality and low efficiency of the drilling and riveting process.
[0003] The human-machine collaborative operation mode combines human dexterity with high-precision automation of machine equipment, providing an effective solution to the above problems. At present, some studies have proposed human-machine collaborative riveting operation modes, but they are only for a single riveting link and do not cover the drilling link; at the same time, there is a lack of solutions to key process problems such as flexible adjustment of drilling and riveting sequence and control of drilling and riveting matching gaps; and in terms of human-machine collaborative interaction, how to ensure the continuity and safety of the collaborative drilling and riveting process is not considered enough. Therefore, it is still difficult to meet the needs of high-quality and efficient drilling and riveting processes in small spaces with complex structures, and new technologies are urgently needed. Summary of the invention
[0004] The purpose of the present invention is to provide a human-machine collaborative drilling and riveting system and method based on augmented reality and visual assistance, so as to realize human-machine collaboration in the whole process of drilling and riveting, so as to overcome the problems that it is difficult to apply automatic drilling and riveting equipment in the narrow space of the current complex structure of aircraft, and only a single manual operation can be used, resulting in poor drilling and riveting quality and low efficiency, and greatly improve the drilling and riveting quality and efficiency in the narrow space of the complex structure of aircraft.
[0005] The object of the present invention is achieved in one aspect as follows: a human-machine collaborative drilling and riveting system based on augmented reality and visual assistance, characterized in that it includes an industrial robot, a drilling and riveting actuator, an operator, an augmented reality device, a drilling and riveting integrated tool, a depth camera and a host computer;
[0006] The industrial robot is located outside the wall panel with an open space, and is used to move the drilling and riveting actuator to the hole position to be operated;
[0007] The drilling and riveting actuator is installed on the end flange of the industrial robot and is used to tighten the wall plate, drill holes, feed nails, rivet and reset operations;
[0008] The operator is located inside a narrow wall panel and performs drilling and riveting in collaboration with the robot and the drilling and riveting actuator by wearing an augmented reality device and holding a drilling and riveting integrated tool;
[0009] The augmented reality device is used for information and instruction interaction between the operator and the host computer;
[0010] The drilling and riveting integrated tool is used to assist the operator to perform collaborative drilling and riveting with the robot and the drilling and riveting actuator;
[0011] The depth camera is used to collect the operator's working posture information in real time;
[0012] The host computer is used to integrate the industrial robot, the drilling and riveting actuator, the operator, the augmented reality device, the drilling and riveting integrated tool and the depth camera into an overall system.
[0013] Furthermore, the drilling and riveting tool is composed of a pneumatic rivet gun, a mechanical sleeve and a pressure sensor; the pneumatic rivet gun is used for riveting operations; the mechanical sleeve is arranged on the pneumatic rivet gun and used as a push iron when drilling; when riveting, it can be freely extended and retracted to assist the pneumatic rivet gun in riveting operations; the pressure sensor is arranged at the front end of the mechanical sleeve, and is used to collect the tightening force status information applied by the operator during the operation.
[0014] Furthermore, the mechanical sleeve includes a base, a telescopic rod, a pull ring, a stopper rod, a locking block and a tensioning block; the base is sleeved on both sides of the pneumatic rivet gun through a clamp, a guide rail and a slider are arranged in the base, the telescopic rod is connected to the guide rail by bolts, and the slider is fixed to the inner side of the base by bolts, a spring 1 is sleeved on the outer side of the telescopic rod and the guide rail, a stopper rod is transversely arranged on the telescopic rod, the spring 1 is compressed under the action of the stopper rod, a limiting groove is arranged on the outer side of the telescopic rod, the end of the locking block is against the outer side of the telescopic rod, a card slot is arranged on the tail of the locking block, and a spring 2 is sleeved on the tail, the tail end of the locking block is placed on the tensioning block, a spring 3 is sleeved on the tensioning block, and the lower end of the tensioning block is connected to the pull ring; the pressure sensor is connected to the front end of the telescopic rod through the front baffle by bolts.
[0015] Another aspect of the purpose of the present invention is achieved as follows: a human-machine collaborative drilling and riveting method based on augmented reality and visual assistance comprises the following steps:
[0016] A human-machine collaborative drilling and riveting method based on augmented reality and visual assistance, characterized in that it includes the following steps:
[0017] 1) The operator evaluates and optimizes the drilling and riveting sequence according to the actual process conditions on site, and determines the hole positions to be operated; the operator sends the hole position information to be operated to the host computer through the augmented reality device; the host computer sends the hole position information to the industrial robot; the industrial robot moves the drilling and riveting actuator to the hole position to be operated according to the hole position information to be operated, and sends its own in-place status information to the host computer; the host computer converts the in-place status information of the industrial robot into a wall panel tightening operation instruction and sends it to the drilling and riveting actuator; the drilling and riveting actuator performs the operation of tightening the outer wall panel and sends the presser foot status information to the host computer;
[0018] 2) The operator holds the drilling and riveting tool and tightens the wall panel from the inside of the wall panel; the drilling and riveting tool sends the operator's tightening force status information to the host computer; the depth camera collects the operator's working posture information and sends it to the host computer; the host computer recognizes the operator's working posture information and judges the tightening force status information; according to the operator's working posture recognition result and the operator's tightening force status, the host computer sends a drilling and riveting operation instruction or a drilling and riveting operation stop instruction to the drilling and riveting actuator;
[0019] 3) The operator uses the drilling and riveting integrated tool to cooperate with the drilling and riveting actuator to complete the drilling and riveting operation. During the collaborative drilling and riveting process, the drilling and riveting actuator sends the drilling and riveting operation status information to the host computer;
[0020] 4) During the entire human-machine collaborative drilling and riveting process, the host computer sends the industrial robot's in-place status information, the drilling and riveting actuator's pressure foot status information, the drilling and riveting operation status information, and the drilling and riveting tool's top tightening force status information to the augmented reality device in real time; the operator can use the augmented reality device to grasp the industrial robot's in-place status, the drilling and riveting actuator's pressure foot status, the drilling and riveting operation status, and the drilling and riveting tool's top tightening force status in real time, and can check the on-site product and process file information at any time.
[0021] In the step 2), the operator's working posture recognition result includes a drilling and riveting posture and a non-drilling and riveting posture. When the operator's posture recognition result is a drilling and riveting posture and the operator's top tightening force state meets the drilling and riveting requirements, the host computer sends a drilling and riveting operation instruction to the drilling and riveting actuator; when the operator's posture recognition result is a non-drilling and riveting posture or the top tightening force state does not meet the drilling and riveting requirements, the host computer sends a stop drilling and riveting operation instruction to the drilling and riveting actuator.
[0022] The step 3) specifically includes: in the drilling stage, the operator holds the drilling and riveting tool to press the wall panel from the inside of the wall panel, and the drilling and riveting actuator performs the drilling operation from the outside of the wall panel; in the riveting stage, the drilling and riveting actuator first completes the nail feeding and riveting operation from the outside of the wall panel, and then the operator holds the drilling and riveting tool to complete the riveting operation from the inside of the wall panel; during the entire drilling and riveting process, the operator always keeps the holding of the drilling and riveting tool to press the wall panel, so as to control the matching clearance between the wall panels within the allowable range.
[0023] The operator holds the drilling and riveting tool to push the wall panel from the inner side of the wall panel specifically includes: the operator holds the drilling and riveting tool to press the wall panel, so that the telescopic rod of the mechanical sleeve moves backward; at the same time, the operator pulls the pull ring downward, and the tensioning block also moves downward; at this time, the tensioning block stops limiting the locking block slot, and the locking block moves toward the telescopic rod under the action of the second spring; when the telescopic rod moves backward for a certain displacement, the locking block returns to the limiting slot of the telescopic rod under the action of the second spring; at this time, the mechanical sleeve locks the entire device and is used as a push iron, and the operator applies corresponding tightening force to the aircraft wall panel.
[0024] The operator holds the drilling and riveting tool to complete the riveting operation from the inside of the wall panel, which specifically includes: the operator appropriately reduces the tightening force while maintaining the pressure on the wall panel, so that the pneumatic rivet gun rebounds backward under the action of spring 1 of the mechanical sleeve; at the same time, the telescopic rod moves forward relative to the pneumatic rivet gun to push out the locking block; at this time, the tensioning block is clamped in the slot of the locking block under the action of spring 3; at this time, spring 1 is retracted and expanded as needed, so that the pneumatic rivet gun moves forward to press the rivet, completing the riveting operation.
[0025] In the step 4), the drilling and riveting operation status information includes tool drilling status information, nail feeding status information and end reset information; the product and process file information includes hole position, hole diameter, riveting method, rivet specifications and operator tightening force value requirements.
[0026] The present invention provides a human-machine collaborative drilling and riveting system and method based on augmented reality and visual assistance, which has the advantages and functions of: the human-machine collaborative drilling and riveting system and method of the present invention covers two process links of drilling and riveting, can realize human-machine collaboration in the whole process of drilling and riveting, and has strong process adaptability and practicality. By utilizing human experience and flexibility, the operator can evaluate the working status in real time and optimize the drilling and riveting hole sequence, which can significantly inhibit the transmission and accumulation of drilling and riveting deviations and effectively control the deformation of complex structural assembly. The pressure sensor of the drilling and riveting tool is used to collect information in real time, and the augmented reality device is used to display the state information of the tightening force applied by the operator in real time. Combined with the state switching of the mechanical sleeve of the drilling and riveting tool, it can ensure that the operator always applies the required tightening force to press the wall plate during the whole process of drilling and riveting, and control the matching clearance within the allowable range, which greatly improves the quality of the drilling and riveting process. With the help of augmented reality and visual assistance technology, and through the host computer, the industrial robot, drilling and riveting actuator, operator, augmented reality equipment, drilling and riveting tool and depth camera are integrated into an overall system. The operator can understand the working status of the robot and drilling and riveting actuator in real time and perform corresponding collaborative actions. The operation execution of the robot and drilling and riveting actuator also depends on the operator's instructions and working status, which effectively ensures the continuity and safety of human-machine collaboration during the whole process of drilling and riveting. The human-machine collaborative drilling and riveting system and method of the present invention have good use effects, and can meet the requirements of high-quality and high-efficiency drilling and riveting in the narrow space of complex aircraft structures in terms of drilling and riveting quality, efficiency, safety, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 Block diagram of the system of the present invention.
[0028] Figure 2 A schematic diagram of the operating principle of the drilling and riveting integrated tool of the present invention.
[0029] Figure 3 A schematic structural diagram of a mechanical sleeve of a drilling and riveting integrated tool according to the present invention.
[0030] Figure 4 A schematic structural diagram of the mechanical sleeve locking block of the present invention.
[0031] Figure 5 A schematic structural diagram of a telescopic rod of the present invention.
[0032] Figure 6 Signal logic diagram of the present invention.
[0033] Figure 7 The overall operation flow chart of the present invention.
[0034] Among them, 1 pneumatic rivet gun, 2 mechanical sleeve, 2-1 base, 2-2 telescopic rod, 2-2-1 limit slot, 2-3 pull ring, 2-4 locking block, 2-4-1 slot, 2-5 tensioning block, 2-6 spring one, 2-7 spring two, 2-8 spring three, 2-9 guide rail, 2-10 slider, 2-11 gear lever, 3 pressure sensor, 4 clamp, 5 front baffle. DETAILED DESCRIPTION
[0035] like Figure 1 A human-machine collaborative drilling and riveting system based on augmented reality and visual assistance is shown, including an industrial robot, a drilling and riveting actuator, an operator, an augmented reality device, a drilling and riveting integrated tool, a depth camera and a host computer;
[0036] The industrial robot is located outside the open wall panel and is used to move the drilling and riveting actuator to the hole to be worked.
[0037] The drilling and riveting actuator is installed on the end flange of the industrial robot and is used to tighten the wall panel, drill holes, feed nails, rivet and reset operations.
[0038] The operator is located on the inner side of the wall panel in a narrow space. He collaborates with the robot and the drilling and riveting actuator to perform drilling and riveting by wearing an augmented reality device and holding a drilling and riveting tool.
[0039] Augmented reality devices are used for information and command interaction between operators and host computers.
[0040] The drilling and riveting integrated tool is used to assist the operator in performing drilling and riveting in collaboration with the robot and the drilling and riveting actuator.
[0041] The depth camera is used to collect the operator's working posture information in real time.
[0042] The host computer is used to integrate the industrial robot, drilling and riveting actuator, operator, augmented reality equipment, drilling and riveting tool and depth camera into an overall system.
[0043] like Figure 2-5 As shown, the drilling and riveting integrated tool is composed of a pneumatic riveting gun 1, a mechanical sleeve 2 and a pressure sensor 3; the pneumatic riveting gun 1 is used for riveting operation; the mechanical sleeve 2 is sleeved on the pneumatic riveting gun 1 and used as a top iron when drilling; when riveting, it can be freely extended and retracted to assist the pneumatic riveting gun 1 in riveting operation; the pressure sensor 3 is arranged at the front end of the mechanical sleeve 2, and is used to collect the state information of the top tightening force applied by the operator during the operation;
[0044] The mechanical sleeve 2 includes a base 2-1, a telescopic rod 2-2, a pull ring 2-3, a locking block 2-4 and a tensioning block 2-5; the base 2-1 is sleeved on both sides of the pneumatic rivet gun 1 through a clamp 4, a guide rail 2-9 and a slider 2-10 are arranged in the base 2-1, the telescopic rod 2-2 is connected to the guide rail 2-9 by bolts, the blocking rod 2-11 is transversely installed in the telescopic rod 2-2, the slider 2-10 is fixed to the inner side of the base 2-1 by bolts, and a spring is sleeved on the outer side of the telescopic rod 2-2 and the guide rail 2-9. 2-6, a limiting groove 2-2-1 is set on the outer side of the telescopic rod 2-2, the end of the locking block 2-4 is against the outer side of the telescopic rod 2-2, a slot 2-4-1 is set on the tail of the locking block 2-4, and a spring 2-7 is sleeved on the tail, the tail end of the locking block 2-4 is placed on the tensioning block 2-5, the tensioning block 2-5 is sleeved with a spring 3 2-8, and the lower end of the tensioning block 2-5 is connected to the pull ring 2-3; the pressure sensor 3 is connected to the front end of the telescopic rod 2-2 through the front baffle 5 by bolts.
[0045] like Figure 6-7 As shown, a human-machine collaborative drilling and riveting method based on augmented reality and visual assistance includes the following steps:
[0046] 1) The operator evaluates and optimizes the drilling and riveting sequence according to the actual process conditions on site, and determines the holes to be operated. The operator sends the hole information to be operated to the host computer through the augmented reality device; the host computer sends the hole information to be operated to the industrial robot; the industrial robot moves the drilling and riveting actuator to the hole to be operated according to the hole information to be operated, and sends its own in-place status information to the host computer; the host computer converts the in-place status information of the industrial robot into a wall panel tightening operation instruction and sends it to the drilling and riveting actuator; the drilling and riveting actuator performs the wall panel tightening operation and sends the presser foot status information to the host computer.
[0047] In step 1), the operator evaluates and optimizes the drilling and riveting sequence according to the actual process conditions on site, determines the hole positions to be operated, and uses the gesture tracking module in Microsoft HoloLens to click on the hole positions in the virtual reality interface, and HoloLens can send information to the host computer. The host computer sends the information received from HoloLens to the robot through the OPCUA protocol. The robot moves the drilling and riveting actuator to the specified hole position and sends its own motion status information to the host computer. After receiving the robot's arrival information, the host computer sends a wall panel pressing instruction to the drilling and riveting actuator through Beckhoff ADS, and the drilling and riveting actuator performs the wall panel pressing operation.
[0048] 2) The operator holds the drilling and riveting tool and tightens the wall panel from the inside of the wall panel; the drilling and riveting tool sends the operator's tightening force status information to the host computer; the depth camera collects the operator's working posture information and sends it to the host computer; the host computer recognizes the operator's working posture information and judges the tightening force status information; according to the operator's working posture recognition result and the operator's tightening force status, the host computer sends a drilling and riveting operation instruction to the drilling and riveting actuator or stops the drilling and riveting operation instruction.
[0049] Step 2) The host computer recognizes the operator's working posture, including the acquisition of the operator's working posture key points and the recognition of the operator's working posture; the operator's working posture acquisition uses the realsense L515 depth camera for image acquisition;
[0050] The acquisition of the operator's working posture key points is mainly combined with the openpose algorithm to process the captured images to extract the desired key point information; Openpose is an open source bottom-up operator posture recognition algorithm, which invented PAFs (Part Affinity Fields, which is a two-dimensional vector field that retains position and direction). Based on PAFs and key point positions, the degree of association of key points can be calculated, and then the complete operator key point information can be obtained.
[0051] The execution process is as follows: first input an image W*H, extract features through the convolutional neural network, obtain a feature map, and then divide it into two branches. One branch obtains the key point position (confidence maps), and the other branch represents the association of key points (PAFs), which is then converted into the matching problem of the operator's skeleton key points. For the image matching problem, the Hungarian algorithm is used (the Hungarian algorithm is a combinatorial optimization algorithm for solving task allocation in polynomial time). This structure decodes the global context and jointly learns through the two branches of the sequence structure neural network, which can obtain the operator's key points in real time with high precision, efficiency and accuracy.
[0052] Operator work posture recognition, based on the acquisition of the operator's work posture key points, first define the operator's work posture, and then identify it. The definition of the operator's work posture is divided into two steps. First, the operator's work posture needs to be collected. This process uses the realsense L515 depth camera for image collection. Then, match the work action label for each image to represent the name of the collected posture action.
[0053] The overall process of operator posture recognition is as follows: first input the video stream, use the openpose framework to extract the key point information of N frames, and preprocess this information for feature extraction. The preprocessing is divided into four steps:
[0054] 1. Unify the coordinates. Since the heights of different objects are different and their distances from the camera are different, the coordinates of the collected data are not unified. Therefore, the first thing to do is to unify the coordinates.
[0055] 2. Remove coordinate information with high similarity, such as head information. The head has five feature points, but the head information of each image is consistent, which has little significance for calculation;
[0056] 3. Fill in missing information. Joints are feature vectors in the feature classification process and must maintain a fixed size. Therefore, missing joints need to be filled with some values to supplement them.
[0057] 4. Skip blank frames or useless data and set some special points to ensure that the person identified is the operator. If no special points are identified, skip them.
[0058] The preprocessed information is input into the classifier for classification to obtain the final recognition result.
[0059] According to the above posture recognition process, in this system, the operator's posture movements during drilling and riveting are first observed; the camera is used to record them. The actions are defined as drilling and riveting, and there are two motion states: non-drilling and riveting. A drilling and riveting operation process is that the operator starts from the stop state, lifts the rivet gun to perform the iron pushing operation, then performs the riveting operation, and finally puts down the rivet gun to enter the non-drilling and riveting state. The method of operator posture recognition is: the camera sampling frequency is 10Hz, first input 5 frames of joint point data as the original features, and then extract more data features that are conducive to feature classification, such as the angle between the joints, the length of the limbs, the average height of the skeleton of the previous N frames, and the speed of the joints as features. The features are connected in series as feature vectors, and the PCA algorithm is applied to reduce the feature dimension, and these data are input into the machine learning classifier.
[0060] The training data is collected by more than 10 professional operators, imitating the operator's work flow. The process is as follows:
[0061] 1. The operator stops the machine (10 seconds)
[0062] 2. Operator is working (35s)
[0063] Each operator follows the time prompts. The camera that records the operator's actions is Intel Realsense L515, the sampling frame rate is set to 10 Hz, and the sampled images are saved according to the name.
[0064] The depth camera uploads the collected images to the host computer through the network for training and recognition; this program trains and recognizes images based on the openpose version of tensorflow, and the environment is tensorflow2.4.0. This program improves some codes based on the tensorflow1.0 version of tensorflow and upgrades it to tensorflow2.0 version. After configuring the environment, first collect the data set and input it into the neural network for training, generate the weight file trained_classifier.pickle, then define the signal received from the camera, define the data path, and then reference the weight file for recognition.
[0065] The host computer receives the tightening force status information from the pressure sensor at the end of the drilling and riveting tool through serial communication. First, the serial communication package in Python is introduced to remove and convert the data transmitted by the force sensor, and then the tightening force status information is judged. When the tightening force status information reaches different force value ranges, the host computer will set the augmented reality device to display in a certain state.
[0066] After the upper computer identifies the operator's working posture and judges the tightening force status information, it generates two kinds of working instructions, namely, drilling and riveting working instructions and non-drilling and riveting working instructions. When the operator's working posture is identified as drilling and riveting and the tightening force state reaches the specified force value range, the drilling and riveting working instruction is sent to the drilling and riveting actuator, and the drilling and riveting actuator performs the next drilling and riveting operation; when the operator's working posture is identified as non-drilling and riveting or the tightening force state does not reach the specified force value range, the non-drilling and riveting working instruction is sent to the drilling and riveting actuator, and the drilling and riveting actuator suspends work until it is identified as the drilling and riveting state, and then performs the next operation.
[0067] 3) The operator uses the drilling and riveting integrated tool to cooperate with the drilling and riveting actuator to complete the drilling and riveting operation. During the collaborative drilling and riveting process, the drilling and riveting actuator sends the drilling and riveting operation status information to the host computer.
[0068] In step 3), during the drilling phase, the operator holds the drill-and-rivet tool and presses the wall panel from the inside; the drill-and-rivet actuator performs the drilling operation from the outside of the wall panel. During the riveting phase, the drill-and-rivet actuator first completes the nail feeding and riveting operations from the outside of the wall panel, and then the operator holds the drill-and-rivet tool and completes the riveting operation from the inside of the wall panel. During the entire drilling and riveting process, the operator always holds the drill-and-rivet tool against the wall panel.
[0069] During the drilling and riveting stage, the operation process of the drilling and riveting integrated tool is as follows: when drilling, first the pressure sensor 3 presses against the wall plate, the telescopic rod 2-2 of the mechanical sleeve 2 moves backward, the spring 1 2-6 is compressed under the action of the blocking rod 2-11, and the operator stretches the pull ring 2-3 downward, and the tensioning block 2-5 also moves downward. At this time, the tensioning block 2-5 stops limiting the slot 2-4-1 of the locking block 2-4, and the locking block 2-4 moves toward the telescopic rod 2-2 under the action of the spring 2-7. When the telescopic rod 2-2 moves backward to a certain displacement, the locking block 2-4 is in the spring 2-7. Under the action, the pneumatic rivet gun 1 rebounds backward under the action of spring one 2-6, and the telescopic rod 2-2 moves forward relatively. At this time, the locking block 2-4 is pushed outward, and the tensioning block 2-5 is stuck in the slot 2-4-1 of the locking block 2-4 under the action of spring three 2-8. At this time, the spring one 2-6 can be retracted and retracted as needed without affecting the riveting operation.
[0070] 4) During the entire human-machine collaborative drilling and riveting process, the host computer sends the industrial robot's in-place status information, the drilling and riveting actuator's pressure foot status information, the drilling and riveting operation status information, and the drilling and riveting tool's top tightening force status information to the augmented reality device in real time; the operator can use the augmented reality device to grasp the industrial robot's in-place status, the drilling and riveting actuator's pressure foot status, the drilling and riveting operation status, and the drilling and riveting tool's top tightening force status in real time, and can check the on-site product and process file information at any time.
[0071] In step 4), the drilling and riveting operation status information includes tool drilling status information, nail feeding status information and end reset information; the product and process file information includes hole position, hole diameter, riveting method, rivet specification and operator tightening force value requirements.
[0072] like Figure 6 As shown, the host computer integrates the action information received from the industrial robot, drilling and riveting actuator, depth camera, and drilling and riveting tool and outputs it to the augmented reality device, communicates the signals of each sensor with the augmented reality device, and displays the required signals and drilling and riveting operation information in the augmented realization interface through the augmented reality device.
[0073] The host computer and the industrial robot use the OPCUA protocol to exchange information. The drilling and riveting actuator exchanges information with the host computer through the Beckhoff controller ADS interface. The drilling and riveting integrated tool transmits information with the host computer through serial communication. The host computer will merge the acquired data information into a string of characters to communicate with the augmented reality device through the TCP / IP protocol, so that the augmented reality device can communicate with different signals; the augmented reality device communicates with the host through the TCP / IP protocol, and the socket module in python is introduced to connect with the augmented reality device (client). Define the receiving variable of the pressure sensor data; declare the signal receiving variable of the drilling and riveting actuator, which includes the presser foot status information, tool drilling status information, nail feeding status information, top riveting status information and end reset information, etc.; declare the industrial robot information receiving variable, which includes the robot in place status information, etc.; the host computer receives the hole position information to be operated sent by the augmented reality device.
[0074] The host computer connects the received information into a string and sends it to the augmented reality device. The augmented reality device then obtains information from the industrial robot, drilling and riveting actuator, depth camera, and drilling and riveting tool force sensor, designs the information in a certain way, and displays it in the augmented reality interface.
[0075] In addition, operators can use augmented reality devices to view product and process file information including hole location, hole diameter, riveting method, rivet specifications, and operator tightening force requirements.
[0076] When the present invention works, Figure 7 As shown, in the scenario of human-machine collaborative drilling and riveting of aircraft panels, the operator wears an augmented reality device and holds a drilling and riveting tool. First, the virtual and real coordinates are aligned, and offline programming is used to determine the hole position information to be operated, including the characteristic coordinates and the coordinates that need to be calibrated. Each sensor self-checks and the communication is checked. The drilling and riveting actuator reaches the initial calibration point and performs an initial test. If the test passes, the augmented reality device indicates the initial point, the depth camera responds, and then the position to be drilled and riveted is locked to determine whether the drilling and riveting integrated tool and the drilling and riveting actuator are tightly against the wall panel. If they are tightly against the wall panel, the drilling and riveting actuator drills holes. The augmented reality device displays the force value range of the end of the handheld drilling and riveting integrated tool in real time, and the operator performs riveting. If the riveting action is completed, the augmented reality device updates the hole position information according to the operator's instructions, and repeats the above drilling (riveting) operation. When the last hole is reached, the gesture recognition of the AR virtual recognition is suspended to check whether all holes have been drilled and riveted. If drilling and riveting are completed, the gesture recognition indicates that drilling is completed, the depth camera is turned off, and the industrial robot returns to the zero position; if drilling and riveting are not completed, the positioning hole is continued to be identified, and the drilling and riveting collaborative operation is continued; if the riveting action is not completed, the gesture recognition is interrupted, and inspection and repair are carried out. After repair, the drilling and riveting operation is continued.
[0077] The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solution disclosed in the present invention, technicians in this field can make some substitutions and deformations to some technical features therein according to the disclosed technical content without creative labor, and these substitutions and deformations are all within the protection scope of the present invention.
Claims
1. A human-machine collaborative drilling and riveting method based on augmented reality and visual assistance, characterized in that: The following steps are involved: 1) The operator evaluates and optimizes the drilling and riveting sequence according to the actual process conditions on site, and determines the hole positions to be operated; the operator sends the hole position information to be operated to the host computer through the augmented reality device; the host computer sends the hole position information to the industrial robot; the industrial robot moves the drilling and riveting actuator to the hole position to be operated according to the hole position information to be operated, and sends its own in-place status information to the host computer; the host computer converts the in-place status information of the industrial robot into a wall panel tightening operation instruction and sends it to the drilling and riveting actuator; the drilling and riveting actuator performs the operation of tightening the outer wall panel and sends the presser foot status information to the host computer; 2) The operator holds the drilling and riveting tool and tightens the wall panel from the inside of the wall panel; the drilling and riveting tool sends the operator's tightening force status information to the host computer; the depth camera collects the operator's working posture information and sends it to the host computer; the host computer recognizes the operator's working posture information and judges the tightening force status information; according to the operator's working posture recognition result and the operator's tightening force status, the host computer sends a drilling and riveting operation instruction or a drilling and riveting operation stop instruction to the drilling and riveting actuator; 3) The operator uses the drilling and riveting integrated tool to cooperate with the drilling and riveting actuator to complete the drilling and riveting operation. During the collaborative drilling and riveting process, the drilling and riveting actuator sends the drilling and riveting operation status information to the host computer; 4) During the entire human-machine collaborative drilling and riveting process, the host computer sends the industrial robot's in-place status information, the drilling and riveting actuator's pressure foot status information, the drilling and riveting operation status information, and the drilling and riveting tool's top tightening force status information to the augmented reality device in real time; the operator can use the augmented reality device to grasp the industrial robot's in-place status, the drilling and riveting actuator's pressure foot status, the drilling and riveting operation status, and the drilling and riveting tool's top tightening force status in real time, and can check the on-site product and process file information at any time.
2. According to claim 1, a human-machine collaborative drilling and riveting method based on augmented reality and visual assistance is characterized in that: In the step 2), the operator's working posture recognition result includes a drilling and riveting posture and a non-drilling and riveting posture; when the operator's posture recognition result is a drilling and riveting posture and the operator's tightening force state meets the drilling and riveting requirements, the host computer sends a drilling and riveting operation instruction to the drilling and riveting actuator; when the operator's posture recognition result is a non-drilling and riveting posture or the tightening force state does not meet the drilling and riveting requirements, the host computer sends a stop drilling and riveting operation instruction to the drilling and riveting actuator.
3. According to claim 1, a human-machine collaborative drilling and riveting method based on augmented reality and visual assistance is characterized in that: The step 3) specifically includes: in the drilling stage, the operator holds the drilling and riveting tool to press the wall panel from the inside of the wall panel, and the drilling and riveting actuator performs the drilling operation from the outside of the wall panel; in the riveting stage, the drilling and riveting actuator first completes the nail feeding and riveting operation from the outside of the wall panel, and then the operator holds the drilling and riveting tool to complete the riveting operation from the inside of the wall panel; during the entire drilling and riveting process, the operator always keeps the holding of the drilling and riveting tool to press the wall panel, so as to control the matching clearance between the wall panels within the allowable range.
4. According to claim 3, a human-machine collaborative drilling and riveting method based on augmented reality and visual assistance is characterized in that: The operator holds the drilling and riveting tool to push the wall panel from the inner side of the wall panel specifically includes: the operator holds the drilling and riveting tool to press the wall panel, so that the telescopic rod of the mechanical sleeve moves backward; at the same time, the operator pulls the pull ring downward, and the tensioning block also moves downward; at this time, the tensioning block stops limiting the locking block slot, and the locking block moves toward the telescopic rod under the action of the second spring; when the telescopic rod moves backward for a certain displacement, the locking block returns to the limiting slot of the telescopic rod under the action of the second spring; at this time, the mechanical sleeve locks the entire device and is used as a push iron, and the operator can apply corresponding tightening force to the aircraft wall panel.
5. According to claim 3, a human-machine collaborative drilling and riveting method based on augmented reality and visual assistance is characterized in that: The operator holds the drilling and riveting tool to complete the riveting operation from the inside of the wall panel, which specifically includes: the operator appropriately reduces the tightening force while maintaining the pressure on the wall panel, so that the pneumatic rivet gun rebounds backward under the action of spring 1 of the mechanical sleeve; at the same time, the telescopic rod moves forward relative to the pneumatic rivet gun to push out the locking block; at this time, the tensioning block is clamped in the slot of the locking block under the action of spring 3; at this time, spring 1 is retracted and expanded as needed, so that the pneumatic rivet gun moves forward to press the rivet, completing the riveting operation.
6. According to claim 1, a human-machine collaborative drilling and riveting method based on augmented reality and visual assistance is characterized in that: The drilling and riveting operation status information in step 4) includes tool drilling status information, nail feeding status information and end reset information; the product and process file information includes hole position, hole diameter, riveting method, rivet specification and operator tightening force value requirement.
Citation Information
Patent Citations
Clamping device for portable boring machine and portable air-drive-drilling machine with clamping device
CN103846471A
Multifunctional automatic drilling and riveting actuator for airplane assembly
CN106334940A
Method, system, computer program and a computer program product for working of a workpiece
CN109843488A
Automatic drilling and riveting device for container door seal structure
CN110802413A
Sensing system, work system, augmented-reality-image displaying method, and storage medium storing program
CN110871441A
