Virtual-reality fusion process guidance method and system for human-machine collaborative drilling and riveting

Abstract

The present invention provides a virtual-reality fusion process guidance method and system for human-machine collaborative drilling and riveting. The method guides human-machine collaborative drilling and riveting by adopting a virtual-reality fusion display method, including the integration of augmented reality information display, force display, robot control and other functions. The augmented reality information display can save workers the time and energy of consulting and memorizing various types of information. The force display is conducive to accurate force control. The control of the robot by the augmented reality device can reduce the operation of workers such as movement, thereby improving work efficiency.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a virtual-reality fusion process guidance method and system for human-machine collaborative drilling and riveting. Background Art

[0002] In the industrial scenario of drilling and riveting aircraft panels, robots offer high precision, stability, efficiency, and the ability to operate in hazardous environments. Human workers, on the other hand, possess greater perception, experience, and adaptability. However, drilling and riveting robots are large and unsuitable for working in confined spaces, so manual drilling and riveting are often required. Manual drilling and riveting, however, requires considering multiple information points and coordinating with the robot, resulting in low efficiency. Summary of the Invention

[0003] To solve the above problems, an embodiment of the present invention provides a virtual-reality fusion process guidance method for human-machine collaborative drilling and riveting, which is applied to an augmented reality device, wherein the augmented reality device is respectively connected to a robot and a force sensing rivet gun through a host computer. The method includes: displaying a registration image of a component to be drilled and riveted, so as to perform virtual-reality registration based on the registration image; after determining a target hole position, displaying drilling and riveting information of the target hole position; the drilling and riveting information includes a virtual hole position; obtaining status information of the robot, and determining whether the drilling and riveting conditions of the target hole position are met based on the status information; if the drilling and riveting conditions of the target hole position are met, obtaining and displaying pressure information of the force sensing rivet gun; the pressure information is the abutment pressure between the force sensing rivet gun and the position of the target hole position; if the pressure information meets the drilling and riveting conditions of the target hole position, controlling the robot to start the drilling and riveting operation.

[0004] Optionally, the method further includes: acquiring and storing pressure information of the force sensing rivet gun during each drilling and riveting process of the target hole; each drilling and riveting process includes at least one of the following: a drilling and riveting preparation stage, a drilling and riveting process stage, and a drilling and riveting completion stage.

[0005] Optionally, the drilling and riveting information further includes process information; the process information includes at least one of the following: hole diameter, riveting method, rivet type, and strength.

[0006] Optionally, the method further includes: displaying a virtual operation panel that moves with the field of view, the virtual operation panel including at least one of the following function switches: a process information display switch, a registration tracking switch, a robot control function button, and a next hole position selection function button.

[0007] Optionally, the method further includes: receiving a hole position selection instruction input by a user, and determining the target hole position according to the hole position selection instruction.

[0008] Optionally, before acquiring the status information of the robot, the method further includes: controlling the robot to move to the target hole position, and executing a drilling and riveting preparation action corresponding to the target hole position.

[0009] Optionally, the method further includes: when the component to be drilled and riveted is modeled in advance, performing registration and tracking according to a model target method; when positioning marks are arranged in advance on the component to be drilled and riveted, performing registration and tracking according to an image target method.

[0010] Optionally, the status information includes at least one of the following: presser foot status, tool feeding status, and drilling and riveting unit nail feeding status.

[0011] An embodiment of the present invention provides a virtual-reality fusion process guidance system for human-machine collaborative drilling and riveting, which is used to execute the above-mentioned virtual-reality fusion process guidance method for human-machine collaborative drilling and riveting guidance.

[0012] In the embodiment of the present invention, a virtual-reality fusion display method is adopted to guide human-machine collaborative drilling and riveting, including the integration of augmented reality information display, force display, robot control and other functions. The augmented reality information display can save workers the time and energy of checking and memorizing various types of information. The force display is conducive to accurate force control. Controlling the robot through the augmented reality device can reduce the worker's movement and other operations, thereby improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0014] Figure 1 Schematic diagram of a scenario of human-machine collaborative drilling and riveting in an embodiment of the present invention;

[0015] Figure 2 Schematic diagram of a flow chart of a virtual-reality fusion process guidance method for human-machine collaborative drilling and riveting in an embodiment of the present invention;

[0016] Figure 3 Schematic diagram of virtual hole positions and prompt information displayed by an AR device in an embodiment of the present invention

[0017] Figure 4 This is a schematic diagram of an operation panel in an embodiment of the present invention;

[0018] Figure 5 Schematic diagram of a single hole guiding process in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] In the existing technology, drilling and riveting are mostly done by robots or manually. The human-machine collaborative drilling and riveting method can adapt to more scenarios. For example, in a scenario where one side is open and the other side is narrow, drilling and riveting are done by a drilling and riveting robot on the open side, and drilling and riveting are done manually on the narrow side where a drilling and riveting robot cannot be arranged.

[0021] An embodiment of the present invention provides a virtual-reality fusion process guidance method and system. In the scenario of human-machine collaborative drilling and riveting, the driller wears an augmented reality (AR) device, such as an AR helmet, AR glasses, etc., and can see the position, aperture, drilling and riveting method and other information of the holes to be drilled and riveted in the field of view; in addition, the force sensor on the force sensing rivet gun can measure the contact force between the force sensing rivet gun and the part to be drilled and riveted in real time, and display the magnitude of the force in real time in the virtual field of view, so that the worker can ensure that the drilling and riveting force is maintained within the desired range according to his own output force; and the system provides guidance operation according to the needs of the worker.

[0022] See also Figure 1 The schematic diagram of the human-machine collaborative drilling and riveting scenario provided by an embodiment of the present invention is shown, taking an aircraft wall panel as an example, showing a drilling and riveting robot 11, a drilling and riveting worker 12, an aircraft wall panel 13, a force sensing rivet gun 14 and an AR helmet 15.

[0023] For example, in a real-world drilling and riveting scenario, a driller 12 stands in front of an aircraft panel 13, while a drilling and riveting robot 11 is positioned behind it. A host computer runs a service program to facilitate information exchange between the AR helmet and the drilling and riveting robot. The driller wears an AR helmet, which communicates with the host computer via a socket within the same local area network. The drilling and riveting robot communicates with the host computer via 2.4G. Furthermore, the driller uses a handheld force-sensing rivet gun to collaborate with the robot during drilling and riveting. The force-sensing rivet gun is equipped with a force sensor, and the host computer can transmit pressure information from the force sensor to the AR helmet.

[0024] See also Figure 2 The figure shows a flow chart of a virtual-reality fusion process guidance method for human-machine collaborative drilling and riveting. The method is applied to the above-mentioned AR device, which is connected to the robot and force sensing riveting gun through the host computer. The method includes the following steps:

[0025] S202: Display the registration image of the component to be drilled and riveted, and perform virtual-real registration based on the registration image.

[0026] Two optional registration methods are available: If the component to be drilled and riveted is pre-modeled, registration and tracking can be performed using a model target approach; if positioning markers are pre-placed on the component to be drilled and riveted, registration and tracking can be performed using an image target approach. Both methods rely on calculating and tracking feature points of the object for matching.

[0027] Based on any of the above registration methods, the registration image of the part to be drilled and riveted is displayed in the field of view of the AR device. The driller and riveter can visually compare the registration image with the actual part to be drilled and riveted, and overlap the registration image with the part to be drilled and riveted, thereby performing virtual-real registration.

[0028] S204: After the target hole position is determined, the drilling and riveting information of the target hole position is displayed.

[0029] This drilling and riveting information can include virtual hole locations. After virtual and real registration, the drilling and riveting process can begin. The driller can select any hole location as the next target hole for drilling and riveting, or the guidance system can automatically select a hole location as the next target hole for drilling and riveting. The guidance system runs on the aforementioned AR device.

[0030] Optionally, a hole position selection instruction input by a user is received, and a target hole position is determined based on the hole position selection instruction; or the target hole position is determined based on a shortest path planning algorithm and the hole position of the last drilled and riveted hole. The hole position selection instruction can be input by the driller and riveter through an AR device.

[0031] S206, obtaining status information of the robot, and determining whether drilling and riveting conditions of the target hole position are met according to the status information.

[0032] After determining the target hole location, the robot can be controlled to move to the target hole location and perform the corresponding drilling and riveting preparation actions. The drilling and riveting preparation actions include placing the robot's pressure foot on the part to be drilled and riveted, controlling the tool feed, and inserting the nail.

[0033] After the robot moves to the target hole location, it can be controlled to continue drilling and riveting preparation. By obtaining the robot's status information, it can be determined whether the robot is ready to perform the drilling and riveting process. Optionally, this status information may include: presser foot status, tool feed status, drilling and riveting unit nail feeding status, etc.

[0034] S208: If the drilling and riveting conditions of the target hole are met, the pressure information of the force sensing riveting gun is acquired and displayed.

[0035] If the drilling and riveting conditions for the target hole are not met, the operator continues to wait until they are met. Once the drilling and riveting conditions are met, the operator can use the AR device to obtain this information and then press the force-sensing rivet gun against the target hole. By acquiring pressure information from the force-sensing rivet gun, it can be determined whether the force-sensing rivet gun has abutted the component to be drilled and riveted and whether the appropriate pressure is being applied. This pressure information represents the contact pressure between the force-sensing rivet gun and the target hole.

[0036] It should be noted that the pressure information of the force-sensing rivet gun can be acquired and displayed in real time, thereby indicating the magnitude of the pressing force being applied by the driller and riveter, and facilitating precise control of the force. In the prior art, riveting does not have accurate force control, and theoretically, force affects riveting quality. Therefore, this embodiment proposes a force-sensing rivet gun with a force measurement function, which increases the collection of force information during the drilling and riveting process, not only enabling precise force control, but also facilitating related experiments and tests.

[0037] For the purposes of the above experiments and tests, the following steps may also be included: acquiring and storing pressure information from the force-sensing riveting gun during each drilling and riveting process at the target hole. The drilling and riveting processes may include: the drilling and riveting preparatory stage, the drilling and riveting process, and the drilling and riveting completion stage. The force at each stage can be measured and stored separately for further study.

[0038] S210: If the pressure information meets the drilling and riveting conditions of the target hole position, the robot is controlled to start the drilling and riveting operation.

[0039] The driller and riveter can use the AR device to control the robot to start the drilling and riveting operation. In this embodiment, the driller and riveter can control the robot's behavior through the AR device. Based on the wireless connection between the AR device and the robot, the driller and riveter can complete the interaction with the robot without moving.

[0040] The embodiment of the present invention provides a virtual-reality fusion display method for guiding human-machine collaborative drilling and riveting, which guides human-machine collaborative drilling and riveting by adopting a virtual-reality fusion display method, including the integration of AR information display, force display, robot control and other functions. AR information display can save workers the time and energy of checking and memorizing various types of information, force display is conducive to accurate force control, and controlling the robot through AR equipment can reduce the worker's movement and other operations, thereby improving work efficiency.

[0041] Furthermore, the drilling and riveting information also includes process information, which may include: hole diameter, riveting method, rivet type, force, etc. Figure 3The diagram of virtual hole positions and prompt information displayed by the AR device shows the registration mark on the upper side, multiple hole positions (numbered 1-8) on the lower side, and the process information of hole position number 5 as follows: "Hole diameter: 10mm, Riveting method: Press riveting, Rivet type: Flat head rivet, Force: 80N". Figure 3 The pressure information is also shown in the figure. Taking the pressure display bar as an example, the length of the shadow of the display bar indicates the current pressure applied. Figure 3 The maximum displayed bar corresponds to 200N and the currently applied pressure corresponds to about 70N.

[0042] The following embodiment uses the work scenario of using an AR helmet to drill and rivet aircraft wall panels as an example to illustrate.

[0043] For example, the AR headset uses Microsoft's HoloLens 2. All displayed AR information is developed using Unity and then published to the headset for execution. Virtual-reality registration is achieved by importing the Vuforia SDK. There are two registration methods: one is to model the aircraft panel and then use the model target method for registration and tracking; the other is to attach markers to the panel and then use the image target method for registration and tracking. Both methods rely on calculating the feature points of the tracked object for matching.

[0044] When a worker begins work, the AR helmet first registers with the aircraft panel and then displays an operation panel that follows the field of view. Based on this, the above method may further include the following steps: displaying a virtual operation panel that follows the field of view. The virtual operation panel may include the following function switches: a switch for displaying process information, a switch for registration tracking, a button for robot control, and a button for selecting the next hole position.

[0045] See also Figure 4 The schematic diagram of the operation panel shown in the figure shows a switch for displaying process information, a help switch, a registration tracking switch, robot control function buttons (Start, Finish), and a button for selecting the next hole position. This operation panel allows users to check the registration effect, select the next drilling position, and choose whether to display process information. When the guidance system starts working, workers can manually select the hole position for drilling and riveting. After determining the drilling and riveting position, the guidance system will guide the worker through the drilling and riveting operations according to the preset guidance process.

[0046] Among them, the guidance system integrates all the information workers need during the drilling and riveting process, including drilling and riveting position, force, robot status information, etc. The guidance system can communicate with the robot and drilling and riveting unit through a preset communication protocol (such as OPC UA), and obtain status information such as the robot's presser foot status, tool feed status, and drilling and riveting unit nail feeding status. Combined with this information, the system prompts workers in the AR helmet with processing parameters such as drilling and riveting position and force. The workers then perform operations based on the force sensor readings (displayed in the AR helmet) and the AR image of the hole position. This human-computer interaction algorithm and virtual-reality fusion display method greatly facilitates the work of drillers and riveters.

[0047] See also Figure 5 The schematic diagram of the single-hole guidance process shown in FIG. 4 may include the following steps:

[0048] S501, determining the serial number of the hole to be operated and the AR display hole position.

[0049] S502, collecting the robot's presser foot status and tool feed status. In this embodiment, the drilling and riveting process is taken as an example for description, including drilling and riveting steps.

[0050] S503: Determine whether the drilling start condition is met. If yes, execute S504; if not, wait and then continue to execute S502.

[0051] S504, the worker holds the force sensing rivet gun to tighten.

[0052] S505: The force sensor readings of the force-sensing rivet gun are collected and displayed. During the worker's tightening and subsequent riveting process, the pressure between the force-sensing rivet gun and the aircraft panel is collected in real time and displayed on the AR headset. This pressure data can be stored in the AR headset or on a host computer for use in experiments or testing.

[0053] S506, confirming that the drilling is completed.

[0054] S507, collecting the robot's presser foot status and the drilling and riveting unit's nail feeding status.

[0055] S508: Determine whether the riveting start condition is met. If yes, execute S509; if not, wait and then continue to execute S507.

[0056] S509, workers hold force-sensing riveting guns to rivet.

[0057] S510, confirming that the riveting is completed.

[0058] S511, user-defined / algorithm specifies the next hole position number.

[0059] To allow workers to monitor their riveting force while performing the work, this embodiment incorporates a removable mechanical structure into the original riveting gun. This structure houses a ring-shaped force sensor, which allows users to select between measurement and riveting modes by adjusting the clamping position of the clip. The force sensor transmits the collected information to a host computer in real time, where it is displayed on an AR headset. This drilling and riveting force data allows workers to adjust the force in real time and can also be used to collect data during the process for other research purposes.

[0060] Compared with the existing technology, the AR display method used in this embodiment can save workers the time of looking up and memorizing the drilling and riveting sequence, force, etc., and at the same time can reduce workers' learning time and improve work efficiency; in the existing technology, riveting does not have accurate force control, and theoretically speaking, force has an impact on riveting quality, so a new force measurement rivet gun is designed to increase the collection of force information, which can facilitate related experiments and tests; the control panel of the AR helmet can control the behavior of the robot, which makes the working method much simpler, and workers can complete the interaction with the robot on the spot.

[0061] An embodiment of the present invention further provides a virtual-reality fusion process guidance system for human-machine collaborative drilling and riveting, which is used to execute the above-mentioned virtual-reality fusion process guidance method for human-machine collaborative drilling and riveting.

[0062] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing a control device through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the above-mentioned method embodiments, wherein the storage medium may be a memory, a disk, an optical disk, etc.

[0063] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0064] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0065] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A virtual-reality fusion process guidance method for human-machine collaborative drilling and riveting, characterized in that: Applied to an augmented reality device, the augmented reality device is connected to a robot and a force sensing riveting gun respectively via a host computer, and the method includes: Displaying a registration image of the component to be drilled and riveted, and performing virtual-real registration based on the registration image; After determining the target hole position, the drilling and riveting information of the target hole position is displayed; the drilling and riveting information includes a virtual hole position; Acquiring status information of the robot, and determining whether drilling and riveting conditions of the target hole position are met according to the status information; If the drilling and riveting conditions of the target hole are met, pressure information of the force sensing riveting gun is acquired and displayed; the pressure information is the abutting pressure between the force sensing riveting gun and the position of the target hole; If the pressure information satisfies the drilling and riveting conditions of the target hole position, controlling the robot to start the drilling and riveting operation; The drilling and riveting information also includes process information; the process information is force; The method further comprises: In the case where the component to be drilled and riveted is modeled in advance, registration and tracking are performed according to the model target; In the case where positioning marks are arranged in advance on the components to be drilled and riveted, registration and tracking are performed in accordance with the image target.

2. The method according to claim 1, characterized in that The method further comprises: Obtain and store pressure information of the force sensing riveting gun during each drilling and riveting process of the target hole; each drilling and riveting process includes at least one of the following: a drilling and riveting preparation stage, a drilling and riveting process stage, and a drilling and riveting completion stage.

3. The method according to claim 1, characterized in that The process information also includes: hole diameter, riveting method, and rivet type.

4. The method according to claim 3, characterized in that The method further comprises: A virtual operation panel is displayed that moves along with the field of view, and the virtual operation panel includes at least one of the following function switches: a process information display switch, a registration tracking switch, a robot control function button, and a next hole position selection function button.

5. The method according to claim 1, wherein The method further comprises: Receive a hole position selection instruction input by a user, and determine the target hole position according to the hole position selection instruction.

6. The method according to claim 1, characterized in that Before obtaining the status information of the robot, the method further includes: The robot is controlled to move to the target hole position and perform drilling and riveting preparation actions corresponding to the target hole position.

7. The method according to claim 1, characterized in that The status information includes at least one of the following: presser foot status, tool feeding status, and drilling and riveting unit nail feeding status.

8. A virtual-real fusion process guidance system for human-machine collaborative drilling and riveting, characterized by: Used to execute the virtual-reality fusion process guidance method for human-machine collaborative drilling and riveting as described in any one of claims 1-7.

Images