Action feedback glove for spraying robot teaching and spraying robot intelligent teaching system and method

Abstract

The invention discloses an action feedback glove for spraying robot teaching and an intelligent spraying robot teaching system and method.The action feedback glove can collect finger pose data and holding pressure data during hand action and conduct vibration and spraying pressure feedback; the teaching system comprises a reality spraying system, a head display device, an action feedback glove, a gesture camera and an integrated control module. According to the method, the integrated control module obtains action control instructions of a virtual robot and a real robot through solving according to finger pose data and a hand action image and obtains pressure control instructions of a virtual spray gun and a real spray gun through solving, and the integrated control module generates a virtual spraying robot, a virtual spray gun and a virtual workpiece to be sprayed; and the integrated control module is used for controlling the virtual spraying robot and the spraying gun to work according to the control instruction to form a virtual spraying scene, transmitting the virtual spraying scene to the head display equipment for displaying, and controlling the real spraying robot and the spraying gun to work according to the corresponding control instruction to form a real spraying scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot teaching system, and particularly to a spraying robot teaching motion feedback glove, a spraying robot intelligent teaching system and method. BACKGROUND

[0002] With the rapid development of robot technology, its application value in toxic and harmful environment is increasingly prominent. Taking spraying operation as an example, volatile organic compounds in the traditional manual spraying process can seriously damage the health of the operator, and even induce cancer. At the same time, manual spraying also has problems such as insufficient quality stability and low spraying efficiency of large-size workpieces. Using teleoperation technology to teach the spraying robot can not only avoid personnel exposure to toxic environment, but also improve the uniformity and controllability of spraying quality. This technology provides an effective path to solve the problem of high-risk scene operation, and has significant application prospect.

[0003] At present, the teleoperation of the spraying robot is mostly online teaching with a handheld teaching device, which requires the spraying personnel to define the motion trajectory through the teaching panel, and the spraying efficiency of complex parts is low. At the same time, there is a lack of real-time visual or tactile spraying feedback in the operation process, and it is difficult to change the pressure of the spray gun and the nozzle diameter in real time during spraying teaching to adjust the atomization effect of the paint. Therefore, it is urgent to develop an intelligent spraying teaching system for the spraying robot to improve the safety of spraying operation and optimize the operation efficiency.

[0004] Chinese patent document No. "CN110238831A" proposes a robot teaching system based on RGB-D image and teaching device, which realizes offline teaching by selecting path points in the RGB image, but the separation of trajectory and attitude cannot provide a natural teaching process, and the work efficiency is low. Chinese patent document No. "CN108214445A" proposes a master-slave heterogeneous teleoperation control system based on ROS, which improves the scalability and universality based on ROS communication, but relies on handle control, and the gesture interaction mode is single, and the operation efficiency of the system is limited. Chinese patent document No. "CN108908298A" proposes a master-slave spraying robot teaching system integrating virtual reality technology, which uses a handheld stylus to teach and can optimize the trajectory fusion, but lacks force feedback to perceive factors such as paint viscosity changes, and the teaching accuracy is limited. Chinese patent document No. "CN113352300A" proposes a spraying robot teaching device, which obtains the pose of the spray gun in the spraying process by fusing IMU and UWB data, and realizes programming-free teaching, but the spraying personnel still need to be in the spraying site, and there is a drift error, and the spraying precision is low.

[0005] Summarizing the existing solutions, the spraying teleoperation solution lacks intelligent spraying with virtual-real fusion, and the demonstration process cannot verify the spraying effect in real time to simultaneously meet the natural spraying on site and the remote safe demonstration; the gesture recognition module and the force feedback module are often used separately as two modules, lacking multi-modal collaboration, and the gesture recognition demonstration cannot timely perceive the spraying pressure and vibration to verify the paint atomization effect, clogging, and other important factors of spraying and dynamically adjust the spraying effect; meanwhile, the human-computer interaction mode of the spraying demonstration is single, and the robustness is low.

[0006] Therefore, it is urgent to develop a spraying robot intelligent demonstration system that integrates high-precision hand motion tracking and multi-dimensional presence feedback. Based on visual-haptic collaboration, the system can map the paint film thickness distribution in real time through the interactive interface, realizing intuitive visual feedback; through the force feedback gloves, the system can reproduce the gun back pressure and the paint rheological properties, providing realistic haptic feedback. The system further combines hand motion capture technology to achieve dexterous control of the spraying robot, thereby completing the spraying task and realizing remote precise reproduction of the spraying process. SUMMARY

[0007] The present application provides a spraying robot demonstration motion feedback glove, a spraying robot intelligent demonstration system and method, to solve the problems of existing spraying robot demonstration methods, such as the need for on-site demonstration, the threat of harmful gases to the health of spraying personnel, the lack of intuitive paint state feedback such as pressure and vibration in the spraying demonstration process, the insufficient stability and low accuracy of the spraying demonstration, and the dependence on a single mode with low robustness, thereby improving the efficiency of spraying demonstration.

[0008] To achieve the above-mentioned purposes, the technical scheme adopted by the present application is as follows:

[0009] The spraying robot demonstration motion feedback glove comprises a glove body (701) worn on the human hand, five groups of motor boxes (706) are installed on the glove body (701) corresponding to the back of the human palm, an IMU module (705) is installed on the glove body (701) corresponding to the position of each finger, a fingertip support (703) is fixed on the glove body (701) corresponding to the position of each fingertip, a pressure sensor (707) is installed on the glove body (701) corresponding to the position of the palm surface and the finger knuckle surface, and a vibration motor (708) is installed on the glove body (701) corresponding to the position of each finger root of the human hand.

[0010] Each motor box (706) is provided with a motor (709), a wire slot (712), a rotating shaft (714), and a gear set. The wire slot (712) is wound with a nylon wire (704), and the wire slot (712) is rotatably installed in the motor box (706) through the rotating shaft (714). The output shaft of the motor (709) is in transmission connection with the rotating shaft (714) through the gear set, and the output shaft of the motor (709) is provided with an encoder;

[0011] The nylon wire (704) on each wire slot (712) is respectively fixedly connected with a fingertip support (703) of each fingertip of the fingers in one-to-one correspondence after being drawn out from the corresponding motor box.

[0012] The IMU module (705) collects the pose data of the corresponding fingers when the human hand moves, and the pressure sensor (707) collects the gripping pressure data when the fingers of the human hand perform the gripping action. When the fingers of the human hand perform the gripping action, the nylon wire (704) is pulled by the fingertip support (703) to rotate the wire slot (712), and then the rotating shaft (714) and the gear set rotate the output shaft of the motor (709). The encoder collects the rotation amount of the output shaft of the motor (709) when the output shaft rotates, and the motor (709) applies a torque to the rotating shaft (714) in the opposite direction of the rotation direction of the wire slot (712), thereby generating resistance to the gripping fingers.

[0013] Further, a spring holder (715) is installed in each motor box, the spring holder (715) is provided with a spring (713), the spring (713) is wound outside the rotating shaft (714), and one end of the spring (713) is fixedly connected to the rotating shaft (714) and the other end is fixed to the spring holder (715).

[0014] A spraying robot intelligent teaching system, comprising:

[0015] A real spraying system, comprising a real spraying robot (14), a real spray gun (13), and a real workpiece to be sprayed (11), for forming a real spraying scene;

[0016] A head-mounted device (8) worn on the head of a spraying person, for displaying a virtual spraying robot (5), a virtual spray gun (4), and a virtual workpiece to be sprayed (2) to the spraying person, the virtual spraying robot (5), the virtual spray gun (4), and the virtual workpiece to be sprayed (2) being used to form a virtual spraying scene;

[0017] The action feedback glove (7) is worn on the hand of the spraying personnel; the IMU module (705) in the action feedback glove (7) collects the pose data of the fingers when the human hand moves; the encoder in the action feedback glove (7) collects the rotation amount of the motor (709) output shaft when the fingers of the human hand perform the pressing action; and the pressure sensor (707) in the action feedback glove (7) collects the pressing force data of the fingers of the human hand when the fingers perform the pressing action;

[0018] The gesture camera (9) is used to collect the hand action image of the human hand when the human hand moves;

[0019] The integrated control module (6) obtains the pose data, the rotation amount, the pressing force data obtained by the action feedback glove (7), and the hand action image obtained by the gesture camera (9);

[0020] The integrated control module (6) obtains the action trajectory of the hand of the spraying personnel according to the fusion information of the finger pose data and the hand action image, generates a virtual robot action control instruction for controlling the virtual spraying robot (5), and generates a real robot action control instruction for controlling the real spraying robot (11);

[0021] The integrated control module (6) calculates the finger flexion degree data according to the rotation amount data when the hand performs the pressing action and the size of the hand of the spraying personnel, and obtains a virtual gun pressure control instruction for controlling the spraying pressure of the virtual gun (4) and a real gun pressure control instruction for controlling the spraying pressure of the real gun (13) according to the pressing force data and the finger flexion degree data;

[0022] The integrated control module (6) generates the virtual spraying robot (5), the virtual gun (4), and the virtual workpiece (2) to be sprayed, controls the virtual spraying robot (5) to perform corresponding actions according to the virtual robot action control instruction, and controls the virtual gun (4) to perform corresponding spraying operations according to the virtual gun pressure control instruction, thereby forming a virtual spraying scene, and transmitting the virtual spraying scene to the head-mounted device (8) for display;

[0023] The integrated control module (6) transmits the generated real robot action control instruction to the real spraying robot (14), and transmits the real gun pressure control instruction to the real gun (13); the real spraying robot (14) performs corresponding actions according to the real robot action control instruction, and the real gun (13) performs corresponding spraying operations according to the real gun pressure control instruction, thereby forming a real spraying scene.

[0024] Further, the real spraying system further comprises a real rotary table (10) for placing a real workpiece to be sprayed (11), and the real rotary table (10) rotates at a set rotating speed, and the real spraying scene is formed by the moving real rotary table (10), the moving real spraying robot (14), the real spraying gun (13) performing the spraying operation, and the real workpiece to be sprayed (11).

[0025] The integrated control module (6) further generates a virtual rotary table (10) for placing a virtual workpiece to be sprayed (2), controls the virtual rotary table (10) to rotate at a set rotating speed by the integrated control module (6), and forms a virtual spraying scene by the moving virtual rotary table (10), the moving virtual spraying robot (5), the virtual spraying gun (4) performing the spraying operation, and the virtual workpiece to be sprayed (2).

[0026] Further, the gesture camera (9) is integrated in the head-mounted device (8).

[0027] Further, the head-mounted device (8) further integrates an environment camera (801) for collecting real operation environment images, and the integrated control module (6) acquires the real operation environment images collected by the environment camera (801) and detects obstacles from the real operation environment images by the integrated control module (6).

[0028] Further, a real scene monitoring camera is further included, which collects real spraying scene images and transmits them to the integrated control module (6), and the integrated control module (6) transmits the real spraying scene images to the head-mounted device (8) for display.

[0029] Further, a vibration sensor is installed on the real spraying gun (13), which collects vibration signals generated when the real spraying gun (13) performs the spraying operation, and the vibration sensor transmits the vibration signal data to the integrated control module (6).

[0030] The integrated control module (6) controls the vibration motor (708) on the action feedback glove (7) to generate corresponding vibration according to the vibration signal data collected by the vibration sensor.

[0031] Further, a spraying pressure sensor is installed in the real spraying gun (13), which collects spraying pressure signal data inside the real spraying gun (13) when the real spraying gun (13) performs the spraying operation, and the spraying pressure sensor transmits the spraying pressure signal data to the integrated control module (6).

[0032] The integrated control module (6) controls the motor (709) in each motor box (706) of the action feedback glove (7) to generate a corresponding reverse torque according to the spraying pressure signal data collected by the spraying pressure sensor.

[0033] A teaching method based on the above spraying robot intelligent teaching system, the process is as follows:

[0034] The spraying personnel wear the action feedback glove (7) and the head-mounted device (8), and complete the gripping action of the hand, and the hand encoder data of fully opening and gripping is collected through the action feedback glove 7; the hand motion solving module in the data processing module of the integrated control module 6 adopts a template matching method to process the encoder data, and obtains the size of the spraying personnel's hand;

[0035] The integrated control module (6) obtains the pose data, gripping pressure data, and rotation amount data generated in the action feedback glove (7) when the spraying personnel's hand moves and grips, and the hand action image collected by the gesture camera (9);

[0036] The integrated control module (6) obtains the action trajectory of the spraying personnel's hand according to the finger pose data and the hand action image, and generates virtual robot action control instructions for controlling the virtual spraying robot (5) and real robot action control instructions for controlling the real spraying robot (11) according to the action trajectory;

[0037] The integrated control module (6) obtains the hand flexion and extension degree data of the spraying personnel according to the rotation amount data combined with the size of the hand, and obtains the virtual spray gun pressure control instructions for controlling the spraying pressure of the virtual spray gun (4) and the real spray gun pressure control instructions for controlling the spraying pressure of the real spray gun (13) combined with the finger flexion and extension degree data and the spraying pressure data collected by the pressure sensor (707);

[0038] Then, the integrated control module (6) generates the virtual spraying robot (5), the virtual spray gun (4), and the virtual workpiece to be sprayed (2), controls the virtual spraying robot (5) to perform corresponding actions according to the virtual robot action control instructions, and controls the virtual spray gun (4) to perform corresponding spraying operations according to the virtual spray gun pressure control instructions, thereby forming a virtual spraying scene, and transmitting the virtual spraying scene to the head-mounted device (8) for display;

[0039] The integrated control module (6) transmits the generated real robot action control instruction to the real spraying robot (14), and transmits the real spraying gun pressure control instruction to the real spraying gun (13); the real spraying robot (14) performs corresponding actions according to the real robot action control instruction, and the real spraying gun (13) performs spraying work of corresponding spraying pressure according to the real spraying gun pressure control instruction, so as to form a real spraying scene;

[0040] When the real spraying scene is formed, the integrated control module (6) acquires vibration signal data generated by a vibration sensor on the real spraying gun (13), and controls a vibration motor (708) on the action feedback glove (7) to generate corresponding vibration according to the vibration signal data, so as to feed back the vibration feeling of the real spraying gun (13) to the spraying personnel; the integrated control module (6) acquires spraying pressure signal data generated by a pressure sensor on the real spraying gun (13), and controls a motor (709) on the action feedback glove (7) to generate corresponding resistance according to the spraying pressure signal data, so as to feed back the spraying pressure inside the real spraying gun (13) to the spraying personnel.

[0041] The present application builds a virtual spraying scene, so that the spraying personnel does not need to be exposed to the real spraying environment which is toxic; the head-mounted device integrated with a gesture camera and an environment camera provides a virtual spraying scene for the spraying personnel, the gesture camera captures the hand movement of the spraying personnel, so as to realize the synchronous control of the real spraying robot and the virtual spraying robot, and the environment camera detects the surrounding environment to avoid obstacles in the real world during spraying; in addition, the spraying personnel can obtain real-time tactile feedback by wearing the action feedback glove, and intuitively perceive the interaction state between the spraying robot and the environment during spraying demonstration.

[0042] The present application realizes the fusion of gesture recognition technology, action feedback and virtual reality technology, so that the spraying personnel can accurately control the spraying trajectory and spraying pressure of the spraying robot in a safe environment to perform spraying work, guarantee the safety of the spraying personnel and the accuracy of the spraying demonstration, and improve the efficiency of the spraying work.

[0043] (1) The present application embeds the visual sensing device for environment and gesture recognition in the head-mounted device, avoids external devices such as base stations, guarantees the safety of the spraying personnel, and improves the convenience and flexibility of spraying;

[0044] (2) The IMU assists the visual tracking to realize instantaneous frame dropping, the vision corrects the long-term drift accumulation of the IMU, and the fusion of the multi-modal perception mode enables the system to adapt to low light conditions appearing for a short time, and improves the stability and accuracy of the spraying work.

[0045] (3) For the spraying scene, the pressure sensor in the action feedback glove collects the pressure applied by the spraying personnel to the virtual spray gun trigger, so that the spraying personnel can dynamically and accurately adjust the spraying force, making the teleoperation teaching process more intuitive and natural; at the same time, the motor torque in the action feedback glove provides the spray pressure signal of the spray gun, and the vibration feedback provides the vibration signal of the spray gun, so that the spraying personnel can obtain a more realistic teaching feeling through the spray pressure signal and the vibration signal, and when the nozzle is blocked or the pressure is too low to cause the paint to flow, the spraying personnel can feel and maintain in time, making the teaching process more stable and improving the efficiency of the spraying work.

[0046] (4) The application also has high scalability, can enrich and expand the static gesture library, switch virtual scenes and spraying piece models of different shapes according to different work sites, and can be widely used in industrial spraying, automobile manufacturing, aviation coating and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 a is a schematic diagram of the system of the embodiment of the application.

[0048] Figure 1 b is a schematic diagram of the real spraying scene in the system of the embodiment of the application.

[0049] Figure 2 is a teaching principle diagram of the system of the embodiment of the application.

[0050] Figure 3 is a principle diagram of the head-mounted device of the embodiment of the application.

[0051] Figure 4 is a structure diagram of the action feedback glove of the embodiment of the application.

[0052] Figure 5 is an internal structure diagram of the motor box in the action feedback glove of the embodiment of the application.

[0053] Figure 6 is a principle block diagram of the system of the embodiment of the application.

[0054] Figure 7 is a principle diagram of the integrated control module of the embodiment of the application.

[0055] Figure 8 is a teaching method flow chart of the embodiment of the application. DETAILED DESCRIPTION

[0056] In order to make the personnel in the technical field better understand the technical scheme of the present application, the embodiments of the present application will be described in detail below in combination with the drawings and examples, so that the process of how the present application applies technical means to solve technical problems and achieve corresponding technical effects can be fully understood and implemented. The embodiments of the present application and various features in the examples can be combined with each other without conflict, and the formed technical solutions are all within the protection scope of the present application.

[0057] Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

[0058] It should be noted that the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusion.

[0059] As shown in Figure 1 a、 Figure 2 The present embodiment discloses a spraying robot intelligent teaching system integrating hand tracking teleoperation and haptic feedback, which comprises a real spraying system, a head-mounted device 8, a motion feedback glove 7, a gesture camera 9, an environment camera 801, a real scene monitoring camera, and an integrated control module 6.

[0060] In the present embodiment, the real spraying system comprises a real rotating table 10, a real spraying robot 14, a real spraying gun 13, and a real workpiece to be sprayed 11. The real workpiece to be sprayed 11 is placed on the real rotating table 10, and the real rotating table 10 rotates at a set speed, so that the real workpiece to be sprayed 11 rotates with it. The real spraying robot 14 is arranged outside one side of the real rotating table 10, and the real spraying gun 13 is installed at the execution end of the real spraying robot 14. When the real spraying robot 14 drives the real spraying gun 13 to move towards the real workpiece to be sprayed 11 rotating with the real rotating table 10, and the real spraying gun 13 performs a spraying operation on the real workpiece to be sprayed 11, a real spraying scene 12 is formed. A vibration sensor and a spraying pressure sensor are installed on the real spraying gun 13. When the real spraying gun 13 performs a spraying operation, the vibration sensor collects vibration signal data generated when the real spraying gun 13 performs a spraying operation, and the spraying pressure sensor collects spraying pressure signal data inside the real spraying gun 13 when the real spraying gun 13 performs a spraying operation.

[0061] In the present embodiment, the real scene monitoring camera is arranged around the real spraying system, and is used to collect images of the above-mentioned real spraying scene 12.

[0062] As shown in Figure 3As shown, in the embodiment, the head-mounted device 8 is a virtual reality device (VR device), which includes a display 803 and a fixed headband 802 connected to the display 803. The head-mounted device 8 is worn on the head of the spraying personnel through the fixed headband 802, and the display surface of the display 803 faces the eyes of the spraying personnel, so that the spraying personnel can watch the picture displayed by the display 803.

[0063] As shown, in the embodiment, the action feedback glove 7 includes a glove body 701, and a fingertip support 703 is fixed at the position of the fingertip of each finger of the human body, respectively, and an IMU module 705 is mounted on each fingertip support 703, respectively, so as to collect the pose data of the corresponding finger when the spraying personnel moves the hand. A pressure sensor 707 is mounted at the position of the palm surface of the human body and the position of the finger joint of each finger of the human body, respectively, so as to collect the pressure data of the finger when the spraying personnel performs the pressing action. Figure 2 Figure 4 As shown, in the embodiment, the action feedback glove 7 includes a glove body 701, and a fingertip support 703 is fixed at the position of the fingertip of each finger of the human body, respectively, and an IMU module 705 is mounted on each fingertip support 703, respectively, so as to collect the pose data of the corresponding finger when the spraying personnel moves the hand. A pressure sensor 707 is mounted at the position of the palm surface of the human body and the position of the finger joint of each finger of the human body, respectively, so as to collect the pressure data of the finger when the spraying personnel performs the pressing action.

[0064] The glove body 701 is fixed with a finger support 702 at the back of each finger of the human body, and five motor boxes 706 are installed at the position of the back of the palm of the human body. As shown, Figure 5 each motor box 706 is provided with a motor 709, a first helical gear 710, a second helical gear 711, a wire slot 712, a spring 713, a rotating shaft 714, and a spring holder 715. In each motor box 706, the motor 709 is fixedly installed in the motor box 706, and the output shaft of the motor 709 is provided with an encoder; the first helical gear 710 is coaxially fixed to the output shaft of the motor 709, the second helical gear 711 is rotatably installed in the motor box 706 through the rotating shaft 714, and the second helical gear 711 is in transmission engagement with the first helical gear 710; the wire slot 712 is coaxially fixedly installed on the rotating shaft 714, and the nylon wire 704 is wound on the wire slot 712; the spring holder 715 is fixed in the motor box 706, and the spring 713 is arranged on the spring holder 715 and wound on the rotating shaft 714, and one end of the spring 713 is fixedly connected to the rotating shaft 714 and the other end is fixedly connected to the spring holder 715.

[0065] The nylon wire 704 on the wire slot 712 in each motor box 706 respectively passes out of the motor box 706. The five nylon wires 704 passing out of the five motor boxes 706 are fixedly connected to the fingertip supports 703 of the fingertips of the five fingers one by one, and each nylon wire 704 respectively passes through the finger support 702 at the back of the corresponding finger. When the motor 709 does not output force, the rotating shaft is acted on by the spring 713, and then the nylon wire 704 in the wire slot 712 coaxially connected to the rotating shaft is slightly tensioned.

[0066] ​When the fingers pull the nylon wires 704 through the finger tip supports 703, the wire grooves 712 rotate to drive the rotating shafts 714 to rotate, and the rotating shafts 714 rotate to drive the output shafts of the motors 709 to rotate through the second bevel gears 711 and the first bevel gears 710. The rotation amount of the output shafts of the motors 709 is collected by the encoders, and at this time, if the motors 709 apply a reverse torque (opposite to the rotation direction of the wire grooves 712) to the rotating shafts 714, the rotation of the wire grooves 712 is resisted, and thus resistance is provided when the fingers pull the nylon wires 704.

[0067] The glove body 701 is also respectively provided with a vibration motor 708 corresponding to the position of each finger root of the human body. When the vibration motor 708 works, vibration is generated on the glove body 701.

[0068] The action feedback glove 7 is worn on the hand of the spraying personnel. When the spraying personnel moves the hand to form a dynamic tracking gesture, and the fingers of the hand press the action to simulate the pressing of the real spray gun trigger, the pose data of the fingers when the hand moves is collected by the IMU module 705, and the pressing pressure data of the human hand when the hand performs the pressing action is collected by the pressure sensor 707. When the fingers of the hand of the spraying personnel perform the pressing action, each finger pulls the corresponding nylon wire 704 through the finger tip support 703, so that the wire groove 712 rotates to release the nylon wire 704, and then the output shaft of the motor 709 rotates. The rotation amount of the wire groove 712 is obtained by collecting the rotation amount of the output shaft of the motor 709 through the encoder. When the motor 709 is controlled to work to generate a reverse torque opposite to the rotation direction of the wire groove 712, resistance is generated when the corresponding finger pulls the nylon wire, thereby simulating the trigger resistance of the real spray gun. When the spraying personnel stretches the fingers after completing the pressing action of the hand, the motor 709 is controlled to work to drive the wire groove 712 to rotate, so as to retract the nylon wire 704.

[0069] In this embodiment, the gesture camera 9 is a binocular camera, and the gesture camera 9 is installed on the lower side of the display 803 in the head-mounted device 8. The gesture camera 9 collects the hand action images of the hand of the spraying personnel when the hand moves and presses.

[0070] In this embodiment, the environment camera 801 is installed on the upper side of the display 803 in the head-mounted device 8. The environment camera 801 collects the real operation environment images around the hand of the spraying personnel.

[0071] As Figure 6As shown, in this embodiment, the integrated control module 6 is in electrical connection with the vibration sensor and the spraying pressure sensor on the real spray gun 13, the IMU module 705 in the motion feedback glove 7, the pressure sensor 707 in the motion feedback glove 7, the encoder in the motion feedback glove 7, the gesture camera 9, the environment camera 801, and the real scene monitoring camera.

[0072] The integrated control module 6 is also in control electrical connection with the real spraying robot 14 and the real spray gun 13 in the real spraying system, and the vibration motor 708 and each motor 709 in the motion feedback glove 7.

[0073] The integrated control module 6 is also in communication connection with the display 803 in the head-mounted device 8.

[0074] Specifically, as shown Figure 7 The integrated control module 6 includes a three-dimensional simulation module, a data transmission module, and a data processing module, and the data processing module includes a hand motion solving module, a data filtering module, a robot motion inverse solution module, a spraying feedback module, and an environmental obstacle early warning module. When the spraying personnel's hand performs a dynamic tracking gesture by moving and pressing, the data processing process of the integrated control module 6 is as follows:

[0075] The data transmission module obtains the finger pose data collected by the IMU module 705, the grip pressure data collected by the pressure sensor 707, the rotation amount data collected by the encoder, the hand action image collected by the gesture camera 9, and the real operation environment image collected by the environment camera 801, and transmits the finger pose data, the grip pressure data, the rotation amount data, the hand action image, and the real operation environment image to the data processing module.

[0076] In the data processing module, the hand motion solving module uses Kalman filtering method to fuse the hand IMU pose data and the hand action image to solve the action trajectory of the hand of the spraying personnel; then the data filtering module smoothes the action trajectory to form a smoothed action trajectory; then the robot inverse kinematics module scales the filtered smoothed motion trajectory according to the size ratio between the virtual spraying scene 3 and the real spraying scene 12 established as needed, solves the joint angles of each joint of the real spraying robot 14 and the virtual spraying robot 5, thereby obtaining the virtual robot action control instruction for controlling the virtual spraying robot 5 and the real robot action control instruction for controlling the real spraying robot 11.

[0077] In the data processing module, the arm motion solving module calculates the flexion and extension degree data of the fingers according to the rotation data collected by the encoder in combination with the pre-measured size of the hand of the spraying personnel, and solves the virtual spray gun pressure control instruction for controlling the spraying pressure of the virtual spray gun 4 and the real spray gun pressure control instruction for controlling the spraying pressure of the real spray gun 13 in combination with the flexion and extension degree data of the fingers and the grip pressure data collected by the pressure sensor 707.

[0078] In this embodiment, a model matching method is used to establish the correspondence relationship between the rotation amount and the finger bending angle, and the flexion and extension degree data of the fingers are calculated in combination with the rotation amount and the size of the hand of the spraying personnel. In this embodiment, a unified model of rotation data and pressure sensor data is established, and the flexion and extension degree data of the fingers and the grip pressure data are mapped into unified virtual spray gun pressure control instructions and real spray gun pressure control instructions.

[0079] In the data processing module, the environmental obstacle early warning module detects obstacles from the real operation environment images collected by the environmental camera 801, and generates an alarm signal when an obstacle is detected, and the data transmission module transmits the alarm signal to the head-mounted device 8 for alarm prompt.

[0080] In the integrated control module 6, the data transmission module transmits the virtual robot action control instruction and the virtual spray gun pressure control instruction generated by the data processing module to the three-dimensional simulation module. The three-dimensional simulation module generates the virtual spraying robot 5, the virtual spray gun 4, the virtual rotating table 1 and the virtual workpiece to be sprayed 2 placed therein, and controls the overall rotation of the virtual rotating table 1 and the virtual workpiece to be sprayed 2 by the three-dimensional simulation module according to the rotating speed set by the real rotating table 10, controls the action of the virtual spraying robot 5 to the virtual workpiece to be sprayed 2 according to the virtual robot action control instruction, and controls the virtual spray gun to perform spraying work with corresponding spraying pressure according to the virtual spray gun pressure control instruction, thereby forming the required virtual spraying scene 3. Finally, the data transmission module transmits the virtual spraying scene 3 generated by the three-dimensional simulation module to the head-mounted device 8 for display.

[0081] And in the integrated control module 6, the real robot action control instructions generated by the data processing module, the real spray gun pressure control instructions are transmitted to the real spraying robot 14 and the real spray gun 13 in the real spraying system respectively by the data transmission module. The real spraying robot 14 in the real spraying system performs corresponding actions according to the real robot action control instructions, and the real spray gun 13 performs spraying work according to the real spray gun pressure control instructions. Thus, the real spraying scene 12 is formed as shown in FIG. b. Figure 1

[0082] When the real spraying system realizes the real spraying scene 12, the real spraying scene 12 image collected by the real scene monitoring camera is obtained by the data transmission module in the integrated control module 6, and the real spraying scene 12 image is transmitted to the head-mounted device 8 for display by the data transmission module.

[0083] When the real spraying system realizes the real spraying scene 12, the real spraying scene 12 image collected by the real scene monitoring camera is obtained by the data transmission module in the integrated control module 6, and the real spraying scene 12 image is transmitted to the head-mounted device 8 for display by the data transmission module.

[0084] Thus, the integrated control module 6 in the embodiment can follow the moving action and the gripping action of the spraying personnel's hand, display the virtual spraying scene 3 and the real spraying scene 12 synchronously through the head-mounted device 8, and can realize the corresponding spraying action of the spraying robot and the spraying work of the corresponding spraying pressure of the spray gun in the virtual spraying scene 3 and the real spraying scene 12 according to the moving and gripping action of the spraying personnel's hand. The vibration generated during the spraying work of the real spray gun in the real spraying scene 12 and the spraying pressure in the spray gun can be fed back to the spraying personnel.

[0085] ​In this embodiment, the integrated control module 6 further includes a user interaction module, which forms a robot interaction interface and a robot monitoring interface and is transmitted to the upper computer through the data transmission module. The robot interaction interface provides control buttons for the user to directly control the real spraying robot 14 in the Cartesian space through the robot interaction interface of the upper computer; the robot monitoring interface provides a third-person perspective monitoring of the virtual spraying scene 3 and the real spraying scene 12 for the user to observe the pictures of the virtual spraying scene 3 and the real spraying scene 12 through the upper computer.

[0086] In this embodiment, the hand movement of the spraying personnel includes not only dynamic tracking gestures but also static gesture movements. The specific gesture movements of the spraying personnel are collected by the gesture camera 9, and the data transmission module in the integrated control module 6 acquires the specific gesture movements and transmits them to the data processing module. The arm movement solving module in the data processing module determines whether the gesture movement instruction of the spraying personnel is a static gesture or a dynamic tracking gesture. When it is a dynamic tracking gesture, the spraying trajectory and the spraying pressure of the robot spraying movement are controlled; when it is a static gesture, the special instructions such as the start and stop of the dynamic tracking mode, the switching of the paint color, the increase and decrease of the paint flow, and the zooming of the virtual scene are controlled.

[0087] As shown in Figure 8 The embodiment also discloses a teaching method based on the above spraying robot intelligent teaching system, and the process is as follows:

[0088] Step 1: The spraying personnel wear the action feedback gloves 7 and the head-mounted device 8, and complete the gripping action of the hands. The completely open and tightly held hand encoder data are collected through the action feedback gloves 7. The hand movement solving module in the data processing module of the integrated control module 6 adopts a template matching method to process the encoder data to obtain the size of the hands of the spraying personnel.

[0089] Step 2: The spraying personnel generates a dynamic tracking gesture to start teaching. The integrated control module 6 acquires the pose data, the gripping pressure data, the rotation amount data generated in the action feedback gloves 7 when the spraying personnel moves and grips, the hand action image collected by the gesture camera 9, and the real operation environment image collected by the environment camera 801.

[0090] The integrated control module 6 detects obstacles from the real operation environment image, and when an obstacle is detected during the hand movement of the spraying personnel, the head-mounted device 8 alarms.

[0091] The integrated control module 6 solves the movement trajectory of the hands of the spraying personnel according to the finger pose data and the hand action image, and generates virtual robot movement control instructions for controlling the virtual spraying robot 5 and real robot movement control instructions for controlling the real spraying robot 11 according to the movement trajectory.

[0092] The integrated control module 6 calculates the finger flexion degree data according to the rotation amount data in combination with the hand size, and solves the virtual spray gun pressure control instruction for controlling the virtual spray gun 4 spraying pressure and the real spray gun pressure control instruction for controlling the real spray gun 13 spraying pressure in combination with the finger flexion degree data and the grip pressure data.

[0093] Step 3, the integrated control module 6 generates the virtual spraying robot 5, the virtual spray gun 4, and the virtual workpiece 2 to be sprayed, and controls the virtual spraying robot 5 to perform corresponding actions according to the virtual robot action control instruction, and controls the virtual spray gun 4 to perform corresponding spraying operations according to the virtual spray gun pressure control instruction, thereby forming a virtual spraying scene, and transmitting the virtual spraying scene to the head-mounted device 8 for display.

[0094] The integrated control module 6 transmits the generated real robot action control instruction to the real spraying robot 14, and transmits the real spray gun pressure control instruction to the real spray gun 13; the real spraying robot 14 performs corresponding actions according to the real robot action control instruction, and the real spray gun 13 performs corresponding spraying operations according to the real spray gun pressure control instruction, thereby forming a real spraying scene.

[0095] Step 4, when forming the real spraying scene, the integrated control module 6 acquires vibration signal data generated by a vibration sensor on the real spray gun 13, and controls the vibration motor 708 to generate corresponding vibration according to the vibration signal data, so as to feedback the real spray gun 13 vibration feeling to the spraying personnel; and the integrated control module 6 acquires spray gun internal spraying pressure signal data generated by a spraying pressure sensor in the real spray gun 13, and controls the motor 709 to generate corresponding reverse torque according to the spray gun internal spraying pressure signal, so as to feedback the resistance when pressing the real spray gun trigger to the spraying personnel.

[0096] In addition, the integrated control module 6 also acquires real spraying scene images collected by a real scene monitoring camera, and transmits the real spraying scene images to the head-mounted device 8 for display.

[0097] The preferred embodiments of the present application are described in detail in combination with the drawings, and the embodiments described in the present application are only used to describe the preferred embodiments of the present application, and do not limit the concept and scope of the present application. In the above specific embodiments, each specific technical feature described in the above specific embodiments can be combined in any appropriate manner without contradiction, and such combination should also be considered as disclosed by the present disclosure as long as it does not deviate from the idea of the present application. In order to avoid unnecessary repetition, the present application does not further describe various possible combinations.

[0098] The present application is not limited to the specific details of the above-described embodiments, and various modifications and improvements to the technical solutions of the present application made by those skilled in the art within the technical concept of the present application and without departing from the design idea of the present application shall fall within the protection scope of the present application, and the technical content of the present application claimed for protection has been entirely recorded in the claims.

Claims

1. A motion feedback glove for teaching a painting robot, characterized in that, The glove body (701) worn on the human hand is provided with five groups of motor boxes (706) at the position corresponding to the back of the palm, and an IMU module (705) is respectively installed at the position corresponding to each finger of the glove body (701), a fingertip support (703) is respectively fixed at the position corresponding to the fingertip of each finger of the glove body (701), a pressure sensor (707) is respectively installed at the position corresponding to the palm surface and the finger knuckle surface of the glove body (701), and a vibration motor (708) is respectively installed at the position corresponding to the root of each finger of the human hand; Each group of motor boxes (706) is respectively provided with a motor (709), a wire slot (712), a rotating shaft (714) and a gear set, the wire slot (712) is wound with a nylon wire (704), the wire slot (712) is rotatably installed in the motor box (706) through the rotating shaft (714), the output shaft of the motor (709) is in transmission connection with the rotating shaft (714) through the gear set, and the output shaft of the motor (709) is provided with an encoder; The nylon wires (704) on each wire slot (712) are respectively fixedly connected with the fingertip supports (703) of the fingertips of each finger in sequence after being drawn out from the corresponding motor box; The IMU module (705) collects the pose data of the corresponding fingers when the human hand moves, the pressure sensor (707) collects the pressure data when the fingers of the human hand perform the gripping action, and the fingertip support (703) pulls the nylon wire (704) to make the wire slot (712) rotate when the fingers of the human hand perform the gripping action, so as to make the output shaft of the motor (709) rotate through the rotating shaft (714) and the gear set, collect the rotation amount of the output shaft of the motor (709) through the encoder, and apply a torque opposite to the rotation direction of the wire slot (712) to the rotating shaft (714) through the motor (709), so as to generate resistance to the gripping fingers.

2. The spray robot teaching motion feedback glove of claim 1, wherein, A spring holder (715) is further installed in each motor box, a spring (713) is arranged on the spring holder (715), the spring (713) is wound outside the rotating shaft (714), and one end of the spring (713) is fixedly connected with the rotating shaft (714) and the other end is fixedly connected with the spring holder (715).

3. A spray robot intelligent teaching system, characterized in that, It comprises: A real spraying system comprising a real spraying robot (14), a real spraying gun (13) and a real workpiece to be sprayed (11) for forming a real spraying scene; A head-mounted device (8) worn on the head of a spraying person for displaying a virtual spraying robot (5), a virtual spraying gun (4) and a virtual workpiece to be sprayed (2) to the spraying person, the virtual spraying robot (5), the virtual spraying gun (4) and the virtual workpiece to be sprayed (2) being used to form a virtual spraying scene; The motion feedback glove (7) according to any one of claims 1-2, worn on the hand of a spraying person; the pose data of the fingers when the human hand moves is collected by the IMU module (705) in the motion feedback glove (7), the rotation amount of the motor (709) output shaft when the fingers of the human hand are pressed is collected by the encoder in the motion feedback glove (7), and the pressing force data of the fingers of the human hand when the fingers are pressed is collected by the pressure sensor (707) in the motion feedback glove (7); A gesture camera (9) is used to collect the hand movement image when the human hand moves; An integrated control module (6) obtains the pose data, rotation amount, and pressing force data obtained by the motion feedback glove (7), and the hand movement image obtained by the gesture camera (9); The integrated control module (6) solves the motion trajectory of the hand of the spraying person according to the fusion information of the finger pose data and the hand movement image, generates a virtual robot motion control instruction for controlling the virtual spraying robot (5), and a real robot motion control instruction for controlling the real spraying robot (11); The integrated control module (6) calculates the finger flexion degree data according to the rotation amount data when the hand is pressed combined with the size of the hand of the spraying person, and solves the virtual gun pressure control instruction for controlling the spraying pressure of the virtual spraying gun (4) and the real gun pressure control instruction for controlling the spraying pressure of the real spraying gun (13) combined with the pressing force data and the finger flexion degree data; The integrated control module (6) generates the virtual spraying robot (5), the virtual spraying gun (4), and the virtual workpiece to be sprayed (2), controls the virtual spraying robot (5) to perform corresponding actions according to the virtual robot motion control instruction, and controls the virtual spraying gun (4) to perform corresponding spraying work of spraying pressure according to the virtual gun pressure control instruction, thereby forming a virtual spraying scene, and transmitting the virtual spraying scene to the head-mounted device (8) for display; The integrated control module (6) transmits the generated real robot motion control instruction to the real spraying robot (14), and transmits the real gun pressure control instruction to the real spraying gun (13); the real spraying robot (14) performs corresponding actions according to the real robot motion control instruction, and the real spraying gun (13) performs corresponding spraying work of spraying pressure according to the real gun pressure control instruction, thereby forming a real spraying scene.

4. The intelligent teaching system for spray robot according to claim 3, wherein, The real spraying system further comprises a real rotating table (10), the real rotating table (10) is used to place the real workpiece to be sprayed (11), and the real rotating table (10) rotates at a set speed, the real rotating table (10), the real spraying robot (14), the real spraying gun (13) performing spraying work, and the real workpiece to be sprayed (11) form a real spraying scene. The integrated control module (6) also generates a virtual rotating table (10) for placing a virtual workpiece to be sprayed (2), the virtual rotating table (10) is controlled to rotate at a set rotating speed by the integrated control module (6), and a virtual spraying scene is formed by the virtual rotating table (10) in action, the virtual spraying robot (5) in action, the virtual spray gun (4) in spraying operation, and the virtual workpiece to be sprayed (2).

5. The intelligent teaching system for spray robot according to claim 3, wherein, The gesture camera (9) is integrated in the head-mounted device (8).

6. The intelligent teaching system for spray robot according to claim 3, wherein The head-mounted device (8) is also integrated with an environment camera (801), which collects real operation environment images; the integrated control module (6) acquires the real operation environment images collected by the environment camera (801), and detects obstacles from the real operation environment images by the integrated control module (6).

7. The intelligent teaching system for spray robot according to claim 3, wherein It also includes a real scene monitoring camera, which collects real spraying scene images and transmits them to the integrated control module (6), and then the integrated control module (6) transmits the real spraying scene images to the head-mounted device (8) for display.

8. The intelligent teaching system for spray robot according to claim 3, wherein, The real spray gun (13) is provided with a vibration sensor, which collects vibration signals generated by the real spray gun (13) during spraying operation, and transmits the vibration signal data to the integrated control module (6) by the vibration sensor; The integrated control module (6) controls the vibration motor (708) on the motion feedback glove (7) to generate corresponding vibration according to the vibration signal data collected by the vibration sensor.

9. The intelligent teaching system for spray robot according to claim 3, wherein, The real spray gun (13) is provided with a spraying pressure sensor, which collects spraying pressure signal data inside the real spray gun (13) during spraying operation, and transmits the spraying pressure signal data to the integrated control module (6) by the spraying pressure sensor; The integrated control module (6) controls the motor (709) in each motor box (706) on the motion feedback glove (7) to generate corresponding reverse torque according to the spraying pressure signal data collected by the spraying pressure sensor.

10. A teaching method for a teaching system of a painting robot based on any one of claims 3 to 9, characterized in that, The process is as follows: The spraying personnel wear the motion feedback glove (7) and the head-mounted device (8), and complete the hand grasping action, and the hand encoder data of fully opening and gripping is collected by the motion feedback glove 7; the hand motion solving module in the data processing module of the integrated control module 6 adopts a template matching method to process the encoder data to obtain the size of the spraying personnel's hand; The integrated control module (6) acquires the pose data, grip pressure data, and rotation amount data generated in the motion feedback glove (7) when the spraying personnel's hand moves and grips, as well as the hand action images collected by the gesture camera (9); The integrated control module (6) obtains the action trajectory of the spraying personnel's hand according to the finger pose data and the hand action images, and generates virtual robot action control instructions for controlling the virtual spraying robot (5) and real robot action control instructions for controlling the real spraying robot (11) according to the action trajectory; The integrated control module (6) obtains the hand flexion and extension degree data of the sprayer according to the rotation amount data combined with the hand size, and obtains the virtual spray gun pressure control instruction for controlling the spraying pressure of the virtual spray gun (4) and the real spray gun pressure control instruction for controlling the spraying pressure of the real spray gun (13) according to the finger flexion and extension degree data and the spraying pressure data collected by the pressure sensor (707); Then, the integrated control module (6) generates the virtual spraying robot (5), the virtual spray gun (4) and the virtual workpiece (2) to be sprayed, controls the virtual spraying robot (5) to perform corresponding actions according to the virtual robot action control instruction, and controls the virtual spray gun (4) to perform spraying work with corresponding spraying pressure according to the virtual spray gun pressure control instruction, thereby forming a virtual spraying scene, and transmitting the virtual spraying scene to the head-mounted device (8) for display; The integrated control module (6) transmits the generated real robot action control instruction to the real spraying robot (14), and transmits the real spray gun pressure control instruction to the real spray gun (13); the real spraying robot (14) performs corresponding actions according to the real robot action control instruction, and the real spray gun (13) performs spraying work with corresponding spraying pressure according to the real spray gun pressure control instruction, thereby forming a real spraying scene; When the real spraying scene is formed, the integrated control module (6) obtains vibration signal data generated by a vibration sensor on the real spray gun (13), controls the vibration motor (708) on the action feedback glove (7) to generate corresponding vibration according to the vibration signal data, so as to feed back the vibration feeling of the real spray gun (13) to the sprayer; the integrated control module (6) obtains spraying pressure signal data generated by a pressure sensor on the real spray gun (13), controls the motor (709) on the action feedback glove (7) to generate corresponding resistance according to the spraying pressure signal data, so as to feed back the spraying pressure inside the real spray gun (13) to the sprayer.

Images