Industrial robot man-machine interaction method and system based on augmented reality technology

Abstract

The invention relates to the technical field of industrial robots, in particular to an industrial robot man-machine interaction method and system based on the augmented reality technology, and the method comprises the steps: collecting on-site video data, audio data and real-time state data of a robot through an information collection module, and positioning the industrial robot; the collected data is transmitted to the remote control module through the communication module; an operator receives data and sends a control instruction through the remote control module; the control instruction is transmitted to the robot execution module through the communication module and executed to control the robot. By combining the augmented reality technology, an operator is allowed to remotely interact with the industrial robot in real time in an immersive mode, operation convenience and efficiency are improved, in addition, the system feeds back state information of the robot in real time, the operator can adjust operation in time according to the state of the robot, accidents caused by misoperation are avoided, and the safety of the robot is improved. And the safety of a working environment is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial robots, and in particular to an industrial robot human-machine interaction method and system based on augmented reality technology. Background Art

[0002] With the continuous development of industrial automation and robots, more and more industrial production, cargo handling and industrial manufacturing industries have begun to use industrial robots for automated operations to improve the efficiency of production, handling and manufacturing. However, when an industrial robot fails, the operator is often required to go to the site to repair the industrial robot, which is time-consuming and labor-intensive. Even if the operator arrives at the site, he may not be able to get close to the industrial robot and thus cannot determine the cause of the failure. In addition, when the path planning system of some industrial robots with navigation functions fails, they cannot reach the destination on time, resulting in the inability to carry out the task. Summary of the invention

[0003] In view of the above-mentioned problems, the present invention provides an industrial robot human-machine interaction method and system based on augmented reality technology.

[0004] The present invention is achieved through the following technical solutions:

[0005] An industrial robot human-machine interaction method based on augmented reality technology is provided, the method comprising the following steps:

[0006] Step S10: Collecting on-site video data, audio data and real-time status data of the robot through the information collection module;

[0007] Step S20: transmitting the collected data to the remote control module through the communication module;

[0008] Step S30: receiving data and sending control instructions through the remote control module;

[0009] Step S40: Transmit the control instruction to the robot execution module through the communication module and execute the control instruction to control the robot.

[0010] Preferably, the remote operation module in step S30 includes AR glasses, and the operator watches the on-site video data received by the remote operation module in the AR glasses in real time by wearing the AR glasses, and sends control instructions to the robot according to the watched video data.

[0011] Preferably, in step S30, the remote control module further includes a data preprocessing unit for performing image enhancement operations on the video data received by the remote control module at the scene.

[0012] Preferably, the image enhancement operation in the data preprocessing unit adopts an image processing technology based on a convolutional neural network, including: constructing a convolutional neural network model.

[0013] Preferably, the convolutional neural network model adopts a MobileNet network structure, including multiple nodes of at least eight layers, wherein the first layer is an input layer, which inputs the received on-site video data; the second layer is a convolution layer, which is used to extract image features of the on-site video data; the third layer is an activation function layer, and each convolution layer is followed by an activation function layer, which is used to introduce nonlinear features; the fourth layer is a batch normalization layer, which is used to standardize the input of each convolution layer; the fifth layer is a global average pooling layer, which is used to convert the extracted image features into a vector form; the sixth layer is a fully connected layer, which is used to associate the feature vector output by the pooling layer with the output category for final classification; the seventh layer is a Softmax layer, which is used to map the output to a probability distribution and output the probability of each category; the eighth layer is an output layer, and the output is optimized image data.

[0014] In addition, to achieve the above purpose, the present invention also provides an industrial robot human-machine interaction system based on augmented reality technology, comprising:

[0015] Information acquisition module: used to collect on-site video data, audio data and real-time status data of the robot, and locate the position of the industrial robot;

[0016] Communication module: used to transmit the collected data to the remote control module through the communication module;

[0017] Remote control module: used to receive data and send control instructions;

[0018] Robot execution module: used to receive control instructions and execute control instructions to control industrial robots.

[0019] The remote operation module includes AR glasses. The operator wears the AR glasses to watch the live video data received by the remote operation module in the AR glasses in real time, and sends control instructions to the robot according to the watched video data.

[0020] The remote control module further includes a data preprocessing unit for performing image enhancement operations on the on-site video data received by the remote control module.

[0021] Preferably, the information acquisition module includes: a camera, a microphone and a sensor installed on the industrial robot, the camera is used to collect video data on site, the microphone is used to collect audio data on site, the sensor is used to collect real-time status data of the industrial robot, and also includes a GPS positioning system installed on the industrial robot for locating the location of the industrial robot.

[0022] Preferably, the communication module comprises:

[0023] Antenna unit: used to send and receive radio waves;

[0024] RF front-end unit: including power amplifier, filter and low noise amplifier, used to convert digital signals into wireless RF signals and process received RF signals;

[0025] RF transceiver unit: used for signal modulation and demodulation;

[0026] Baseband signal processor unit: used to detect and correct signal errors.

[0027] Preferably, the remote control module includes:

[0028] Data processing unit: used to perform image enhancement on the video data received by the remote control module at the scene;

[0029] User interface unit: including AR glasses, where the operator views the image-enhanced video data on the AR glasses interface, and the real-time status data and location of the industrial robot are displayed on the AR glasses interface;

[0030] Input device unit: including gesture recognition device and voice recognition device, used for receiving gestures and voice commands issued by the operator;

[0031] Logic control unit: used to convert gestures and voice commands issued by the operator into control commands for the industrial robot;

[0032] Interactive feedback unit: used to provide visual, auditory and tactile feedback to inform the operator of the operation results;

[0033] Data protection unit: used to protect data security and prevent unauthorized access.

[0034] Preferably, the robot execution module includes:

[0035] Motion control unit: used to control the movement of the industrial robot according to the received control instructions;

[0036] Execution control unit: used to perform specific actions, such as grabbing, roller and robot arm movements;

[0037] Safety protection unit: used to immediately stop the movement of the industrial robot when a collision or overload occurs.

[0038] The advantages and effects of the present invention are:

[0039] The method of the present invention enables the operator to interact with the industrial robot in real time in a remote immersive manner, so that the operator can understand the situation on site and the status and faults of the industrial robot through AR glasses without rushing to the industrial site, thereby improving the convenience and efficiency of operation. The industrial robot status information, location and environment fed back in real time through the AR glasses interface enable the operator to adjust the operation in time, avoid accidents caused by operational errors, and ensure the safety of the working environment. In addition, the operator can immersively control the movement of the industrial robot through AR glasses to help the industrial robot complete the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 The present invention is a flowchart of an industrial robot human-machine interaction method based on augmented reality technology.

[0042] Figure 2 This is a structural diagram of the convolutional neural network model for image enhancement of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The present invention provides an industrial robot human-machine interaction method based on augmented reality technology, such as Figure 1 As shown, the following steps are included:

[0045] Step S10: Collecting on-site video data, audio data and real-time status data of the robot through the information collection module;

[0046] Among them, the real-time status data of the robot includes position and posture data, such as coordinate position, posture angle, joint angle and precision positioning; speed and acceleration data, such as linear velocity, angular velocity and acceleration; sensor data: torque sensor data, tactile sensor data, visual sensor data and distance sensor data; dynamic data: motor current and voltage, robot battery power and driver temperature, current, etc.; communication status data: robot's network connection status such as WIFI signal strength, communication delay, data transmission rate, packet loss rate, etc.

[0047] Step S20: Transmit the collected data to the remote control module through the communication module.

[0048] Among them, the communication module adopts wired communication protocols and wireless communication protocols. Wired communication protocols such as Ethernet, RS-232, RS-485, etc. are adopted when the distance between the operating console and the industrial robot is far; wireless communication protocols such as Wi-Fi, Zigbee, Bluetooth, etc. are adopted when the distance between the operating console and the industrial robot is close.

[0049] Step S30: receiving data and sending control instructions through the remote control module.

[0050] Among them, the remote operation module includes AR glasses. The operator wears AR glasses to watch the on-site video data received by the remote operation module in the AR glasses in real time, and sends control instructions to the robot according to the viewed video data; it also includes a data preprocessing unit for performing image enhancement operations on the on-site video data received by the remote control module.

[0051] Among them, the image enhancement operation in the data preprocessing unit adopts image processing technology based on convolutional neural network, including: building a convolutional neural network model.

[0052] The convolutional neural network model uses the MobileNet network structure. MobileNet is a lightweight deep learning model designed for mobile and embedded devices. Its goal is to significantly reduce the amount of computation and model size while ensuring high accuracy, so that it can run on resource-constrained devices. It is very suitable for real-time image enhancement operations on the images taken by the cameras installed on industrial robots. It includes at least eight layers of multiple nodes, such as Figure 2 As shown:

[0053] The first layer is the input layer, which inputs the received on-site video data;

[0054] The second layer is a convolutional layer, which is used to extract image features of the on-site video data;

[0055] The third layer is the activation function layer. There is an activation function layer after each convolution layer to introduce nonlinear features.

[0056] The fourth layer is the batch normalization layer, which is used to standardize the input of each convolutional layer;

[0057] The fifth layer is the global average pooling layer, which is used to convert the extracted image features into vector form;

[0058] The sixth layer is a fully connected layer, which is used to associate the feature vector output by the pooling layer with the output category for final classification;

[0059] The seventh layer is the Softmax layer, which is used to map the output to the probability distribution and output the probability of each category;

[0060] The eighth layer is the output layer, and the output is the optimized image data.

[0061] Step S40: Transmit the control instruction to the robot execution module through the communication module and execute the control instruction to control the robot.

[0062] After receiving the control instruction, the robot execution module converts the control instruction into a data packet and sends it to the industrial robot control system. After receiving the data packet, the industrial robot control system parses it into specific control instructions and controls the industrial robot's forward, backward, steering and joint movement according to the parsed specific control instructions.

[0063] In addition, the present invention also proposes an industrial robot human-machine interaction system based on augmented reality technology, comprising:

[0064] Information acquisition module: used to collect on-site video data, audio data and real-time status data of the robot, and locate the position of the industrial robot;

[0065] Communication module: used to transmit the collected data to the remote control module through the communication module;

[0066] Remote control module: used to receive data and send control instructions;

[0067] Robot execution module: used to receive control instructions and execute control instructions to control industrial robots.

[0068] The remote operation module includes AR glasses. The operator wears the AR glasses to watch the live video data received by the remote operation module in the AR glasses in real time, and sends control instructions to the robot according to the watched video data.

[0069] The remote control module further includes a data preprocessing unit for performing image enhancement operations on the on-site video data received by the remote control module.

[0070] Specifically, the information collection module includes a camera, a microphone and a sensor installed on the industrial robot. The camera is used to collect video data on site, the microphone is used to collect audio data on site, and the sensor is used to collect real-time status data of the industrial robot. In addition, it also includes a GPS positioning system installed on the industrial robot for locating the location of the industrial robot.

[0071] Specifically, the communication module includes:

[0072] Antenna unit: used to send and receive radio waves. Sending refers to converting the RF signal output from the RF transceiver unit into electromagnetic waves in space, and receiving refers to converting the received electromagnetic waves into electrical signals and transmitting them to the RF front-end unit. Depending on the application scenario, indoors or outdoors, the antenna can be an omnidirectional antenna, a directional antenna, a phased array antenna, etc.

[0073] RF front-end unit: including power amplifier, filter and low-noise amplifier, used to convert digital signals into wireless RF signals and process received RF signals. The power amplifier is used to amplify the power of the signal output by the baseband signal processor unit to ensure that the signal has enough energy to propagate in the air. The filter is used to filter out unnecessary frequency components to prevent interference and stray signals from entering the receiving link. The low-noise amplifier is used to amplify the received weak signal while minimizing the introduction of noise to improve the receiving sensitivity.

[0074] RF transceiver unit: used for signal modulation and demodulation, including modulator, demodulator and frequency synthesizer. The modulator is used to modulate the baseband signal onto a high-frequency carrier so that it can be sent out through the antenna unit. The demodulator is used to demodulate the received RF signal and restore the original baseband signal. The frequency synthesizer is used to generate the required carrier frequency for modulation and demodulation.

[0075] Baseband signal processor unit: used to detect and correct signal errors, encode the received data to improve the reliability of transmission, decode the data before sending and restore the original information, add redundant information through forward error correction, so that the receiving end can detect and correct errors in transmission, and encrypt the data to protect data security. Decryption is used to restore the original data.

[0076] Specifically, the remote control module includes:

[0077] Data processing unit: used to perform image enhancement on the video data received by the remote control module at the scene, and use the image processing technology based on convolutional neural network to perform real-time image enhancement operation on the camera shooting picture;

[0078] User interface unit: including AR glasses. The operator views the enhanced video data on the AR glasses interface, and the real-time status data and location of the industrial robot are displayed on the AR glasses interface. The real-time status data of the industrial robot includes the movement speed, direction, joint angle, etc. of the industrial robot. The location is the precise positioning of the current industrial robot, which can help the operator better understand the working environment of the industrial robot.

[0079] Input device unit: including gesture recognition device and voice recognition device, used to receive gestures and voice commands issued by the operator, specifically, to capture the operator's gestures through a camera or other sensors and recognize specific gesture actions, and to capture the operator's voice commands through a microphone and convert them into text or control commands through voice recognition technology;

[0080] Logic control unit: used to convert gestures and voice commands issued by the operator into control commands for the industrial robot, including command parsing, which refers to parsing gestures and voice commands into corresponding control commands.

[0081] Interactive feedback unit: used to provide visual, auditory and tactile feedback to inform the operator of the operation results. Visual feedback refers to the display of operation results on the AR glasses interface, such as confirmation that the action has been executed, warning information, etc. Auditory feedback refers to playing sound prompts through speakers or headphones to inform the operator of the action status. Tactile feedback refers to providing physical feedback through vibration or other tactile devices to enhance the operating experience.

[0082] Data protection unit: used to protect data security and prevent unauthorized access, including encrypted transmission, identity authentication, permission management and log recording. Encrypted transmission refers to encrypting the transmitted data to prevent the data from being intercepted during transmission. Authentication refers to authenticating the operators who access the system to ensure that only authorized personnel can operate. Permission management refers to setting different permission levels for different operators to limit access to sensitive data. Logging refers to recording all operation logs to track and audit operation behaviors.

[0083] Specifically, the robot execution module includes:

[0084] Motion control unit: used to control the movement of the industrial robot according to the received control instructions, including controller, driver and sensor. The controller can be a microcontroller (MCU) or digital signal processor (DSP) for processing control logic, the driver can be a servo driver and stepper motor driver for driving the motor, and the sensor can be an encoder and position sensor for feedback of the actual position and status.

[0085] Execution control unit: used to perform specific actions, such as grabbing, roller and robot arm movements;

[0086] Safety protection unit: used to immediately stop the action of the industrial robot when a collision and overload occur, including collision detection sensors, such as contact sensors, lidar, etc., for detecting external collisions, overload protection devices, such as current detection circuits, torque sensors, etc., for detecting overload conditions, emergency braking devices, such as emergency stop buttons, software protection mechanisms, etc., for immediately stopping the action.

[0087] The industrial robot human-machine interaction system based on augmented reality technology provided by the present application adopts the industrial robot human-machine interaction method based on augmented reality technology in the above-mentioned embodiment, which can solve the technical problem of low efficiency of traditional industrial robot human-machine interaction. Compared with the prior art, the beneficial effects of the industrial robot human-machine interaction system based on augmented reality technology provided by the present application are the same as the beneficial effects of the industrial robot human-machine interaction method based on augmented reality technology provided by the above-mentioned embodiment, and other technical features of the industrial robot human-machine interaction system based on augmented reality technology are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0088] The various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0089] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An industrial robot human-machine interaction method based on augmented reality technology, characterized in that: The method comprises the following steps: Step S10: collecting on-site video data, audio data and real-time status data of the robot through the information collection module, and positioning the industrial robot; Step S20: transmitting the collected data to the remote control module through the communication module; Step S30: receiving data and sending control instructions through the remote control module; Step S40: transmitting the control instruction to the robot execution module through the communication module and executing the control instruction to control the robot; In step S30, the remote operation module includes AR glasses, and the operator watches the live video data received by the remote operation module in the AR glasses in real time by wearing the AR glasses, and sends control instructions to the robot according to the watched video data; Wherein, in step S30, the remote control module further includes a data preprocessing unit for performing image enhancement operation on the on-site video data received by the remote control module.

2. The industrial robot human-machine interaction method based on augmented reality technology according to claim 1, characterized in that: The image enhancement operation in the data preprocessing unit adopts an image processing technology based on a convolutional neural network, including: constructing a convolutional neural network model; The convolutional neural network model adopts the MobileNet network structure, including multiple nodes of at least eight layers, wherein the first layer is the input layer, which inputs the received on-site video data; the second layer is the convolution layer, which is used to extract the image features of the on-site video data; the third layer is the activation function layer, and each convolution layer has an activation function layer after it, which is used to introduce nonlinear features; the fourth layer is the batch normalization layer, which is used to standardize the input of each convolution layer; the fifth layer is the global average pooling layer, which is used to convert the extracted image features into vector form; the sixth layer is the fully connected layer, which is used to associate the feature vector output by the pooling layer with the output category for final classification; the seventh layer is the Softmax layer, which is used to map the output to the probability distribution and output the probability of each category; the eighth layer is the output layer, and the output is the optimized image data.

3. An industrial robot human-machine interaction system based on augmented reality technology, characterized in that: include: Information acquisition module: used to collect on-site video data, audio data and real-time status data of the robot, and locate the position of the industrial robot; Communication module: used to transmit the collected data to the remote control module; Remote control module: used to receive data and send control instructions; Robot execution module: used to receive control instructions and execute control instructions to control the robot; The remote operation module includes AR glasses. The operator wears the AR glasses to watch the live video data received by the remote operation module in the AR glasses in real time, and sends control instructions to the robot according to the watched video data. The remote control module further includes a data preprocessing unit for performing image enhancement operations on the on-site video data received by the remote control module.

4. The industrial robot human-machine interaction system based on augmented reality technology according to claim 3 is characterized in that: The information acquisition module includes a camera, a microphone and a sensor installed on the industrial robot. The camera is used to collect video data on site, the microphone is used to collect audio data on site, and the sensor is used to collect real-time status data of the industrial robot. In addition, it also includes a GPS positioning system installed on the industrial robot to locate the location of the industrial robot.

5. The industrial robot human-machine interaction system based on augmented reality technology according to claim 3, characterized in that: The communication module comprises: Antenna unit: used to send and receive radio waves; RF front-end unit: including power amplifier, filter and low noise amplifier, used to convert digital signals into wireless RF signals and process received RF signals; RF transceiver unit: used for modulation and demodulation of signals; Baseband signal processor unit: used to detect and correct signal errors.

6. The industrial robot human-machine interaction system based on augmented reality technology according to claim 3, characterized in that: The remote control module includes: Data processing unit: used for performing image enhancement on the received live video data; User interface unit: including AR glasses, where the operator views the image-enhanced video data on the AR glasses interface, and the real-time status data and location of the industrial robot are displayed on the AR glasses interface; Input device unit: including gesture recognition device and voice recognition device, used for receiving gestures and voice commands issued by the operator; Logic control unit: used to convert gestures and voice commands issued by the operator into control commands for the industrial robot; Interactive feedback unit: used to provide visual, auditory and tactile feedback to inform the operator of the operation results; Data protection unit: used to protect data security and prevent unauthorized access.

7. The industrial robot human-machine interaction system based on augmented reality technology according to claim 3, characterized in that: The robot execution module includes: Motion control unit: used to control the movement of the industrial robot according to the received control instructions; Execution control unit: used to perform specific actions, including grabbing, roller and robot arm actions; Safety protection unit: used to immediately stop the movement of the industrial robot when a collision or overload occurs.

Images