Remote control system based on visual technology and neural network algorithm

Abstract

The invention relates to the technical field of teleoperation, in particular to a remote control system based on a visual technology and a neural network algorithm. Image information of an operator is obtained through a recognition camera, and mechanical arm actions of the robot, the visual angle of a robot head camera and movement of the robot are controlled through the image information of the operator through a palm recognition unit, a head camera moving unit and a synchronous moving unit. The system can be used in long-time operation or complex tasks, and the burden of an operator is not increased.

Description

Technical Field

[0001] The invention relates to the technical field of remote operation, in particular to a remote control system based on visual technology and neural network algorithm. Background Art

[0002] In reality, many operations require robots to have the ability to walk, perform fine operations with both hands, and adjust their visual field like humans. There are several important backgrounds behind this demand that have prompted the development of teleoperation technology. First, humans have many limitations when performing high-risk or complex tasks. For example, in the radiation area of ​​a nuclear power plant, in places with explosion hazards, or in deep-sea environments, the safety of operators cannot be effectively guaranteed. Teleoperation technology enables robots to replace humans and enter these dangerous environments to perform operations, thereby greatly improving safety. In addition, many tasks require extremely delicate operations, such as minimally invasive surgery in the medical field or precision assembly in the industrial field, which require robots to have the flexibility and precision of human hands. Through teleoperation technology, operators can remotely control the robot arm to complete complex and delicate operations. In terms of vision, many operating scenarios require robots to be able to adjust their own visual field and observe and perceive the environment from different angles, such as searching by adjusting the camera angle in search and rescue missions, or testing product quality on the production line. The application of teleoperation technology enables robots to not only respond flexibly in complex and dynamic environments, but also form efficient collaborative operations with human operators, thereby achieving more efficient, safe and precise operations. Therefore, the demand for robots to have the ability to walk, perform fine operations and make visual adjustments has greatly promoted the development and application of remote operation technology.

[0003] Existing teleoperation technologies are mainly divided into the following two types: 1. Posture sensor synchronization technology: This method captures the body's motion and posture information by wearing a posture sensor on the human body, and then synchronizes this information to the robot in real time. This enables the robot to imitate human movements and achieve precise remote control operation.

[0004] 2. Master-end teaching technology: This involves teaching actions by wearing an exoskeleton device or using a desktop robotic arm. In this method, the joints or ends of the master end are directly mapped to the robot, enabling the robot to replicate complex movements and operations.

[0005] However, with the above two methods, operators need to wear multiple sensors and devices for a long time, which causes physical discomfort and fatigue. In addition, the teaching process is long and requires high physical strength and endurance of the operator. These factors limit the application of these technologies in long-term operations or complex tasks, and increase the burden on operators. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a remote control system based on visual technology and neural network algorithm, which can be used in long-term operations or complex tasks without increasing the burden on the operator.

[0007] In order to solve the above problems, the following technical solutions are provided: A remote control system based on visual technology and neural network algorithm, including: Identification camera, used to obtain image information of the operator; A palm recognition unit, adapted to be connected to the recognition camera, is used to recognize the palm position and posture information of the operator from the image information of the operator, and control the robot arm movement according to the palm position and posture information of the operator; A head camera moving unit is adapted to be connected with the recognition camera, and is used to recognize the operator's head posture information from the operator's image information, and control the robot's head camera posture through the operator's head posture information to ensure that the robot's head camera viewing angle is consistent with the operator's viewing angle; The synchronous moving unit is adaptively connected to the recognition camera and is used to identify the operator's body center position and posture information from the operator's image information, and control the robot to move synchronously with the operator through the operator's body center position and posture information.

[0008] In the above-mentioned remote control system based on visual technology and neural network algorithm, the process of the palm recognition unit recognizing the palm position and posture information of the operator and controlling the robot's mechanical arm movement is: First, 21 key feature points of the human palm are identified from the operator's image information; Next, the two-dimensional pixel coordinates of the 21 key feature points of the human palm are projected into the recognition camera coordinate system, thereby accurately describing the position and posture of the palm in three-dimensional space; Then, the normal vector of the palm is calculated through the three feature points of the thumb, middle finger and little finger on the palm, which is used to describe the direction of the palm and the posture of the palm in three-dimensional space; Afterwards, based on the calculated normal vector and the position information of the palm, the posture of the palm is projected into the robot's workspace, so that the position and posture of the palm become the target posture of the end effector of the robot's mechanical arm; Afterwards, the robot arm end effector controls the arm to reach the target posture; In the above-mentioned remote control system based on visual technology and neural network algorithm, the 21 key feature points of the human palm include the positions of various joints of the fingers and the wrist.

[0009] In the above-mentioned remote control system based on vision technology and neural network algorithm, the wrist point is the reference position of the end effector of the robot arm.

[0010] In the above-mentioned remote control system based on vision technology and neural network algorithm, when the end effector of the robotic arm controls the robotic arm to reach the target posture, an analytical solution method is used to calculate the rotation angle of each joint of the robotic arm.

[0011] In the above-mentioned remote control system based on visual technology and neural network algorithm, the process in which the head camera moving unit recognizes the operator's head posture information from the operator's image information and controls the robot's head camera posture is: First, the operator’s facial features are detected and recognized in real time; Next, the relative position changes of facial feature points are analyzed to calculate the Euler angle transformation of the face; Afterwards, the robot's head camera posture is controlled through Euler angle transformation.

[0012] In the above-mentioned remote control system based on vision technology and neural network algorithm, the Euler angle includes the yaw angle, pitch angle and roll angle of the operating face.

[0013] In the above-mentioned remote control system based on visual technology and neural network algorithm, the process in which the synchronous movement unit recognizes the operator's body center position and posture information from the operator's image information and controls the robot to move synchronously with the operator is: First, identify the characteristic points of each key joint of the human body; Next, the feature points of each key joint of the human body are tracked to obtain and calculate the initial position and posture of the center of the human body; Then, when the operator moves, the detection calculates the offset of the body center relative to the initial position, as well as the change in the body posture; Afterwards, the robot chassis motion is controlled according to the calculated offset and posture changes.

[0014] In the above-mentioned remote control system based on vision technology and neural network algorithm, the characteristic points of each key joint of the human body include shoulders, elbows, hip joints, and knees.

[0015] In the above-mentioned remote control system based on vision technology and neural network algorithm, the chassis movement amount of the robot is determined by an odometer.

[0016] The above solution has the following advantages: Since the remote control system based on visual technology and neural network algorithm of the present invention obtains the image information of the operator through the recognition camera, the palm recognition unit, the head camera movement unit and the synchronous movement unit are used to control the robot's mechanical arm movement, the robot's head camera angle and the robot's movement respectively through the operator's image information. This ensures that the operator does not need to wear any physical sensors or exoskeleton equipment, and can operate only through the image data captured by the camera. This method reduces the physical burden and discomfort of wearing the device on the human body, while eliminating the time and complexity of wearing the device. This improves the comfort and convenience of operation, enables the operator to operate more freely, and reduces the risk of equipment damage or failure. Therefore, the system can be applied to tasks that require long-term operation or complex tasks, and will not increase the burden on the operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural topology diagram of the invented remote control system based on visual technology and neural network algorithm. DETAILED DESCRIPTION

[0018] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0019] like Figure 1 As shown, the remote control system based on visual technology and neural network algorithm of the present invention is a remote control master end, which is used to control the work of the slave end, the slave end robot. In this embodiment, the robot controlled by the remote control master end is a mobile lifting dual-arm robot.

[0020] The remote control master end includes a recognition camera, a palm recognition unit, a head camera moving unit and a synchronous moving unit. The recognition camera is connected to the palm recognition unit, the head camera moving unit and the synchronous moving unit by electrical signals. The palm recognition unit, the head camera moving unit and the synchronous moving unit can be integrated on the robot or integrated on the recognition camera. When the palm recognition unit, the head camera moving unit and the synchronous moving unit can be integrated on the robot, the recognition camera is wirelessly connected to the palm recognition unit, the head camera moving unit and the synchronous moving unit, and the palm recognition unit, the head camera moving unit and the synchronous moving unit are wirelessly connected or wired to the robot master controller. When the palm recognition unit, the head camera moving unit and the synchronous moving unit are integrated with the recognition camera, the palm recognition unit, the head camera moving unit and the synchronous moving unit are wired or wirelessly connected to the recognition camera, and the palm recognition unit, the head camera moving unit and the synchronous moving unit are wirelessly connected to the robot. The specific wireless connection and wired connection structure belong to the prior art and will not be repeated here.

[0021] The recognition camera is used to obtain the image information of the operator in real time. In this embodiment, the recognition camera adopts an RGBD camera.

[0022] The palm recognition unit is adapted to be connected with the recognition camera, and is used to recognize the palm position and posture information of the operator from the image information of the operator, and control the robot arm movement by the palm position and posture information of the operator, and the specific steps are: First, a neural network for visual recognition of human palm joints (such as Mediapipe HandLandmark Detection, OmniHand, ACR, IntagHand, Yolov8, etc.) is used to identify 21 key feature points of the human palm from the operator's image information. In this embodiment, the 21 key feature points of the human palm include the positions of each joint of the finger and the wrist, so that the posture and movement trajectory of the palm can be accurately captured.

[0023] Next, the two-dimensional pixel coordinates of the 21 key feature points of the human palm are projected into the recognition camera coordinate system, thereby accurately describing the position and posture of the palm in three-dimensional space. In this embodiment, the wrist point is selected as the reference position of the end effector of the robot arm, so that the end of the arm will be adjusted accordingly according to the movement of the wrist point to achieve accurate tracking of the palm movement.

[0024] Then, the normal vector of the palm is calculated through the three feature points of the thumb, middle finger and little finger on the palm, which is used to describe the direction of the palm and the posture of the palm in three-dimensional space.

[0025] Afterwards, based on the calculated normal vector and the position information of the palm, the posture of the palm is projected into the robot's workspace, so that the position and posture of the palm become the target posture of the end effector of the robot's mechanical arm.

[0026] Afterwards, the robot arm end effector controls the arm to reach the target posture. In this embodiment, when the end effector of the arm controls the arm to reach the target posture, an analytical solution method is used to calculate the rotation angle of each joint of the arm. The analytical solution method is used to calculate the rotation angle of each joint of the arm, thereby ensuring that the teleoperated arm can flexibly respond to the movement and rotation of the operator's palm and achieve high-precision remote control operation. This not only improves the accuracy and efficiency of teleoperation, but also enhances the interactive experience between the operator and the robot.

[0027] The head camera moving unit is connected with the recognition camera adapter to identify the operator's head posture information from the operator's image information, and control the robot's head camera posture through the operator's head posture information to ensure that the robot's head camera perspective is consistent with the operator's perspective. The specific process is: First, the operator’s facial features are detected and recognized in real time through the neural network for visual recognition of human faces (such as SynergyNet, RankPose, 2DASL, 3DDFA- V2, PRNet, 2DASL, BlazeFace, etc.).

[0028] Next, the relative position changes of facial feature points are analyzed to calculate the Euler angle transformation of the face, including the yaw angle (rotation around the vertical axis), pitch angle (rotation around the horizontal axis), and roll angle (rotation around the front-back axis). By calculating these angles, the posture changes of the operator's head in three-dimensional space can be determined. Using these calculated Euler angles, the system can synchronize the posture changes of the operator's head in real time and use them to control the robot's head camera viewing angle.

[0029] Afterwards, the robot's head camera posture is controlled through Euler angle transformation.

[0030] Specifically, when the operator turns his head, the camera will adjust its viewing angle synchronously to maintain a field of view consistent with the operator's line of sight. This synchronization mechanism enables the camera to dynamically respond to the operator's head movement, improving the system's immersive and interactive experience. Through this head-camera synchronous movement technology, the teleoperation system can not only achieve more natural viewing angle control, but also enhance the operator's perception of the remote environment. This intuitive control method helps to improve the control accuracy and efficiency in complex tasks, and is widely used in virtual reality, augmented reality, and remote operation.

[0031] The synchronous moving unit is adapted to be connected with the recognition camera, and is used to identify the operator's body center position and posture information from the operator's image information, and control the robot to move synchronously with the operator through the operator's body center position and posture information, specifically: First, computer vision algorithms (such as MediaPipe Pose Landmark Detection, Poseformer, OpenPose, MoveNet, DCPose, DeepCut, DensePose, AlphaPose, etc.) are used to read the image from the recognition camera in real time, so as to monitor the operator's body in real time and identify the feature points of each key joint of the human body. These joint feature points usually include shoulders, elbows, hip joints, knees, etc.

[0032] Next, the feature points of each key joint of the human body are tracked to obtain and calculate the initial position and posture of the center of the human body.

[0033] Then, when the operator moves, the detection calculates the offset of the body center relative to the initial position, as well as the change in the body posture (such as the rotation angle).

[0034] Afterwards, the odometer is used to determine the motion instructions of the robot chassis wheels based on the calculated offset and posture changes in order to control the robot's motion. According to the actual situation, the system needs to calculate whether the chassis wheels should rotate forward or reverse at each moment, and how many turns they should make, in order to match the operator's movement intention.

[0035] Controlling the robot's movements includes: forward and backward: when the center of the body is displaced along the forward direction of the chassis, the system instructs the chassis wheels to rotate forward to achieve forward movement; when the displacement direction is opposite, the wheels are instructed to reverse to achieve backward movement.

[0036] Left and right turns: By detecting the human body's posture offset, when the operator rotates the body to change direction, the system calculates the corresponding angle and instructs the chassis wheels to perform differential rotation to achieve left or right turns. This vision-based mobile chassis synchronous motion control method enables the system to flexibly respond to the operator's movement and rotation to achieve precise synchronous motion. It not only improves the naturalness and intuitiveness of operation, but also provides efficient mobile control without the need for additional sensors. This technology is suitable for application scenarios that require high flexibility and precision, such as service robots, warehouse automation, and auxiliary medical equipment.

[0037] The remote control system based on visual technology and neural network algorithm of the present invention has the following specific advantages: 1. Contactless control: Operators do not need to wear any physical sensors or exoskeleton equipment, and can operate only through image data captured by the camera. This method reduces the physical burden and discomfort of wearing equipment on the human body, while eliminating the time and complexity of wearing equipment. This improves the comfort and convenience of operation, allows operators to operate more freely, and reduces the risk of equipment damage or failure.

[0038] 2. High flexibility and adaptability: The system is able to operate normally under a variety of environmental conditions, including different lighting conditions, background complexity, and various operating scenarios. The technology does not rely on a specific physical interface or environmental setting.

[0039] Advantages: This flexibility allows the technology to be widely used in multiple fields such as industrial, medical, and service robotics, suitable for different mission requirements and operating environments.

[0040] 3. Enhanced intuitiveness and natural interaction: By directly using the operator's natural movements (such as gestures, head, and walking movements) for control, the system can achieve a natural and intuitive interaction method that conforms to human natural movement habits. The threshold for learning and use is lowered, and users can adapt to and operate the system more quickly, thereby improving usage efficiency and user experience.

[0041] 4. Precise motion control: Through advanced computer vision algorithms, the system can accurately identify and track the feature points of the hands and head in real time, achieving precise control of the robotic arm and camera. This high-precision control enables the system to perform complex operating tasks and is suitable for application scenarios that require high precision and reliability.

[0042] 5. Reduce costs and simplify system design: Since there is no need for complex physical sensors and exoskeleton equipment, the hardware cost and complexity of the system are significantly reduced, and the design is more simplified. The initial investment and maintenance costs are reduced, making the technology more attractive in commercial applications and conducive to widespread promotion and application.

[0043] 6. Real-time response capability: The system can process visual data in real time with extremely low latency, ensuring a quick response to the operator's actions. This involves optimizing the data processing flow and improving computing efficiency. This improves the smoothness and efficiency of operations and is suitable for application scenarios that require fast response and high-dynamic operations, such as remote control and virtual reality.

[0044] 7. Security and Scalability: Since there is no need to physically restrain the operator, the safety of the system during use is improved. At the same time, the system design has good scalability and can be expanded according to different application requirements. While ensuring operational safety, the system can flexibly expand its functions and adapt to the development of new technologies to meet the diverse needs of different application fields.

[0045] 8. Applicable to complex environments: In remote operations or dangerous environments, this technology can reduce the operator's direct exposure to risks and complete tasks through visual teleoperation. It improves the safety of operators while still being able to efficiently complete tasks in complex and dangerous environments, such as nuclear power plant maintenance, deep sea exploration, or outer space operations.

Claims

1. A remote control system based on visual technology and neural network algorithm, characterized in that: include: Identification camera, used to obtain image information of the operator; A palm recognition unit, adapted to be connected to the recognition camera, is used to recognize the palm position and posture information of the operator from the image information of the operator, and control the robot arm movement according to the palm position and posture information of the operator; A head camera moving unit is adapted to be connected with the recognition camera, and is used to recognize the operator's head posture information from the operator's image information, and control the robot's head camera posture through the operator's head posture information to ensure that the robot's head camera viewing angle is consistent with the operator's viewing angle; The synchronous moving unit is adaptively connected to the recognition camera and is used to identify the operator's body center position and posture information from the operator's image information, and control the robot to move synchronously with the operator through the operator's body center position and posture information.

2. The remote control system based on visual technology and neural network algorithm as claimed in claim 1, characterized in that: The process of the palm recognition unit recognizing the palm position and posture information of the operator and controlling the robot's mechanical arm movement is: First, 21 key feature points of the human palm are identified from the operator's image information; Next, the two-dimensional pixel coordinates of the 21 key feature points of the human palm are projected into the recognition camera coordinate system, thereby accurately describing the position and posture of the palm in three-dimensional space; Then, the normal vector of the palm is calculated through the three feature points of the thumb, middle finger and little finger on the palm, which is used to describe the direction of the palm and the posture of the palm in three-dimensional space; Afterwards, based on the calculated normal vector and the position information of the palm, the posture of the palm is projected into the robot's workspace, so that the position and posture of the palm become the target posture of the end effector of the robot's mechanical arm; Afterwards, the robot arm end effector controls the arm to reach the target position.

3. The remote control system based on visual technology and neural network algorithm as claimed in claim 2, characterized in that: The 21 key feature points of the human palm include the positions of various joints of the fingers and the wrist.

4. The remote control system based on visual technology and neural network algorithm as claimed in claim 2, characterized in that: The wrist point is the reference position of the end effector of the robot arm.

5. The remote control system based on visual technology and neural network algorithm as claimed in claim 2, characterized in that: When the end effector of the robotic arm controls the robotic arm to reach the target posture, an analytical solution method is used to calculate the rotation angle of each joint of the robotic arm.

6. The remote control system based on visual technology and neural network algorithm as claimed in claim 1, characterized in that: The process of the head camera moving unit identifying the operator's head posture information from the operator's image information and controlling the robot's head camera posture is: First, the operator’s facial features are detected and recognized in real time; Next, the relative position changes of facial feature points are analyzed to calculate the Euler angle transformation of the face; Afterwards, the robot's head camera posture is controlled through Euler angle transformation.

7. The remote control system based on visual technology and neural network algorithm as claimed in claim 6, characterized in that: The Euler angles include a yaw angle, a pitch angle, and a roll angle of the operating face.

8. The remote control system based on visual technology and neural network algorithm as claimed in claim 1, characterized in that: The process in which the synchronous movement unit identifies the operator's body center position and posture information from the operator's image information and controls the robot to move synchronously with the operator is: First, identify the characteristic points of each key joint of the human body; Next, the feature points of each key joint of the human body are tracked to obtain and calculate the initial position and posture of the center of the human body; Then, when the operator moves, the detection calculates the offset of the body center relative to the initial position, as well as the change in the body posture; Afterwards, the robot chassis motion is controlled according to the calculated offset and posture changes.

9. The remote control system based on visual technology and neural network algorithm as claimed in claim 8, characterized in that: The characteristic points of each key joint of the human body include shoulders, elbows, hip joints, and knees.

10. The remote control system based on visual technology and neural network algorithm as claimed in claim 8, characterized in that: The chassis movement of the robot is determined by an odometer.

Images