Large mechanical arm first-view remote control method based on internet of things and augmented reality

Abstract

The application relates to the field of intelligent control, and discloses a large mechanical arm first-view remote control method based on the Internet of Things and augmented reality, which specifically comprises the following steps: acquiring multi-source fusion data; constructing a visual anchor point of a first-view operation interface based on a visual reference identifier, analyzing and calculating the relative spatial pose between an AR remote controller and a large mechanical arm end effector according to the multi-source fusion data, and constructing the first-view operation interface; when receiving a motion instruction of the first-view operation interface, performing cross-coordinate system operation vector mapping according to a vector redirection algorithm to obtain a physical motion vector; and inputting the redirected physical motion vector into an inverse kinematics model to solve and generate a joint driving vector. The application converts complex joint operations into intuitive end traction, corrects the orientation illusion of cross-coordinate system operations, dynamically compensates for accumulated visual drift, reduces cognitive load, and improves the accuracy and stability of first-view control of the large mechanical arm.

Description

Technical Field

[0001] This invention relates to the field of intelligent control, and in particular to a first-person perspective remote control method for a large robotic arm based on the Internet of Things and augmented reality. Background Technology

[0002] Large robotic arms are core equipment in construction, encompassing concrete placing booms, mine wet spraying trolleys, mining drilling jumbos, and drilling and anchoring robots, among other large robotic arm on-site operation machines. Current end-effector control primarily relies on two modes: "close-range manual traction" and "remote control from a fixed control panel," both of which have significant drawbacks. While the "close-range manual traction" mode is intuitive, it forces support personnel into high-risk work areas, exposing them for extended periods to the robotic arm's swing radius and material splash range, greatly increasing the risk of collisions. Furthermore, the complex environment can lead to low positioning efficiency. While the "fixed console remote control" mode achieves separation of human and machine, it has the following limitations: First, the console's spatial position is fixed, making it difficult to move flexibly with changing construction areas, resulting in poor adaptability; second, this mode heavily relies on panoramic images transmitted from a third-party independent camera. Due to the large amount of image data, video lag is easily generated in a wireless transmission environment, making it difficult to support high-frequency mobile operations; the most critical drawback is that the operator is confined to a single "bystander" perspective far from the work site, lacking in-depth spatial interactive perception, resulting in extremely high cognitive load and making it difficult to meet the demands of modern construction for refined operations.

[0003] Furthermore, existing large robotic arm control systems primarily employ joint-space-based mapping control logic. Since large robotic arm systems typically have multiple degrees of freedom, and the end effector's posture constantly changes with joint movement, the direction of operator commands often differs from the actual trajectory of the end effector. This forces operators to perform intensive coordinate transformations and spatial visualizations in their minds, significantly increasing cognitive load and operational barriers. This non-intuitive control method is not only inefficient but also prone to errors under complex conditions. Therefore, constructing an immersive "first-person perspective" interaction paradigm is crucial. However, existing AR (augmented reality) first-person perspective interaction methods, relying solely on visual features, are prone to cumulative drift in virtual-real alignment and lack safety mechanisms for abnormal operator postures, making it difficult to guarantee operational accuracy and stability.

[0004] Therefore, how to design a first-person perspective remote control method for large robotic arms to avoid the operational deviation caused by traditional first-person perspective enhancement schemes and improve the accuracy and stability of remote control has become an urgent problem to be solved. Summary of the Invention

[0005] Based on this, this invention proposes a first-person perspective remote control method for a large robotic arm based on the Internet of Things and augmented reality. This method acquires multi-source fusion data; constructs visual anchor points for the first-person perspective operation interface based on a visual reference marker; calculates the relative spatial pose between the AR remote controller and the end effector of the large robotic arm based on multi-source fusion data analysis; and constructs the first-person perspective operation interface. When a motion command is received from the first-person perspective operation interface, a cross-coordinate system operation vector mapping is performed according to a vector redirection algorithm, using the first-person perspective view axis as a reference to obtain physical motion vectors. The redirected physical motion vectors are then input into an inverse kinematics model to calculate and generate joint drive vectors. This invention transforms complex joint operations into intuitive end-effector traction, effectively corrects orientation illusions in cross-coordinate system operations, dynamically compensates for accumulated visual drift, thereby significantly reducing cognitive load and improving the accuracy and stability of first-person perspective control of the large robotic arm.

[0006] This invention proposes a first-person perspective remote control method for a large robotic arm based on the Internet of Things and augmented reality, comprising: Acquire multi-source fusion data, which includes joint encoder angle data, base absolute orientation data, UWB tag ranging data, remote controller absolute orientation data, target posture deviation data, and real-time image data; The visual anchor points of the first-view operation interface are constructed based on the visual reference marker. The relative spatial pose between the AR remote controller and the end effector of the large robotic arm is calculated based on multi-source fusion data analysis, and the first-view operation interface is constructed accordingly. When a motion command is received from the first-view operation interface, cross-coordinate system operation vector mapping is performed according to the vector redirection algorithm to obtain the physical motion vector, which is based on the coordinate system of the large robotic arm base. The redirected physical motion vectors are input into the inverse kinematics model to generate joint drive vectors. These joint drive vectors act on the large robotic arm's main controller, and the large robotic arm performs trajectory movements based on these joint drive vectors.

[0007] Furthermore, the step of constructing visual anchor points for the first-view operation interface based on the visual reference marker specifically includes: An active visual reference marker is fixed on the outer periphery of the end effector of a large robotic arm. The visual reference marker is composed of a high-brightness LED array and forms a ring-shaped light-emitting feature that covers 360 degrees continuously and closedly in the circumferential direction of the end effector of the large robotic arm. The ring-shaped luminescence feature is continuously captured to extract the geometric projection feature data of the visual anchor point; Based on the captured visual anchor point data, the rendering position of AR virtual augmentation information is corrected in real time to lock the spatial alignment relationship between the AR virtual guidance image and the end effector of the large robotic arm. When a large robotic arm is detected to be performing a large-span displacement, continuous feedback is provided based on the visual anchor point to dynamically compensate for the accumulated errors in kinematic calculations, correct the expected drift of the first-person view, and synchronize the motion state of the first-person view.

[0008] Furthermore, the step of calculating the relative spatial pose between the AR remote controller and the end effector of the large robotic arm based on multi-source fusion data analysis and constructing a first-view operation interface specifically includes: Construct a physical world coordinate system, a large robotic arm base coordinate system, a large robotic arm end effector coordinate system, an AR remote control coordinate system, and a tool coordinate system. The origin of the tool coordinate system is defined as the mechanical junction between the end of the large robotic arm link and the large robotic arm end effector. The origin of the large robotic arm end effector coordinate system is the geometric center of the visual reference marker. The origin of the tool coordinate system and the origin of the large robotic arm end effector coordinate system maintain a unique and fixed physical offset distance. The three-dimensional absolute attitude data of the large robotic arm base and the AR remote controller are acquired. This absolute attitude data includes absolute azimuth, absolute pitch, and absolute roll angles. The angle differences between these angles are calculated, and a three-dimensional spatial orientation alignment rotation operator is constructed based on these differences. This operator is used to determine the relative rotational mapping relationship between the coordinate system of the large robotic arm base and the coordinate system of the AR remote controller in the entire space. The specific algorithm for the three-dimensional spatial orientation alignment rotation operator is as follows: , , , , , , , in, This represents the rotation matrix for yaw about the Z-axis. This represents the rotation matrix for pitch about the Y-axis. This represents the rotation matrix for rolling about the X-axis. This represents a three-dimensional spatial orientation alignment rotation operator. This represents the difference in yaw angle. This represents the pitch angle difference. This represents the difference in roll angle. and These represent the absolute azimuth angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. and These represent the absolute pitch angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. and These represent the absolute roll angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. The homogeneous pose transformation matrix of the tool coordinate system and the large robotic arm base coordinate system is calculated based on the joint encoder angle data. Then, based on the UWB tag ranging data and target posture deviation data, multi-source fusion correction is performed on the homogeneous pose transformation matrix. The specific algorithm for the homogeneous pose transformation matrix is ​​as follows: , in, This represents the homogeneous pose transformation matrix of the tool coordinate system relative to the large robotic arm base coordinate system. This represents the pose transformation matrix between the base of the large robotic arm and the first joint of the large robotic arm. This represents the pose transformation matrix between the first joint and the second joint of the large robotic arm. The homogeneous pose transformation matrix represents the pose transformation matrix between the nth joint of the large robotic arm and the origin of the tool coordinate system; each of the homogeneous pose transformation matrices synchronously includes the three-dimensional spatial orientation alignment rotation operator of the corresponding node coordinate system; The relative spatial pose between the AR remote controller and the end effector of the large robotic arm is determined based on the three-dimensional spatial orientation alignment rotation operator and the homogeneous pose transformation matrix, and a first-view operation interface is constructed.

[0009] Furthermore, the first-view operation interface specifically includes: The geometric center of the visual reference marker is set as the origin of the motion control virtual coordinate system of the first-view operation interface, and virtual guides in three orthogonal operation directions (horizontal, vertical, and vertical) are rendered according to the origin of the motion control virtual coordinate system. The layout area of ​​the first-view operation interface includes a left area and a right area. The left area includes the current operation status, and the right area includes the relative position coordinate information and the current relative motion speed information between the AR remote controller and the end effector of the large robotic arm. Real-time detection of the horizontal attitude data of the AR remote control; When the detected change in the horizontal attitude data of the AR remote control is less than or equal to the preset safe tilt threshold, virtual visual pitch angle compensation is performed to correct the visual coordinate deviation. When the AR remote controller detects a change in horizontal posture data that exceeds a preset safety tilt threshold, visual tracking and motion control command transmission are interrupted, and a pop-up notification is displayed in the first-person view operation interface.

[0010] Furthermore, the step of inputting the redirected physical motion vectors into the inverse kinematics model to calculate and generate joint drive vectors specifically includes: When an action command is received from the first-person view interface, the desired motion vector in the AR remote control coordinate system is obtained. This desired motion vector is then mapped to the large robotic arm base coordinate system using a 3D spatial orientation alignment rotation operator, resulting in a mapped motion vector. A pure translation transformation matrix is ​​then constructed based on this mapped motion vector. Finally, the target homogeneous pose transformation matrix of the large robotic arm's end effector is calculated through the product of these pure translation transformation matrices. The specific algorithm for the target homogeneous pose transformation matrix is ​​as follows: , = , in, This represents a 4×4 homogeneous translation transformation matrix constructed based on the mapped motion vectors. , and These represent the translation components of the mapped motion vector along the X, Y, and Z axes of the spatial coordinate system, respectively. Represents the mapped motion vector. This represents the homogeneous pose transformation matrix of the current tool coordinate system relative to the large robotic arm base coordinate system. This represents the updated homogeneous pose transformation matrix of the target hocie positional transformation of the end effector of the large robotic arm. Based on the principle of inverse kinematics, a transformation operator product relationship equation containing the coordinate system origin of each joint of the large robotic arm and the tool is established to solve for the driving vector of each joint of the large robotic arm.

[0011] This invention proposes a first-person perspective remote control system for a large robotic arm based on the Internet of Things and augmented reality, comprising: The acquisition module is used to acquire multi-source fusion data, which includes joint encoder angle data, base absolute orientation data, UWB tag ranging data, remote controller absolute orientation data, target posture deviation data, and real-time image data. The first-person view operation module is used to construct the visual anchor points of the first-person view operation interface based on the visual reference marker, calculate the relative spatial pose between the AR remote controller and the end effector of the large robotic arm based on multi-source fusion data analysis, and construct the first-person view operation interface. The vector redirection module is used to perform cross-coordinate system operation vector mapping according to the vector redirection algorithm when receiving action commands from the first-view operation interface in order to obtain physical motion vectors. The physical motion vectors are based on the coordinate system of the large robotic arm base. The inverse motion trajectory generation module is used to input the redirected physical motion vector into the inverse kinematics model to calculate and generate joint drive vectors. The joint drive vectors act on the large robotic arm body controller, and the large robotic arm performs trajectory motion according to the joint drive vectors.

[0012] The present invention also provides a storage medium that stores one or more programs, which, when executed by a processor, implement the first-person perspective remote control method for a large robotic arm based on the Internet of Things and augmented reality as described above.

[0013] The present invention also provides a computer device, the computer device including a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the first-person remote control method for a large robotic arm based on the Internet of Things and augmented reality as described above.

[0014] Compared with the prior art, the present invention has the following three advantages: 1. This invention combines AR first-person perspective with vector redirection algorithms to construct a coordinate mapping relationship of "same orientation, different origins". This design effectively avoids spatial cognitive misalignment caused by hand-eye non-coaxiality in traditional teleoperation, significantly lowers the operational threshold, and makes complex cross-coordinate system control as natural and accurate as direct end-effector control.

[0015] 2. The system utilizes UWB and an electronic compass for global constraints at the macro level, and constructs high-precision visual anchor points by extracting geometric features from an active visual reference marker at the micro level. This fusion mechanism can dynamically compensate for the accumulated errors in kinematic calculations in real time, overcoming the visual drift problem during the large-span movement of the robotic arm and significantly improving the landing accuracy in on-site operations.

[0016] 3. The system monitors the remote control's spatial attitude in real time and performs anti-shake compensation within safe thresholds. If the tilt exceeds the limit, it determines there is a risk of operator fall or equipment overturning, and the system instantly triggers a low-level interlock, interrupting commands and issuing an alarm. This mechanism effectively prevents misoperation accidents caused by abnormal attitudes under harsh working conditions. Attached Figure Description

[0017] Figure 1 This is a flowchart of a first-person perspective remote control method for a large robotic arm based on the Internet of Things and augmented reality, as proposed in the first embodiment of the present invention. Figure 2This is a schematic diagram of the structure of a first-person view remote control system for a large robotic arm based on the Internet of Things and augmented reality, as proposed in the second embodiment of the present invention. Figure 3 This is a flowchart of the overall control logic of the first embodiment of the present invention; Figure 4 This is a diagram illustrating the spatial coordinate mapping and physical structure integration of a large robotic arm in the first embodiment of the present invention.

[0018] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0019] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0020] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0022] Please see Figure 1 The diagram shows a flowchart of a first-person perspective remote control method for a large robotic arm based on the Internet of Things (IoT) and augmented reality (AR), as proposed in the first embodiment of the present invention. This first-person perspective remote control method for a large robotic arm based on IoT and AR includes steps S01 to S04, wherein: Step S01: Acquire multi-source fusion data; It should be noted that this embodiment adopts a distributed architecture design, specifically consisting of two major subsystems: a concrete placing boom system and an AR smart remote control. These two systems establish a data connection through a wireless communication network, forming a complete closed-loop control system. To ensure reliable operation of the system under various complex working conditions, this invention constructs a multi-layered fusion perception scheme: at the spatial positioning layer, it combines UWB precise ranging with electronic compass orientation; at the mechanical control layer, it integrates joint encoder data and kinematic models; and at the visual interaction layer, it fuses real-time video with augmented reality display.

[0023] For the specific collaborative operation mechanism of the above distributed architecture and multi-layered perception scheme, please refer to [link / reference needed]. Figure 3 The overall control logic flowchart is shown below. It should be noted that after the system completes initialization and successfully passes attitude monitoring and target locking, the multi-source data acquisition phase will be triggered. Combined with... Figure 3 The logical node here acquires multi-source fusion data based on key parameters extracted from the aforementioned multi-layered perception scheme. Specifically, this includes: joint encoder angle data provided by the mechanical control layer; UWB tag ranging data, base absolute orientation data, and remote control absolute orientation data provided by the spatial positioning layer; and target posture deviation data and real-time image data provided by the visual interaction layer. The system then performs deep fusion of these three sets of data to calculate relative pose and drive precise movement of the actuator.

[0024] For details on physical hardware deployment and spatial coordinate mapping, please refer to [link / reference]. Figure 4 The diagram shown illustrates the spatial coordinate mapping and physical structure integration of a large robotic arm. The fabric placing machine body, serving as the actuator of the large robotic arm, employs a multi-joint robotic arm design.

[0025] At the global positioning and base end, an electronic compass and UWB base station are fixed at the base of the fabric placing machine to provide an absolute orientation reference and positioning reference node. Based on this, the system establishes the base coordinate system {B} of the large robotic arm under the global reference of the world coordinate system {W}. At the mechanical linkage transmission end, joint encoders are installed at each of the main physical joints of the robotic arm, and joint coordinate systems {J1} to {J4} are established at each hinge point, thereby constructing a positive kinematic space transformation chain for continuous pose transformation from the base to the end effector.

[0026] At the end of the operation, the active visual reference marker is deployed on the periphery of the actuator. Its physical geometric center is precisely defined as the origin of the end coordinate system {E}, while the mechanical rigid junction above it is defined as the tool coordinate system {T}. The origin of the tool coordinate system {T} and the origin of the large robotic arm end actuator coordinate system {E} maintain a unique and fixed physical offset distance.

[0027] In addition, the system introduces an AR smart remote control as a spatial interaction hub, and establishes a remote control coordinate system {C} within it. For example... Figure 4 As shown by the dashed line, the vision module directly locks the end-effector coordinate system {E} from the remote control coordinate system {C}, establishing a first-person perspective operation reference. By fusing the aforementioned physical sensor data and utilizing a homogeneous transformation matrix for rigorous cross-coordinate system spatial calculations, the system accurately transforms the intuitive spatial vectors issued by the operator based on the remote control {C} into motion commands that drive the rotation of physical joints {J1}-{J4}, achieving a high degree of unification between physical mechanical hardware and abstract spatial coordinate transformation.

[0028] The concrete placing boom, acting as the actuator of a large robotic arm, employs a multi-joint design. Its end effector connects to a concrete delivery hose, and the joints are electrically or hydraulically driven to adapt to on-site construction loads and range of motion requirements. Joint encoders are installed at each of the four joints to detect joint rotation angles, providing data support for calculating joint coordinate system parameters. A first electronic compass and a UWB base station are fixed at the boom's base. The electronic compass provides an absolute orientation reference, while the UWB base station serves as a positioning reference node, paired with the UWB tag of the intelligent AR remote control. A first power module is installed inside the boom, featuring overvoltage and overcurrent protection to ensure stable power supply to all components and establishing a wireless connection with the boom's motion control module and the first wireless data transmission module.

[0029] Centered on the main controller, various functional modules are integrated. UWB tags, a second electronic compass, a vision module, an AR video display module, and a second wireless data transmission module are connected to the main controller circuitry. The vision module is linked to the pitch motion control module, allowing for camera angle adjustment via motor drive to ensure rapid capture of the visual reference marker. A second power module, installed inside the remote control handle, features overvoltage and overcurrent protection, providing stable power to the remote control handle and related components. It also establishes a circuit connection with the main controller and the second wireless data transmission module, ensuring reliable operation of all components in the remote control subsystem and enabling remote control, data processing, and AR interaction of the concrete placing boom. The second wireless data transmission module is paired with the first wireless data transmission module of the concrete placing boom system; tested communication latency is less than 50ms, ensuring real-time data transmission.

[0030] In terms of structural composition, the outer surface of the visual reference marker is covered with a set of high-brightness LED light-emitting units. These LED units are evenly distributed along the circumference of the tube's cross-section, forming a continuous closed annular light strip of 360 degrees when powered on.

[0031] In terms of working principle, when the AR smart remote control's vision module captures an image of the end of the fabric laying machine, regardless of the hose's rotation angle, the high-brightness LED ring light strip will consistently display a high-contrast geometric feature on the camera's imaging sensor. When the remote control is at eye level, this projected feature appears as a rectangular light bar; when at a tilt angle, it appears as an elliptical halo with a specific eccentricity. Thanks to the use of a high-brightness, spectrally specific active light source, this visual feature can be effectively separated from complex backgrounds, ensuring the system can stably lock the position of the end hose even in strong light, backlight, cloudy, or nighttime construction scenarios.

[0032] Step S02: Construct visual anchor points for the first-view operation interface based on the visual reference marker, calculate the relative spatial pose between the AR remote controller and the end effector of the large robotic arm based on multi-source fusion data analysis, and construct the first-view operation interface. It should be noted that in this embodiment, an active visual reference marker is fixed on the outer periphery of the end effector of the large robotic arm. The visual reference marker is composed of a high-brightness LED array, and the visual reference marker forms a ring-shaped light-emitting feature that covers 360 degrees continuously and closedly in the circumferential direction of the end effector of the large robotic arm. The ring-shaped luminescence feature is continuously captured to extract the geometric projection feature data of the visual anchor point; Based on the captured visual anchor point data, the rendering position of AR virtual augmentation information is corrected in real time to lock the spatial alignment relationship between the AR virtual guidance image and the end effector of the large robotic arm. When a large robotic arm is detected to be performing a large-span displacement, continuous feedback is provided based on the visual anchor point to dynamically compensate for the accumulated errors in kinematic calculations, correct the expected drift of the first-person view, and synchronize the motion state of the first-person view.

[0033] Construct a physical world coordinate system, a large robotic arm base coordinate system, a large robotic arm end effector coordinate system, an AR remote control coordinate system, and a tool coordinate system. The origin of the tool coordinate system is defined as the mechanical junction between the end of the large robotic arm link and the large robotic arm end effector. The origin of the large robotic arm end effector coordinate system is the geometric center of the visual reference marker. The origin of the tool coordinate system and the origin of the large robotic arm end effector coordinate system maintain a unique and fixed physical offset distance. The three-dimensional absolute attitude data of the large robotic arm base and the AR remote controller are acquired. This absolute attitude data includes absolute azimuth, absolute pitch, and absolute roll angles. The angle differences between these angles are calculated, and a three-dimensional spatial orientation alignment rotation operator is constructed based on these differences. This operator is used to determine the relative rotational mapping relationship between the coordinate system of the large robotic arm base and the coordinate system of the AR remote controller in the entire space. The specific algorithm for the three-dimensional spatial orientation alignment rotation operator is as follows: , , , , , , , in, This represents the rotation matrix for yaw about the Z-axis. This represents the rotation matrix for pitch about the Y-axis. This represents the rotation matrix for rolling about the X-axis. This represents a three-dimensional spatial orientation alignment rotation operator. This represents the difference in yaw angle. This represents the pitch angle difference. This represents the difference in roll angle. and These represent the absolute azimuth angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. and These represent the absolute pitch angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. and These represent the absolute roll angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. The homogeneous pose transformation matrix of the tool coordinate system and the large robotic arm base coordinate system is calculated based on the joint encoder angle data. Then, based on the UWB tag ranging data and target posture deviation data, multi-source fusion correction is performed on the homogeneous pose transformation matrix. The specific algorithm for the homogeneous pose transformation matrix is ​​as follows: , in, This represents the homogeneous pose transformation matrix of the tool coordinate system relative to the large robotic arm base coordinate system. This represents the pose transformation matrix between the base of the large robotic arm and the first joint of the large robotic arm. This represents the pose transformation matrix between the first joint and the second joint of the large robotic arm. The homogeneous pose transformation matrix represents the pose transformation matrix between the nth joint of the large robotic arm and the origin of the tool coordinate system; each of the homogeneous pose transformation matrices synchronously includes the three-dimensional spatial orientation alignment rotation operator of the corresponding node coordinate system; The relative spatial pose between the AR remote controller and the end effector of the large robotic arm is determined based on the three-dimensional spatial orientation alignment rotation operator and the homogeneous pose transformation matrix, and a first-view operation interface is constructed.

[0034] The geometric center of the visual reference marker is set as the origin of the motion control virtual coordinate system of the first-view operation interface, and virtual guides in three orthogonal operation directions (horizontal, vertical, and vertical) are rendered according to the origin of the motion control virtual coordinate system. The layout area of ​​the first-view operation interface includes a left area and a right area. The left area includes the current operation status, and the right area includes the relative position coordinate information and the current relative motion speed information between the AR remote controller and the end effector of the large robotic arm. Real-time detection of the horizontal attitude data of the AR remote control; When the detected change in the horizontal attitude data of the AR remote control is less than or equal to the preset safe tilt threshold, virtual visual pitch angle compensation is performed to correct the visual coordinate deviation. When the AR remote controller detects a change in horizontal posture data that exceeds a preset safety tilt threshold, visual tracking and motion control command transmission are interrupted, and a pop-up notification is displayed in the first-person view operation interface.

[0035] Step S03: When an action command is received from the first-view operation interface, cross-coordinate system operation vector mapping is performed according to the vector redirection algorithm to obtain the physical motion vector; It should be noted that in this embodiment, the physical motion vector is based on the coordinate system of the large robotic arm base, with the first-viewpoint eye axis as the reference.

[0036] Step S04: Input the redirected physical motion vectors into the inverse kinematics model to calculate and generate joint drive vectors; It should be noted that in this embodiment, the joint drive vector acts on the large robotic arm body controller. The large robotic arm performs trajectory movement according to the joint drive vector. When it receives an action command from the first-view operation interface, it obtains the desired motion vector in the AR remote control coordinate system and maps the desired motion vector to the large robotic arm base coordinate system using a three-dimensional spatial orientation alignment rotation operator to obtain the mapped motion vector. Then, it constructs a pure translation transformation matrix based on the mapped motion vector, and then calculates the target homogeneous pose transformation matrix of the large robotic arm end effector by multiplying the pure translation transformation matrix. The specific algorithm for the target homogeneous pose transformation matrix is ​​as follows: , = , in, This represents a 4×4 homogeneous translation transformation matrix constructed based on the mapped motion vectors. , and These represent the translation components of the mapped motion vector along the X, Y, and Z axes of the spatial coordinate system, respectively. Represents the mapped motion vector. This represents the homogeneous pose transformation matrix of the current tool coordinate system relative to the large robotic arm base coordinate system. This represents the updated homogeneous pose transformation matrix of the target hocie positional transformation of the end effector of the large robotic arm. Based on the principle of inverse kinematics, a transformation operator product relationship equation containing the coordinate system origin of each joint of the large robotic arm and the tool is established to solve for the driving vector of each joint of the large robotic arm.

[0037] Please see Figure 2 The diagram shows a first-person remote control system for a large robotic arm based on the Internet of Things and augmented reality, as proposed in the second embodiment of the present invention. The system includes: A first-person remote control system for a large robotic arm based on the Internet of Things and augmented reality, characterized in that it includes: The acquisition module 10 is used to acquire multi-source fusion data, which includes joint encoder angle data, base absolute orientation data, UWB tag ranging data, remote controller absolute orientation data, target posture deviation data, and real-time image data. The first-view operation module 20 is used to construct the visual anchor points of the first-view operation interface based on the visual reference marker, calculate the relative spatial pose between the AR remote controller and the end effector of the large robotic arm based on multi-source fusion data analysis, and construct the first-view operation interface. The vector redirection module 30 is used to perform cross-coordinate system operation vector mapping according to the vector redirection algorithm to obtain physical motion vectors when receiving action commands from the first-view operation interface. The physical motion vectors are based on the coordinate system of the large robotic arm base. The inverse motion trajectory generation module 40 is used to input the redirected physical motion vector into the inverse kinematics model to calculate and generate joint drive vectors. The joint drive vectors act on the large robotic arm body controller, and the large robotic arm performs trajectory motion according to the joint drive vectors.

[0038] The present invention also proposes a computer storage medium storing one or more programs that, when executed by a processor, implement the above-described first-person remote control method for a large robotic arm based on the Internet of Things and augmented reality.

[0039] The present invention also proposes a computer device, including a memory and a processor, wherein the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to realize the above-mentioned first-person perspective remote control method for a large robotic arm based on the Internet of Things and augmented reality.

[0040] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0041] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0042] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0043] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0044] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A first-person perspective remote control method for a large robotic arm based on the Internet of Things and augmented reality, characterized in that, include: Acquire multi-source fusion data, which includes joint encoder angle data, base absolute orientation data, UWB tag ranging data, remote controller absolute orientation data, target posture deviation data, and real-time image data; The visual anchor points of the first-view operation interface are constructed based on the visual reference marker. The relative spatial pose between the AR remote controller and the end effector of the large robotic arm is calculated based on multi-source fusion data analysis, and the first-view operation interface is constructed accordingly. The step of calculating the relative spatial pose between the AR remote controller and the end effector of the large robotic arm based on multi-source fusion data analysis and constructing a first-view operation interface specifically includes: Construct a physical world coordinate system, a large robotic arm base coordinate system, a large robotic arm end effector coordinate system, an AR remote control coordinate system, and a tool coordinate system. The origin of the tool coordinate system is defined as the mechanical junction between the end of the large robotic arm link and the large robotic arm end effector. The origin of the large robotic arm end effector coordinate system is the geometric center of the visual reference marker. The origin of the tool coordinate system and the origin of the large robotic arm end effector coordinate system maintain a unique and fixed physical offset distance. The three-dimensional absolute attitude data of the large robotic arm base and the AR remote controller are acquired. This absolute attitude data includes absolute azimuth, absolute pitch, and absolute roll angles. The angle differences between these angles are calculated, and a three-dimensional spatial orientation alignment rotation operator is constructed based on these differences. This operator is used to determine the relative rotational mapping relationship between the coordinate system of the large robotic arm base and the coordinate system of the AR remote controller in the entire space. The specific algorithm for the three-dimensional spatial orientation alignment rotation operator is as follows: ， ， ， ， ， ， ， in, This represents the rotation matrix for yaw about the Z-axis. This represents the rotation matrix for pitch about the Y-axis. This represents the rotation matrix for rolling about the X-axis. This represents a three-dimensional spatial orientation alignment rotation operator. This represents the difference in yaw angle. This represents the pitch angle difference. This represents the difference in roll angle. and These represent the absolute azimuth angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. and These represent the absolute pitch angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. and These represent the absolute roll angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. The homogeneous pose transformation matrix of the tool coordinate system and the large robotic arm base coordinate system is calculated based on the joint encoder angle data. Then, based on the UWB tag ranging data and target posture deviation data, multi-source fusion correction is performed on the homogeneous pose transformation matrix. The specific algorithm for the homogeneous pose transformation matrix is ​​as follows: ， in, This represents the homogeneous pose transformation matrix of the tool coordinate system relative to the large robotic arm base coordinate system. This represents the pose transformation matrix between the base of the large robotic arm and the first joint of the large robotic arm. This represents the pose transformation matrix between the first joint and the second joint of the large robotic arm. The homogeneous pose transformation matrix represents the pose transformation matrix between the nth joint of the large robotic arm and the origin of the tool coordinate system; each of the homogeneous pose transformation matrices synchronously includes the three-dimensional spatial orientation alignment rotation operator of the corresponding node coordinate system; The relative spatial pose between the AR remote controller and the end effector of the large robotic arm is determined based on the three-dimensional spatial orientation alignment rotation operator and the homogeneous pose transformation matrix, and a first-view operation interface is constructed. When a motion command is received from the first-view operation interface, a cross-coordinate system operation vector mapping is performed according to the vector redirection algorithm to obtain the physical motion vector, which is based on the coordinate system of the large robotic arm base. The redirected physical motion vectors are input into the inverse kinematics model to generate joint drive vectors. These joint drive vectors act on the large robotic arm's main controller, and the large robotic arm performs trajectory movement according to the joint drive vectors. The step of inputting the redirected physical motion vectors into the inverse kinematics model to calculate and generate joint drive vectors specifically includes: When an action command is received from the first-person view interface, the desired motion vector in the AR remote control coordinate system is obtained. This desired motion vector is then mapped to the large robotic arm base coordinate system using a 3D spatial orientation alignment rotation operator, resulting in a mapped motion vector. A pure translation transformation matrix is ​​then constructed based on this mapped motion vector. Finally, the target homogeneous pose transformation matrix of the large robotic arm's end effector is calculated through the product of these pure translation transformation matrices. The specific algorithm for the target homogeneous pose transformation matrix is ​​as follows: ， = ， in, This represents a 4×4 homogeneous translation transformation matrix constructed based on the mapped motion vectors. , and These represent the translation components of the mapped motion vector along the X, Y, and Z axes of the spatial coordinate system, respectively. Represents the mapped motion vector. This represents the homogeneous pose transformation matrix of the current tool coordinate system relative to the large robotic arm base coordinate system. This represents the updated homogeneous pose transformation matrix of the target hocie positional transformation of the end effector of the large robotic arm. Based on the principle of inverse kinematics, a transformation operator product relationship equation containing the coordinate system origin of each joint of the large robotic arm and the tool is established to solve for the driving vector of each joint of the large robotic arm.

2. The first-person perspective remote control method for a large robotic arm based on the Internet of Things and augmented reality as described in claim 1, characterized in that, The step of constructing visual anchor points for the first-view operation interface based on the visual reference marker specifically includes: An active visual reference marker is fixed on the outer periphery of the end effector of a large robotic arm. The visual reference marker is composed of a high-brightness LED array and forms a ring-shaped light-emitting feature that covers 360 degrees continuously and closedly in the circumferential direction of the end effector of the large robotic arm. The ring-shaped luminescence feature is continuously captured to extract the geometric projection feature data of the visual anchor point; Based on the captured visual anchor point data, the rendering position of AR virtual augmentation information is corrected in real time to lock the spatial alignment relationship between the AR virtual guidance image and the end effector of the large robotic arm. When a large robotic arm is detected to be performing a large-span displacement, continuous feedback is provided based on the visual anchor point to dynamically compensate for the accumulated errors in kinematic calculations, correct the expected drift of the first-person view, and synchronize the motion state of the first-person view.

3. The first-person perspective remote control method for a large robotic arm based on the Internet of Things and augmented reality as described in claim 1, characterized in that, The first-view operation interface specifically includes: The geometric center of the visual reference marker is set as the origin of the motion control virtual coordinate system of the first-view operation interface, and virtual guides in three orthogonal operation directions (horizontal, vertical, and vertical) are rendered according to the origin of the motion control virtual coordinate system. The layout of the first-view operation interface includes a status indication area and a motion parameter display area. The status indication area includes the current operation status, and the motion parameter display area includes the relative position coordinate information and the current relative motion speed information between the AR remote controller and the end effector of the large robotic arm. Real-time detection of the horizontal attitude data of the AR remote control; When the detected change in the horizontal attitude data of the AR remote controller is less than or equal to the preset safe tilt threshold, virtual visual pitch angle compensation is performed to correct the visual coordinate deviation. When the AR remote controller's horizontal attitude data changes more than the preset safety tilt threshold, visual tracking and motion control command transmission are interrupted, and a pop-up prompt is displayed in the first-person view operation interface.

4. A first-person remote control system for a large robotic arm based on the Internet of Things and augmented reality, characterized in that, include: The acquisition module is used to acquire multi-source fusion data, which includes joint encoder angle data, base absolute orientation data, UWB tag ranging data, remote controller absolute orientation data, target posture deviation data, and real-time image data. The first-person view operation module is used to construct the visual anchor points of the first-person view operation interface based on the visual reference marker, calculate the relative spatial pose between the AR remote controller and the end effector of the large robotic arm based on multi-source fusion data analysis, and construct the first-person view operation interface. The step of calculating the relative spatial pose between the AR remote controller and the end effector of the large robotic arm based on multi-source fusion data analysis and constructing a first-view operation interface specifically includes: Construct a physical world coordinate system, a large robotic arm base coordinate system, a large robotic arm end effector coordinate system, an AR remote control coordinate system, and a tool coordinate system. The origin of the tool coordinate system is defined as the mechanical junction between the end of the large robotic arm link and the large robotic arm end effector. The origin of the large robotic arm end effector coordinate system is the geometric center of the visual reference marker. The origin of the tool coordinate system and the origin of the large robotic arm end effector coordinate system maintain a unique and fixed physical offset distance. The three-dimensional absolute attitude data of the large robotic arm base and the AR remote controller are acquired. This absolute attitude data includes absolute azimuth, absolute pitch, and absolute roll angles. The angle differences between these angles are calculated, and a three-dimensional spatial orientation alignment rotation operator is constructed based on these differences. This operator is used to determine the relative rotational mapping relationship between the coordinate system of the large robotic arm base and the coordinate system of the AR remote controller in the entire space. The specific algorithm for the three-dimensional spatial orientation alignment rotation operator is as follows: ， ， ， ， ， ， ， in, This represents the rotation matrix for yaw about the Z-axis. This represents the rotation matrix for pitch about the Y-axis. This represents the rotation matrix for rolling about the X-axis. This represents a three-dimensional spatial orientation alignment rotation operator. This represents the difference in yaw angle. This represents the pitch angle difference. This represents the difference in roll angle. and These represent the absolute azimuth angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. and These represent the absolute pitch angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. and These represent the absolute roll angles of the large robotic arm base coordinate system and the AR remote control coordinate system, respectively. The homogeneous pose transformation matrix of the tool coordinate system and the large robotic arm base coordinate system is calculated based on the joint encoder angle data. Then, based on the UWB tag ranging data and target posture deviation data, multi-source fusion correction is performed on the homogeneous pose transformation matrix. The specific algorithm for the homogeneous pose transformation matrix is ​​as follows: ， in, This represents the homogeneous pose transformation matrix of the tool coordinate system relative to the large robotic arm base coordinate system. This represents the pose transformation matrix between the base of the large robotic arm and the first joint of the large robotic arm. This represents the pose transformation matrix between the first joint and the second joint of the large robotic arm. The homogeneous pose transformation matrix represents the pose transformation matrix between the nth joint of the large robotic arm and the origin of the tool coordinate system; each of the homogeneous pose transformation matrices synchronously includes the three-dimensional spatial orientation alignment rotation operator of the corresponding node coordinate system; The relative spatial pose between the AR remote controller and the end effector of the large robotic arm is determined based on the three-dimensional spatial orientation alignment rotation operator and the homogeneous pose transformation matrix, and a first-view operation interface is constructed. The vector redirection module is used to perform cross-coordinate system operation vector mapping according to the vector redirection algorithm when receiving action commands from the first-view operation interface in order to obtain physical motion vectors. The physical motion vectors are based on the coordinate system of the large robotic arm base. The inverse motion trajectory generation module is used to input the redirected physical motion vector into the inverse kinematics model to calculate and generate joint drive vectors. The joint drive vectors act on the large robotic arm body controller, and the large robotic arm performs trajectory movement according to the joint drive vectors. The step of inputting the redirected physical motion vectors into the inverse kinematics model to calculate and generate joint drive vectors specifically includes: When an action command is received from the first-person view interface, the desired motion vector in the AR remote control coordinate system is obtained. This desired motion vector is then mapped to the large robotic arm base coordinate system using a 3D spatial orientation alignment rotation operator, resulting in a mapped motion vector. A pure translation transformation matrix is ​​then constructed based on this mapped motion vector. Finally, the target homogeneous pose transformation matrix of the large robotic arm's end effector is calculated through the product of these pure translation transformation matrices. The specific algorithm for the target homogeneous pose transformation matrix is ​​as follows: ， = ， in, This represents a 4×4 homogeneous translation transformation matrix constructed based on the mapped motion vectors. , and These represent the translation components of the mapped motion vector along the X, Y, and Z axes of the spatial coordinate system, respectively. Represents the mapped motion vector. This represents the homogeneous pose transformation matrix of the current tool coordinate system relative to the large robotic arm base coordinate system. This represents the updated homogeneous pose transformation matrix of the target hocie positional transformation of the end effector of the large robotic arm. Based on the principle of inverse kinematics, a transformation operator product relationship equation containing the coordinate system origin of each joint of the large robotic arm and the tool is established to solve for the driving vector of each joint of the large robotic arm.

5. A storage medium, characterized in that, The storage medium stores one or more programs that, when executed by a processor, implement the first-person remote control method for a large robotic arm based on the Internet of Things and augmented reality as described in any one of claims 1-3.

6. A computer device, characterized in that, The computer device includes a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the first-person remote control method for a large robotic arm based on the Internet of Things and augmented reality as described in any one of claims 1-3.

Images