An augmented reality based first-person view teleoperation concrete construction system

Abstract

The application relates to a first-view-angle remote operation concrete construction system based on augmented reality and belongs to the fields of robot remote operation and AR technology. The system comprises a concrete distribution machine subsystem, an integrated annular light source target, an electronic compass and a motion control module; an augmented remote operation subsystem, an integrated first-view-angle motion solution module, a remote operation handle and AR glasses. The relative positions of the tool coordinate system and the camera coordinate system are solved by fusing the annular light source target visual positioning, the global absolute orientation data of the double electronic compass and the joint angle data; meanwhile, the AR glasses render the coordinate system and the state information with the center of the annular light source target as the origin; the orientation alignment relationship between the two coordinate systems is established based on the global absolute orientation data of the double electronic compass; and then the first-view-angle operation instruction is mapped into the target position in the base coordinate system and is driven to be executed. The first-view-angle solution module, the AR and the remote operation are used to realize the intuitive space mapping remote operation, so that the operation safety is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of robot teleoperation and augmented reality (AR) interaction technology, and in particular to a first-person perspective teleoperated concrete construction system based on augmented reality. Background Technology

[0002] Concrete placing booms are key equipment in construction. Currently, the end-point control in this field mainly relies on the traditional mode of "control console operation + human assistance": the operator makes judgments remotely using a two-dimensional control panel and limited visual feedback, while another assistant is required to go into the hazardous work area to provide feedback via walkie-talkie or directly perform contact-based manual traction.

[0003] The traditional approach exposes support personnel to the hazardous environment of heavy concrete placing booms and flowing concrete at close range, facing threats such as collisions and splashes, resulting in poor safety. CN118547891B discloses a method to mitigate collision risks by adding a flexible structure to the end effector, but this is only a passive protection and does not change the inherent dangers of contact control; at the same time, the approach also poses a risk of disconnect between operation and perception, as the operator cannot intuitively obtain end effector depth and attitude information.

[0004] In the more general field of robot teleoperation technology, existing methods also have inherent limitations. Traditional methods mainly rely on a fixed camera's third-party observation perspective and joystick-based joint mapping control. With the former, it is difficult for the operator to perceive the depth and posture of the end effector in three-dimensional space from a two-dimensional image, and insufficient spatial feedback leads to difficulties in fine manipulation; with the latter, there is no intuitive spatial correspondence between the operation command and the actual movement direction of the end effector, requiring the operator to perform coordinate transformations through imagination, resulting in a high cognitive load and a certain rate of error.

[0005] In summary, there is an urgent need in this field for an innovative control system solution that can fundamentally transform the existing interaction paradigm and enable contactless, intuitive, and safe remote operation of concrete placing booms. Summary of the Invention

[0006] To address the shortcomings of existing technologies, such as the inability to physically isolate operators from hazardous environments and the lack of intuitive spatial mapping between control commands and end-effector movements, this invention provides an augmented reality-based first-person perspective remote-operated concrete construction system. The technical solution of this invention is as follows: A first-view enhanced teleoperation system for concrete construction includes a concrete placing machine subsystem and an enhanced teleoperation subsystem.

[0007] Furthermore, the concrete placing boom subsystem includes a power module, a placing boom body, a motion control module, a data acquisition module, and a wireless data transmission module. The power module supplies power to the concrete placing boom subsystem. The placing boom body performs the actual work tasks. The motion control module, connected to the placing boom body, adjusts the joint angles of the robotic arm and the posture of the placing tube according to drive signals. The wireless data transmission module transmits data between the two subsystems. The enhanced remote operation subsystem includes a remote control handle, a safety helmet, and AR glasses. The safety helmet integrates sensors and a processing module. The AR glasses enable visual computing and 3D rendering.

[0008] Furthermore, the data acquisition module includes: an electronic compass, a joint encoder, and a ring light source target. The electronic compass provides global absolute orientation data for the fabric laying machine base; the ring light source target is mounted on the end hose for visual positioning marking.

[0009] Furthermore, the remote control handle includes: a second power module, directional buttons, a safety button, and a second wireless data transmission module; the second power module is used to power the remote control handle; the directional buttons are used to output remote control commands on the XYZ axes in the first viewpoint coordinate system; the safety button is used to ensure the output of any command with the operator's permission; and the second wireless data transmission module is used to send the remote control commands to the safety helmet.

[0010] Furthermore, the safety helmet includes: a power module three, a power supply module, a first-view motion calculation module, a wireless data transmission module three, and a wired data transmission module one for the safety helmet; the power module three is used to supply power to the integrated components inside the safety helmet; the power supply module is used to transmit the power from the power module two to the AR glasses; the first-view motion calculation module is used to convert the relative pose between coordinate systems into the target pose in the base coordinate system, and then calculate the drive signals of each joint and the attitude control signals of the end effector based on the target pose; the wireless data transmission module three is used to receive the status information of the concrete placing machine subsystem and send the calculated motion commands to the concrete placing machine subsystem; the wired data transmission module one for the safety helmet is used to send rendering information to the AR glasses.

[0011] Furthermore, the AR glasses include: a miniature camera, an optical engine, a second electronic compass, an AR data processing module, and a second wired data transmission module for the AR glasses; the miniature camera is used to capture environmental images; the optical engine is used to display augmented reality information; the second electronic compass is used to provide global absolute orientation data for the AR glasses; the AR data processing module is used to receive the relative pose results and system status data calculated by the first-view motion calculation module, and generate image rendering frames containing augmented reality information such as target coordinate system, text labels, and status icons; the second wired data transmission module for the AR glasses is used to receive rendering information from the safety helmet.

[0012] Specifically, the augmented reality solution of the AR glasses uses a waveguide scheme. The optical engine includes a display module, a waveguide, and a coupler. The display module generates the original virtual image. The waveguide uses the principle of total internal reflection to send the image to the human eye. The coupler guides light into and out of the waveguide to complete the input and output of the image.

[0013] Furthermore, the annular light source target is composed of a series of short, closely arranged LED light sources. These light sources are evenly distributed along the circumference formed by two symmetrical support screws, and their high brightness and good continuity ensure that a sufficient number of feature light points can be observed from any angle from the camera's perspective. The lower surface of the annular light source target must be perpendicular to the axis of the flexible tube. This is a geometric prerequisite for the subsequent visual algorithm to accurately calculate the z-axis of the tool coordinate system. The target as a whole is stably positioned at a certain height above the discharge port, thereby preventing concrete splashes from below from contaminating the light source.

[0014] The core data streams of the system include motion control stream, AR rendering stream, and operation state stream.

[0015] Furthermore, the motion control flow process is as follows: S1: The wireless data transmission module three continuously receives system status feedback from the concrete placing machine subsystem through the wireless data transmission module one, including global absolute orientation data provided by the electronic compass of the placing machine base and joint angle data provided by the encoders of each joint; S2: The wireless data transmission module three receives motion commands from the remote control handle through the wireless data transmission module two; at the same time, it receives the ring light source target image captured by the miniature camera transmitted from the AR glasses and the global absolute orientation data of the AR glasses provided by the built-in electronic compass through the wired data transmission module one of the safety helmet. S3: The first-person perspective motion calculation module collects the above multi-source information and performs fusion processing: based on the ring light source target image, camera intrinsic parameters, and the known size of the ring light source target, the relative pose of the tool coordinate system with respect to the camera coordinate system is calculated; based on joint angle data and global absolute orientation data of the base, the theoretical pose of the tool coordinate system with respect to the base coordinate system is calculated through the robot's forward kinematics; and based on the two global absolute orientation data, the orientation alignment relationship between the AR glasses coordinate system and the cloth machine base coordinate system is established. S4: Wireless data transmission module three collects the remote operation commands transmitted by wireless data transmission module two. The first-view motion calculation module maps the remote operation commands into the target pose in the tool coordinate system under the coordinate system of the fabric laying machine base, based on the orientation alignment relationship. Then, it calculates the control signals, including joint drive signals and attitude control signals, through inverse kinematics. S5: The control command is sent to the wireless data transmission module 1 via the wireless data transmission module 3, and the motion control module drives the robotic arm and end effector to move according to the control signal.

[0016] Furthermore, in the AR rendering stream transmission process: the first-view motion calculation module sends the calculated pose of the tool coordinate system relative to the AR glasses coordinate system and the real-time system status data to the AR glasses through the wired data transmission module of the safety helmet 3; the AR data processing module inside the AR glasses receives the above data and generates a rendering frame including a virtual coordinate system with the center of the ring light source target as the origin and a status information panel; the rendering frame is sent to the optical engine, which converts the image information into light rays, and combines them with the light rays of the real scene through waveguides and couplers, finally forming an augmented reality fused image on the display module.

[0017] Furthermore, the operation status stream transmission process is as follows: the first-view motion calculation module determines the current system operation status in real time based on a preset safety distance; the determination result is used as an operation status command and sent to the AR glasses for display via the wired data transmission module of the safety helmet; the AR glasses render and display the received operation status information to the operator in real time; the operator controls the opening and closing of the safety button according to the operation status information, and closes the safety button when the status information is "ready" or "in motion"; when the status information is "emergency stop" or "collaborative calibration", the safety button is opened, and the motion control module also stops working due to physical disconnection, thereby realizing closed-loop management of interactive permissions.

[0018] The motion control flow, AR rendering flow, and operation state flow are executed in parallel and continuously in a loop, ensuring a precise operation loop, a status display loop, and a safe interaction loop, thereby jointly achieving a smooth, fast, and safe teleoperation experience for the system.

[0019] The coordinate system transformation relationship involves the following core coordinate systems: the fabric placing machine base coordinate system {B}, the coordinate system of each joint of the robotic arm {J_i}, the virtual coordinate system {D} with the center of the ring light source target as the origin, and the camera coordinate system {C}.

[0020] The global absolute orientation angle α1 of the fabric machine base and the global absolute orientation angle α2 of the AR glasses are obtained based on the data from the electronic compass.

[0021] The translation vector from joint i-1 to joint i is used To represent, rotation matrix is ​​used To represent the homogeneous transformation matrix, we use... To express.

[0022] S1: Based on the angle parameters of each joint of the robotic arm transmitted by the concrete placing boom subsystem, and the parameters of each robotic arm, the translation vector between the coordinate systems of each joint can be obtained. ; S2: This allows us to obtain the translation vector of the tool coordinate system relative to the base coordinate system {B}. ; S3: Based on the global orientation angles of the fabric laying machine base and the AR glasses, the rotation matrices of the fabric laying machine base coordinate system {B} and the camera coordinate system {C} can be calculated. ; S4: The orientation of the tool coordinate system {T} is determined by the camera coordinate system {C}, thereby calculating the rotation matrix between the tool coordinate system {T} and the fabric placing machine base coordinate system {B}. ; S5: By translating vectors and rotation matrix Then, the homogeneous transformation matrix between the two coordinate systems can be calculated. :

[0023] Furthermore, in the first-view motion calculation module, the visual detection and pose calculation process of the miniature camera for the ring-shaped light source target is as follows: S1: The miniature camera continuously acquires scene images containing the ring light source target, and preprocesses the images, including color space-based region segmentation and edge enhancement processing, to highlight the contour features of the target marker; S2: Perform contour extraction and analysis on the preprocessed image. Through contour tracking and shape matching algorithms, identify and separate the complete image contours of the two ring light source targets. Based on the contours, calculate the geometric features of the rectangular contours of each ring light source target, including the center image coordinates, width, and pixel distance between the centers of the two rectangles. S3: Based on the fixed left-right positional relationship of the two ring-shaped light source targets, establish the correspondence between image coordinates and physical coordinates. By determining the horizontal orientation of the two identified rectangular contours in the image, they are distinguished as the left and right light strips, thus completing the matching; S4: Based on the rectangular contour geometric features calculated in step S2, the previously calibrated camera intrinsic parameters, and the correspondence established in step S3, the rotation matrix and translation vector from the target coordinate system to the camera coordinate system are calculated using the PnP algorithm. The known fixed physical distance between the two ring-shaped light source targets provides an absolute scale reference for monocular vision, enabling stable and reliable calculation of the tool coordinate system's three-dimensional coordinates and three-axis orientation in the camera coordinate system, thus completing the pose calculation of the tool coordinate system relative to the camera coordinate system.

[0024] Furthermore, the core of the rendering process for the AR glasses lies in the optical engine's processing and synthesis of real-time data: S1: The wired data transmission module 2 collects the end pose and state data calculated by the first-view motion calculation module. The AR data processing module generates an image data frame containing target coordinate system, text, and icon graphic elements based on the data. S2: The image data is sent to the optical engine of the AR glasses, and the display module of the optical engine generates original light carrying virtual image information based on the image data; S3: The original light beam enters the waveguide through the input coupler, and is propagated and expanded inside the waveguide through total internal reflection; S4: The light beam, after being guided and expanded by the waveguide, leaves the waveguide through the output coupler and finally combines with the real ambient light passing through the waveguide, and is projected together into the operator's pupil, thus forming a stable and accurate superposition of augmented reality information in the real field of vision.

[0025] Furthermore, the AR glasses are configured such that, after the first-view motion calculation module calculates the pose relationship, the AR glasses, based on the calculated pose relationship between the AR glasses and the ring light source target, render a three-dimensional coordinate axis with directional arrows, with the center of the ring light source target as the origin, to provide the operator with an intuitive spatial motion reference; simultaneously, in the side area of ​​the field of view, the real-time status parameters of the system are overlaid and displayed in the form of text or graphic panels, including the relative pose of the end effector with respect to the AR glasses coordinate system, real-time motion speed, and system operation status such as "ready", "in motion", "emergency stop" and "cooperative calibration". Beneficial effects

[0026] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: 1. It realizes intuitive spatial mapping teleoperation. Through AR glasses, a virtual coordinate system fixed to the ring light source target is presented in the operator's field of vision. The direction seen is the control direction. It fundamentally solves the problem of unintuitive spatial mapping between commands and end actions in traditional teleoperation, and significantly improves control intuition and positioning accuracy.

[0027] 2. A non-contact remote safe operation mode has been established, allowing operators to obtain an immersive first-person perspective through AR glasses in a safe area. No personnel are required to enter the dangerous area to assist in the operation, achieving physical isolation between personnel and dangerous sources such as heavy robotic arms and flowing concrete, which greatly reduces safety risks.

[0028] 3. By integrating information from multiple sensors to achieve stable closed-loop control, the system combines visual target positioning, dual electronic compass orientation perception, and joint encoder data to achieve high-precision end-effector pose calculation and status feedback. Information is also intuitively overlaid through an AR interface, enhancing the anti-interference capability and control stability in complex construction environments. Attached Figure Description

[0029] Figure 1 This is a system configuration diagram of the present invention; Figure 2 This is a system flowchart of the present invention; Figure 3 This is a data transmission diagram of the present invention; Figure 4 Diagram showing the installation location of the ring light source target; Figure 5 This is a schematic diagram of the coordinate transformation relationship of the present invention, wherein {B} is the base coordinate system, {S} is the world coordinate system, {J1} is the first joint coordinate system, {J2} is the second joint coordinate system, {J3} is the third joint coordinate system, {J4} is the fourth joint coordinate system, {T} is the tool coordinate system, {G} is the target coordinate system, {A} is the target coordinate system, {C} is the camera coordinate system, and {D} is the virtual coordinate system; Figure 6 This is a system operation logic flowchart of Embodiment 1 of the present invention; Figure 7 This is a system operation logic flowchart of Embodiment 2 of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and key points of this invention clearer, the invention will be further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the invention.

[0031] In this invention, the data acquisition module of the system mainly includes three types of functional units: a sensor unit for sensing environmental and status information, a marker unit as a visual positioning reference, and a display output unit for information presentation, and is not limited to the sensor unit.

[0032] like Figure 1 The diagram shown illustrates the overall structure of the augmented reality-based first-person perspective remote-operated concrete construction system described in this invention. The system consists of two main parts: a concrete placing machine subsystem and an augmented remote operation subsystem, connected by a data transmission module.

[0033] The concrete placing boom subsystem is deployed at the construction site, and its core components include: Concrete placing boom body 1: The main structure of the robot that performs the concrete placing task, including: a motion control module: connected to the concrete placing boom body 1, used to adjust the joint angles of the robotic arm and the posture of the end-effector tube according to the received drive signals; a data acquisition module: integrating an electronic compass to provide global absolute orientation data of the concrete placing boom base, joint encoders to collect the joint angles, and a ring light source target set on the end-effector tube as a visual positioning marker; a wireless data transmission module 1: used for bidirectional data communication with the enhanced teleoperation subsystem; and a power supply module 1: used to supply power to the entire concrete placing boom subsystem.

[0034] The enhanced teleoperation subsystem is worn and operated by the operator, and its core components include: Remote control handle 2: includes directional buttons, safety buttons, and wireless data transmission module 2, used to send remote operation commands defined in the first-view coordinate system to the system; Safety helmet 3: internally integrates a first-view motion calculation module, a wireless data transmission module 3, and a wired data transmission module 1 for the safety helmet 3; AR glasses 4: includes a miniature camera for capturing environmental images, an optical engine for displaying augmented reality information, an electronic compass 2 for providing global absolute orientation data for the AR glasses 4, and an AR data processing module.

[0035] like Figure 2 The diagram shown illustrates the core operation flow of the system of this invention. The system's workflow mainly includes the following steps: S1: Start the concrete placing machine subsystem. The motion control module controls the robotic arm to extend to the ready position, and at the same time activates the electronic compass, ring light source target, joint encoder and wireless data transmission module.

[0036] S2: The operator wears the enhanced remote operation subsystem. The miniature camera on the AR glasses 4 begins to capture an image containing a target with a ring light source; simultaneously, the safety helmet 3 begins to receive base global absolute orientation data and joint angle data from the wireless data transmission module 1 of the concrete placing boom subsystem via the wireless data transmission module 3.

[0037] S3: The first-person perspective motion calculation module synchronously processes visual images, dual electronic compass orientation data, and joint angle data to calculate the pose of the tool coordinate system at the end of the robotic arm relative to the camera coordinate system, and establishes the orientation alignment relationship between the camera coordinate system and the cloth laying machine base coordinate system.

[0038] S4: The operator issues motion commands via the remote control handle 2. Based on the orientation alignment relationship, the first-view motion calculation module maps the commands in the first-view coordinate system to the target pose in the tool coordinate system under the base coordinate system, and then solves the drive signals of each joint and the end effector attitude control signals through robot inverse kinematics.

[0039] S5: The AR glasses 4 render a virtual 3D coordinate system in real time in the operator's field of vision based on the calculated pose of the tool coordinate system relative to the camera coordinate system, with the center of the ring light source target as the origin, and overlay system status information.

[0040] S6: Control commands are sent to the motion control module of the concrete placing boom subsystem via wireless network to drive the robotic arm and end effector to move; at the same time, the robot status is fed back in real time, and the calculation and display are updated to form a closed loop of "perception-calculation-control-feedback".

[0041] like Figure 3 The diagram shows the main data transmission paths within the system of this invention. After system startup, wireless data transmission module one transmits the electronic compass data and joint encoder data of the fabric laying machine to safety helmet 3 via wireless transmission module three. Then, based on the remote operation commands sent by wireless data transmission module two, it generates robotic arm joint drive signals and fabric tube attitude control signals, which are then transmitted to wireless data transmission module one. Wired data transmission module one of safety helmet 3 receives image information and electronic compass data from wired data transmission module two of AR glasses 4 and forwards them to wireless data transmission module three. Based on the first-view motion calculation module within the safety helmet, it outputs operation status information, which is then transmitted to wired data transmission module two of AR glasses 4 via wireless data transmission module three. Simultaneously, it sends the rendering information calculated by the AR processing module to wired data transmission module two of AR glasses 4. Afterward, the AR glasses display content on the display module based on the rendering information and operation status information.

[0042] like Figure 4 The diagram illustrates the coordinate system transformation relationships of this invention. The coordinate systems include the world coordinate system {S}, the fabric placing machine base coordinate system {B}, and the coordinate systems of each joint of the robotic arm {...}. The coordinate system includes the target coordinate system {A}, the tool coordinate system {T} at the end of the robotic arm, the camera coordinate system {C}, and the virtual coordinate system {D} with the center of the ring light source target as the origin (i is the joint number, i≥1).

[0043] like Figure 5 The diagram shown is a schematic representation of the structure, installation position, and dimensional relationship of the ring light source target in the system of the present invention.

[0044] The ring-shaped light source target 5 is the core component for the system to achieve high-precision visual positioning. This target consists of a series of high-brightness light sources arranged uniformly and closely along the circumference, forming a complete luminous ring. To securely mount it to the periphery of the flexible concrete placing hose 7 at the end, an openable and closable ring-shaped mounting bracket is used for support and fixation.

[0045] The bracket 6 adopts a split design, specifically consisting of two symmetrical semi-circular shells connected by screws. This design allows the entire target assembly to be directly installed or disassembled on-site without moving or disassembling the hose 7 itself, greatly improving the convenience of installation and maintenance. During installation, the inner surface of the target 5 is firmly fixed to the outer periphery of the mounting bracket 6.

[0046] During installation, two key spatial relationships must be ensured: first, the lower surface of the target must be perpendicular to the axis of the hose to provide a stable, directly facing observation plane for the vision system; second, the target must be suspended at a certain distance above the discharge port to avoid concrete splashing and ensure a good field of view.

[0047] like Figure 6 The diagram shows the system flow of the present invention under standard operating conditions where the safety helmet 3 and AR glasses 4 are worn accurately and the miniature camera completely captures images of the ring-shaped light source target. S1: The concrete placing boom subsystem performs a power-on self-test. The motion control module controls the robotic arm to extend to the ready-to-work position, activating the electronic compass, ring light target, and wireless data transmission module one. The global absolute orientation data of the placing boom base and the angle data of each joint begin to be continuously transmitted through the wireless data transmission module one.

[0048] S2: The operator wears the enhanced remote operation subsystem. The wireless data transmission module 3 inside the safety helmet 3 begins to receive global absolute orientation data from the concrete placing boom subsystem, including the global absolute orientation angle α1 of the placing boom base, and the global absolute orientation angle α2 and joint angle data from the AR glasses 4. Simultaneously, the AR glasses 4 are powered on, and its built-in miniature camera begins to continuously capture scene images of the ring light source target, while its built-in electronic compass begins to generate global absolute orientation data for the AR glasses 4.

[0049] S3: The first-person motion resolution module performs calculations based on data received synchronously from three channels by the wireless data transmission module. The wired data transmission module 1 of the safety helmet 3 acquires the ring light source target image captured by the miniature camera and the global absolute orientation data of the AR glasses 4; The base global absolute orientation data and joint angle data from the concrete placing boom subsystem are obtained through the wireless data transmission module 1. The wireless data transmission module 2 receives remote operation commands from the remote control handle 2. S4: The first-person motion solution module performs the following core calculations based on the above multi-source information: Define the translation vector from joint i-1 to joint i as follows: The rotation matrix is The homogeneous transformation matrix is

[0050] Target coordinate system pose calculation: Based on the ring light source target image, camera intrinsic parameters and the known size of the ring light source target, combined with the global absolute orientation data of AR glasses 4, the pose of the tool coordinate system relative to the AR glasses 4 coordinate system is calculated. Motion pose calculation: The translation vector calculation is based on the coordinate system {B} of the placing boom base; the translation vector is the relative position of the origin of the tool end coordinate system {T}. It is obtained by successively multiplying the homogeneous transformation matrices of each joint and the tool end, and its expression is:

[0051] Rotation matrix calculation: based on the global absolute orientation angle of the placing boom base. Global absolute orientation angle of AR glasses The rotation matrix of the tool end coordinate system {T} relative to the base coordinate system {B} Its expression is:

[0052] in, Represents the cosine operation. Represents the sine operation; Based on the joint angle data obtained from S3 and the global absolute orientation data of the base, the first-view motion calculation module calculates the theoretical pose of the tool coordinate system relative to the base coordinate system through the robot's forward kinematics. S5: The AR data processing module receives the pose of the tool coordinate system relative to the AR glasses 4 coordinate system, calculated by the first-view motion calculation module. The AR glasses 4, with the center of the ring-shaped light source target as the origin, stably renders and overlays the virtual coordinate system onto the operator's real field of view. Simultaneously, the relative pose, motion speed, and "ready" status of the end effector are updated on the graphics panel to the side of the field of view. S6: The operator observes the virtual coordinate system in the AR interface and presses the direction button on the remote control corresponding to the positive X-axis. This command is defined as the desired motion vector in the camera coordinate system {C}. And it is transmitted through the wireless data transmission module.

[0053] The first-person motion calculation module utilizes the calculated rotation matrix of the tool coordinate system {T} relative to the AR glasses coordinate system {C}. , will v c Transform to the tool coordinate system to obtain the desired displacement vector in the tool coordinate system:

[0054] where a=(a x ,a y ,a z The original tool coordinate system {T} is moved along this direction by 'a' and becomes the new coordinate system {T'}.

[0055] Let the homogeneous transformation matrix of the current tool coordinate system {T} relative to the base coordinate system {B} be... Its rotating part is Transform the displacement 'a' from the tool coordinate system to the base coordinate system:

[0056] Construct the pure translation transformation matrix:

[0057] The updated end pose is:

[0058] according to The angle parameters of each joint of the robotic arm after movement are calculated using inverse kinematics, thereby obtaining the drive signals for each joint and the attitude control signals for the end effector. The end effector pose after movement can also be directly represented by joint variables:

[0059] S7: Control commands are sent to the robot motion control module via the wireless data transmission module 3, driving the robotic arm and end effector hose to move precisely. During the movement, steps S2 to S5 are executed in a high-speed loop: the real-time status of the concrete placing machine subsystem is fed back to the remote control terminal, the first-view motion calculation module updates the calculation results accordingly, and the AR glasses 4 update the coordinate system and status information display in real time, thus forming a precise operation closed loop of "perception-calculation-control-feedback".

[0060] like Figure 7As shown, this invention demonstrates how, in the case where the operator's head tilt is too large, causing severe perspective distortion of the visual marker and the miniature camera to fully capture the ring-shaped light source target system, the robustness of pose calculation and the continuity of system operation are ensured by using proactive human-computer interaction guidance as the core strategy.

[0061] S1: The operator's head tilt causes perspective distortion in the ring-shaped target image captured by the miniature camera, transforming its regular elongated outline into an irregular trapezoid in the image. The AR data processing module calculates the aspect ratio deviation based on the image information acquired by the miniature camera, and this geometric deformation index serves as the basis for evaluating the deformation threshold.

[0062] S2: When the geometric deformation index exceeds the preset deformation threshold, the AR data processing module determines that the current view is unreliable and non-ideal, and generates a visual prompt instruction.

[0063] S3: This visual cue is sent to the optical engine of AR glasses 4, rendering the clear text "Collaborative Calibration" and an intuitive warning icon in the upper left corner of the operator's field of vision. Based on this cue, the operator actively adjusts their head posture to reduce the tilt angle between themselves and the ring-shaped light source target.

[0064] S4: The system continuously monitors visual quality assessment indicators. When the operator adjusts their head posture to the optimal range, restoring the geometric deformation index to within an acceptable threshold, the AR data processing module determines that the viewing angle has been optimized. The AR glasses 4 then clear the previous prompt information and generate a "ready" status prompt, switching to S2 of Embodiment 1.

[0065] The processes described in S1 to S4 constitute a high-speed "detection-prompt-correction-recovery" monitoring closed loop. This closed loop executes in parallel with the main control flow (S2 to S6) described in Example 1, continuously ensuring the observation quality from the first-person perspective, thereby ensuring the intuitive spatial mapping teleoperation of the entire system.

[0066] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A first-person remote-operated concrete construction system based on augmented reality, characterized in that, Includes a concrete placing boom subsystem and an enhanced remote operation subsystem; The concrete placing boom subsystem includes a power module 1, a placing boom body (1), a motion control module, a data acquisition module, and a wireless data transmission module 1. The power module 1 is used to supply power to the concrete placing boom subsystem. The placing boom body (1) is used to perform actual work tasks. The motion control module is connected to the placing boom body (1) and is used to adjust the joint angle of the robotic arm and the posture of the placing tube according to the drive signal. The wireless data transmission module 1 is used to transmit data between the two subsystems. The enhanced teleoperation subsystem includes a teleoperation handle (2), a safety helmet (3), and AR glasses (4); the safety helmet (3) is used to integrate a processing module and a data transmission module; AR glasses (4) are used to realize visual computing and 3D rendering; The data acquisition module includes: an electronic compass, a joint encoder, and a ring light source target; the electronic compass is used to provide global absolute orientation data of the fabric laying machine base; the ring light source target is set on the end hose for visual positioning marking. The safety helmet (3) includes: a power module three, a power supply module, a first-view motion calculation module, a wireless data transmission module three, and a wired data transmission module one for the safety helmet (3); the power module three is used to supply power to the integrated components inside the safety helmet (3); the power supply module is used to transmit the power from the power module three to the AR glasses (4); the first-view motion calculation module is used to convert the relative pose between coordinate systems into the target pose in the base coordinate system, and then calculate the drive signals of each joint and the attitude control signals of the end effector based on the target pose; the wireless data transmission module three is used to receive the status information of the concrete placing machine subsystem and send the calculated motion command to the concrete placing machine subsystem; the wired data transmission module one for the safety helmet (3) is used to send rendering information to the AR glasses; The AR glasses (4) include: a power supply module, a miniature camera, an optical engine, an electronic compass II, an AR data processing module, and a wired data transmission module II for the AR glasses (4); the miniature camera is used to capture environmental images; the optical engine is used to display augmented reality information, the optical engine includes a display module, a waveguide and a coupler, the display module generates the original virtual image; the waveguide uses the principle of total internal reflection to send the image to the human eye; the coupler guides light into and out of the waveguide to complete the input and output of the image; the electronic compass II is used to provide global absolute orientation data for the AR glasses (4); the AR data processing module is used to receive the relative pose result and system status data calculated by the first-view motion calculation module, and generate an image rendering frame containing augmented reality information such as target coordinate system, text label, and status icon; the wired data transmission module II for the AR glasses (4) is used to receive rendering information from the safety helmet (3).

2. The system according to claim 1, characterized in that, The remote control handle (2) includes: a second power module, a directional button, a safety button, and a second wireless data transmission module; the second power module is used to power the remote control handle (2); the directional button is used to output remote operation commands on the XYZ axes in the first view coordinate system; the safety button is used to ensure the output of any command with the operator's permission; the second wireless data transmission module is used to send the remote operation commands to the safety helmet (3).

3. The system according to claim 1, characterized in that, The system implementation logic includes the following steps: S1: Start the system, prepare the concrete placing machine subsystem, control the robotic arm to unfold using the motion control module, and activate the electronic compasses one and two, the ring light source target and the data transmission module at the same time. S2: The operator wears the enhanced remote operation subsystem, and the miniature camera inside the AR glasses (4) begins to capture the image of the ring light source target; the wireless data transmission module three inside the safety helmet (3) begins to receive the base global absolute orientation data and joint angle data from the wireless transmission module one of the concrete placing machine subsystem; S3: The first-view motion calculation module synchronously performs the following processing: based on the ring light source target image, the pre-stored camera intrinsic parameters and the known size of the ring light source target, and combined with the global absolute orientation data of the AR glasses (4), the pose of the tool coordinate system relative to the camera coordinate system is calculated; based on the joint angle data collected by the joint encoder, the theoretical pose of the tool coordinate system relative to the base coordinate system is calculated through the robot's forward kinematics. Based on the global absolute orientation data of the AR glasses (4) and the fabric machine base, establish the orientation alignment relationship between the camera coordinate system and the fabric machine base coordinate system; S4: The operator issues a motion command defined in the first-view coordinate system through the remote control handle; the first-view motion calculation module calculates the orientation alignment relationship and the pose of the tool coordinate system relative to the camera coordinate system based on step S3, and maps the motion command to the target pose of the tool coordinate system in the cloth laying machine base coordinate system. S5: Based on the target pose, the drive signals of each joint and the attitude control signals of the end effector are calculated by solving the robot inverse kinematics, and the signals are sent to the motion control module of the concrete placing machine subsystem. S6: The AR glasses (4) based on the pose of the tool coordinate system relative to the camera coordinate system calculated in step S3, render a virtual coordinate system with the center of the ring light source target as the origin and aligned with the tool coordinate system in the operator's field of vision, and overlay and display system status information in real time. S7: The motion control module drives the robotic arm and end effector to move, completing actions consistent with the operator's first-person perspective commands, forming a closed-loop control.

4. The system according to claim 1, characterized in that, The ring light source target (5) is composed of high-brightness light sources that are uniformly and closely arranged along the circumference, and is fixed around the periphery of the end hose (7) by an openable ring mounting bracket (6); the mounting bracket is composed of two symmetrical semi-circular shells connected by screws, and the inner surface of the target is fixed to the outer periphery of the bracket; after installation, the lower surface of the target is perpendicular to the axial direction of the hose, and its whole is located above the outlet at a certain height.

5. The system according to claim 1, characterized in that, The first-view motion calculation module is configured as follows: S1: Receive the ring light source target image captured by the miniature camera, the global absolute orientation data provided by the AR glasses (4) and the electronic compass of the fabric machine base respectively, the motion command issued by the remote control handle and the joint angle data collected by the joint encoder; S2: Based on the target image of the ring light source, the known size and the camera intrinsic parameters of the micro camera, and combined with the global absolute orientation data of the AR glasses (4), the pose of the tool coordinate system relative to the AR glasses (4) coordinate system is calculated. S3: Based on the global absolute orientation data of the AR glasses (4) and the fabric machine base, establish the orientation alignment relationship between their coordinate systems; S4: Based on the orientation alignment relationship, the motion command issued by the remote control handle in the first view coordinate system is mapped to the target pose in the tool coordinate system in the cloth laying machine base coordinate system; S5: Based on the target pose, the drive signals of each joint and the attitude control signals of the end effector are calculated by solving the robot inverse kinematics and sent to the motion control module.

6. The system according to claim 1, characterized in that, The AR glasses (4) are configured as follows: Based on the relative pose of the tool coordinate system and the camera coordinate system calculated by the first perspective motion calculation module, a virtual three-dimensional coordinate system is rendered in the display module with the center of the ring light source target as the origin. Meanwhile, the current operating status, including "Ready", "In Motion", "Emergency Stop" or "Collaborative Calibration", is superimposed on the left side of the display module, and the relative pose and real-time motion speed of the end effector are superimposed on the right side of the display module.

Images