WebXR-based panoramic immersive robot teleoperation system and method

Abstract

The invention relates to the technical field of XR operation, and discloses a WebXR-based panoramic immersive robot teleoperation system and method. The method comprises the following steps: carrying out distortion correction and splicing fusion on original images collected by a double-fisheye lens to obtain a panoramic image frame; encoding the panoramic image frame into a video code stream, transmitting the video code stream to a browser end, and decoding the video code stream at the browser end to obtain a panoramic video texture; mapping the panoramic video texture to the inner surface of a sphere, setting a virtual camera at the center of the sphere, and constructing a panoramic sphere rendering model; and converting the attitude quaternion of the VR head-mounted display into a sight decoupling rotation matrix, performing local GPU rendering on the panoramic sphere rendering model to obtain a real-time visual angle picture, and mapping a rocker shaft value of the gamepad into a control instruction to be sent to a robot dog end. In the whole view angle switching process, vertex transformation and texture sampling are completed at the browser end, no pan-tilt rotation instruction is generated or sent to the robot dog end through the network, and delay is reduced.

Description

Technical Field

[0001] This invention relates to the field of XR operation technology, and in particular to a panoramic immersive robot teleoperation system and method based on WebXR. Background Technology

[0002] Currently, remote teleoperation technology for quadruped robots faces significant technical bottlenecks in complex environment applications. Traditional solutions use fixed, narrow-angle cameras with a field of view typically below 120 degrees. This results in operators being unable to perceive the environment to their sides or rear when performing lateral movements or reversing, creating severe blind spots and greatly increasing the risk of collisions.

[0003] To address the limited field of view issue, existing technologies have introduced mechanical gimbals to attempt to expand the field of view through physical rotation. However, when an operator wearing a VR headset rotates their head, the posture data needs to be transmitted over the network to drive the remote gimbal motor, and the resulting image is then encoded and transmitted back. The entire chain has a cumulative latency of 100-300 milliseconds. This phenomenon, where visual updates lag behind head movements, causes a time misalignment between the operator's visual input and the vestibular system's perception, triggering severe motion sickness and making it impossible for the operator to engage in immersive operation for extended periods. Summary of the Invention

[0004] The main objective of this invention is to provide a WebXR-based panoramic immersive robot teleoperation system and method. The operator's head posture data is used only to update the view matrix parameters of the virtual camera in the local GPU. The entire viewpoint switching process is completed on the browser side for vertex transformation and texture sampling. No gimbal rotation commands are generated or sent to the robot dog via the network. This eliminates the high-latency links in the prior art and compresses the viewpoint update latency from 100-300 milliseconds to 8-11 milliseconds of GPU frame rendering time. This eliminates motion sickness caused by visual-vestibular time misalignment and enables the operator to perform immersive teleoperation for extended periods.

[0005] To achieve the above objectives, this invention provides a WebXR-based panoramic immersive robot teleoperation method, comprising the following steps: The original images captured by the double fisheye lens are subjected to distortion correction and stitching fusion to obtain panoramic image frames; The panoramic image frames are encoded into a video stream and transmitted to the browser, and the panoramic video texture is obtained by decoding on the browser. The panoramic video texture is mapped onto the inner surface of the sphere and a virtual camera is set at the center of the sphere to construct a panoramic sphere rendering model. The VR headset's pose quaternion is converted into a gaze decoupling rotation matrix, and the panoramic sphere rendering model is rendered locally on the GPU based on the gaze decoupling rotation matrix to obtain a real-time view. At the same time, the joystick axis values ​​of the game controller are mapped into control commands and sent to the robot dog terminal.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the original image acquired by the double fisheye lens is subjected to distortion correction and stitching fusion to obtain a panoramic image frame, including: Distortion correction is performed on the original image acquired by the double fisheye lens based on the camera intrinsic parameter matrix and distortion coefficients to obtain the distortion-corrected image; Feature points are extracted in the overlapping region of the distortion-corrected image and image registration is performed using a homography matrix to obtain a registered image. The registered image is then seamlessly stitched together in the overlapping region to obtain a spherical image. The spherical coordinates of the spherical image are converted into the horizontal and vertical pixel coordinates of the planar image to obtain a panoramic image frame.

[0007] Optionally, in a second implementation of the first aspect of the present invention, encoding the panoramic image frame into a video stream and transmitting it to a browser, and decoding it on the browser to obtain the panoramic video texture, includes: The panoramic image frames are compressed into a video stream, the NAL units of the video stream are encapsulated into RTP data packets and encrypted with SRTP, and transmitted to the browser through a peer-to-peer connection established by WebRTC. The browser receives the RTP data packet through the RTCPeerConnection interface and sequentially performs SRTP decryption and RTP decapsulation to decode it into video frame data. The video frame data is then uploaded to a WebGL texture object to obtain a panoramic video texture.

[0008] Optionally, in a third implementation of the first aspect of the present invention, mapping the panoramic video texture onto the inner surface of the sphere and setting the virtual camera at the center of the sphere to construct a panoramic sphere rendering model includes: The vertex position coordinates and texture coordinates of the sphere are generated by subdividing the latitude and longitude grid. The vertex position coordinates and texture coordinates are then uploaded to the vertex buffer object to obtain the sphere grid data. Compile the vertex shader and fragment shader and link them into a shader program. Bind the panoramic video texture to a WebGL texture unit as the sampling source for the fragment shader to obtain a texture mapping shader. The virtual camera is set at the origin of the sphere's center, and the perspective projection matrix and initial view matrix are configured. By combining the sphere mesh data and the texture mapping shader, a panoramic sphere rendering model is obtained.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, a virtual camera is set at the origin of the sphere's center and configured with a perspective projection matrix and an initial view matrix. Combining the sphere mesh data and the texture mapping shader, a panoramic sphere rendering model is obtained, including: Set the position of the virtual camera to the origin of the sphere's center and set the initial orientation to forward and the upward direction to the zenith to obtain the camera's initial posture; Calculate the perspective projection matrix based on the field of view, canvas aspect ratio, near clipping plane distance, and far clipping plane distance, and set the identity matrix as the initial view matrix; The perspective projection matrix and the initial view matrix are passed to the uniform variable of the texture mapping shader, the vertex array object of the sphere mesh data is bound, and the rendering loop is started to execute the drawing instructions to obtain the panoramic sphere rendering model.

[0010] Optionally, in the fifth implementation of the first aspect of the present invention, the pose quaternion of the VR headset is converted into a gaze decoupling rotation matrix, and the panoramic sphere rendering model is rendered locally using the gaze decoupling rotation matrix to obtain a real-time view. Simultaneously, the joystick axis values ​​of the game controller are mapped to control commands and sent to the robot dog terminal, including: In each frame rendering loop, the WebXR pose interface is called to obtain the pose quaternion of head rotation from the VR headset; The three-dimensional rotation matrix is ​​calculated based on the attitude quaternion and expanded into a four-dimensional homogeneous transformation matrix to obtain the line-of-sight decoupling rotation matrix. The view decoupling rotation matrix is ​​passed to the shader program to update the view matrix parameters of the virtual camera and vertex transformation is performed on the local GPU, while no gimbal rotation command is generated and no data is sent to the robot dog terminal via the network; Based on the line-of-sight decoupling rotation matrix, the panoramic sphere rendering model is rendered locally using GPU to obtain a real-time view. At the same time, the joystick axis values ​​of the game controller are mapped into control commands and sent to the robot dog terminal.

[0011] Optionally, in the sixth implementation of the first aspect of the present invention, the panoramic sphere rendering model is rendered locally using a GPU based on the line-of-sight decoupling rotation matrix to obtain a real-time view, and the joystick axis values ​​of the game controller are mapped to control commands and sent to the robot dog terminal, including: The line-of-sight decoupling rotation matrix is ​​passed to the vertex shader, the vertex position coordinates of the sphere mesh data are transformed, and the interpolated texture coordinates are output to the fragment shader. In the fragment shader, texture sampling is performed from the corresponding coordinate region of the panoramic video texture according to the interpolated texture coordinates to obtain the texture sampling result; The texture sampling results are output to the frame buffer in the local GPU's rendering pipeline to complete rasterization rendering and obtain a real-time view. Read the joystick axis values ​​of the game controller and map them as forward linear velocity, lateral linear velocity, and yaw angular velocity, then send them to the robot dog terminal.

[0012] Optionally, in the seventh implementation of the first aspect of the present invention, the view decoupling rotation matrix is ​​passed to the vertex shader, the vertex position coordinates of the sphere mesh data are transformed, and the interpolated texture coordinates are output to the fragment shader, including: The line-of-sight decoupling rotation matrix is ​​passed to the vertex shader through the uniform interface to generate shader matrix parameters; In the vertex shader, matrix multiplication is performed sequentially on the vertex position coordinates of the sphere mesh data with the shader matrix parameters and the perspective projection matrix to obtain the transformed vertex positions. Based on the transformed vertex positions, the texture coordinates of the sphere mesh data are output from the vertex shader to the fragment shader via varying variables.

[0013] Optionally, in the eighth implementation of the first aspect of the present invention, reading the joystick axis values ​​of the game controller and mapping them as forward linear velocity, lateral linear velocity, and yaw angular velocity and sending them to the robot dog terminal includes: In the rendering loop, the HTML5 gamepad interface is called to read the axes array of the gamepad, obtain the joystick axis values, perform dead zone filtering on the joystick axis values, and set the axis values ​​with absolute values ​​less than the dead zone threshold to zero to obtain the filtered axis values. The left joystick longitudinal axis value in the filtered axis values ​​is converted into forward linear velocity through a velocity mapping function, the left joystick lateral axis value is converted into lateral linear velocity, and the right joystick lateral axis value is converted into yaw rate. The forward linear velocity, the lateral linear velocity, and the yaw rate are serialized into data packets according to a binary protocol format and sent to the robot dog terminal through the WebRTC data channel.

[0014] This invention also provides a WebXR-based panoramic immersive robot teleoperation system, comprising: The image fusion module is used to perform distortion correction and stitching fusion on the original images captured by the double fisheye lens to obtain panoramic image frames; The transmission decoding module is used to encode the panoramic image frame into a video stream and transmit it to the browser, and decode it on the browser to obtain the panoramic video texture. The texture mapping module is used to map the panoramic video texture onto the inner surface of the sphere and set the virtual camera to the center of the sphere to construct a panoramic sphere rendering model. The real-time rendering module is used to convert the pose quaternion of the VR headset into a gaze decoupling rotation matrix, and perform local GPU rendering on the panoramic sphere rendering model based on the gaze decoupling rotation matrix to obtain a real-time view. At the same time, it maps the joystick axis values ​​of the game controller into control commands and sends them to the robot dog terminal.

[0015] In summary, this invention uses operator head posture data only to update the view matrix parameters of the virtual camera in the local GPU. The entire perspective switching process is completed on the browser side, including vertex transformation and texture sampling. No gimbal rotation commands are generated, nor are they sent over the network to the robot dog. This eliminates the high-latency links found in existing technologies, compressing the perspective update latency from 100-300 milliseconds to 8-11 milliseconds of GPU frame rendering time. This eliminates motion sickness caused by visual-vestibular time misalignment, allowing operators to perform immersive remote operations for extended periods. Combined with 360-degree panoramic video captured by a dual-fisheye lens and texture mapping technology on the inner surface of a sphere, operators can instantly acquire environmental images from any direction without waiting for the physical gimbal to adjust its angle. This completely eliminates the blind spots in the side and rear view of traditional narrow-angle cameras, significantly improving the safety and accuracy of complex operations such as navigating narrow spaces, lateral obstacle avoidance, and reversing. Furthermore, the browser / server architecture based on WebXR and WebRTC standards enables cross-platform compatibility and plug-and-play deployment without the need to install dedicated client software. The HTML5 gamepad interface supports precise control with consumer-grade gamepads, significantly reducing system deployment costs and the learning curve, while avoiding the hardware costs and failure risks associated with mechanical gimbals. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the steps of a WebXR-based panoramic immersive robot teleoperation method in one embodiment of the present invention; Figure 2 This is a block diagram of the WebXR-based panoramic immersive robot teleoperation system in an embodiment of the present invention.

[0017] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] Reference Figure 1 This embodiment provides a WebXR-based panoramic immersive robot teleoperation method, including the following steps: S1, the original images captured by the fisheye lens are subjected to distortion correction and stitching fusion to obtain panoramic image frames; S2, encodes the panoramic image frames into a video stream and transmits it to the browser, and decodes it on the browser to obtain the panoramic video texture; S3 maps the panoramic video texture onto the inner surface of the sphere and sets the virtual camera to the center of the sphere to construct a panoramic sphere rendering model. S4 converts the VR headset's pose quaternion into a gaze decoupling rotation matrix, and performs local GPU rendering on the panoramic sphere rendering model based on the gaze decoupling rotation matrix to obtain a real-time view. At the same time, it maps the joystick axis values ​​of the game controller into control commands and sends them to the robot dog terminal.

[0020] In one example, the original images captured by the double fisheye lens are subjected to distortion correction and stitching fusion to obtain panoramic image frames, including: Distortion correction is performed on the original image acquired by the double fisheye lens based on the camera intrinsic parameter matrix and distortion coefficients to obtain the distortion-corrected image; Feature points are extracted in the overlapping region of the distortion-corrected image and image registration is performed using the homography matrix to obtain the registered image. The registered images are then seamlessly stitched together in the overlapping region to obtain a spherical image. The spherical coordinates of the spherical image are converted into the horizontal and vertical pixel coordinates of the planar image to obtain the panoramic image frame.

[0021] In this example, the raw images captured by the dual fisheye camera are input into the image processing module. Pixel-level distortion correction is performed on the images based on the pre-calibrated camera intrinsic parameter matrix and distortion coefficient vector. The intrinsic parameter matrix includes focal length parameters and principal point coordinates, while the distortion coefficients include radial and tangential distortion components. The original pixel coordinates are normalized to normalized image coordinates. The radial and tangential components of the normalized coordinates are corrected using a distortion model. The corrected normalized coordinates are projected back to the pixel plane through inverse mapping, generating two distortion-corrected images. Stable feature points are extracted in the overlapping region of the two images using algorithms such as SIFT or ORB. An efficient matching algorithm, such as FLANN, is used to establish the correspondence between key points in the two images. Finally, the RANSAC algorithm is used to estimate a 3×3 homography matrix in three-dimensional space under noise-resistant conditions, describing how one image undergoes an affine transformation in the overlapping region to align with the coordinate system of the other image. After registration, the image registration module uses the homography matrix to perform a perspective transformation on one image to align its content with the other image within the overlapping region. At this point, the two images are registered. A multi-band image fusion algorithm is applied to the overlapping region of the registered images, decomposing the images into Gaussian and Laplacian pyramids of multiple scales. Each band is then weighted and synthesized using a gradient weighting function (such as a cosine window), resulting in natural low-frequency color transitions and seamless high-frequency textures, outputting a spherical image. A coordinate system based on spherical coordinates (i.e., longitude θ and latitude φ) is established, and an equidistant rectangular projection mode is set as the output format, linearly mapping longitude θ to horizontal pixel coordinates x. e Latitude φ is linearly mapped to vertical pixel coordinate y e , where x e With y e Calculate them as x respectively e = W·θ / 2π,y e = H·(φ +π / 2) / π, where W and H are the pixel width and height of the output image. By mapping the spherical image pixel by pixel into an equidistant rectangular projection image on a standard two-dimensional image plane, a panoramic image frame covering 360 degrees horizontally and 180 degrees vertically is obtained.

[0022] When seamlessly stitching the registered images in the overlapping region, a multi-band fusion algorithm is used to process different frequency components to eliminate stitching gaps. This includes: performing Gaussian pyramid decomposition and Laplacian pyramid decomposition on the registered images to generate a 5-layer pyramid structure, decomposing the image into frequency components from layer 0 to layer 4. Layers 0 to 3 are high-frequency components containing texture details, and layer 4 is a low-frequency component containing color information, resulting in multi-band image components. In the overlapping region, the multi-band image components are weighted and fused using transition bands of different widths. The low-frequency components of layer 4 are then weighted... A wide transition band of 40 pixels was used for gradient blending to smooth color differences. A narrow transition band of 10 pixels was used for sharp blending of high-frequency components from layer 0 to layer 3 to preserve texture edges. The blending weight function adopted a cosine gradient form to continuously transition from 0 to 1 in the overlapping area to obtain the frequency division fusion result. The frequency division fusion result was reconstructed using a Laplacian pyramid. The frequency components of each layer were upsampled in sequence and accumulated to restore the complete image. A seamless stitching effect with smooth color transition and clear texture details was formed in the overlapping area, resulting in a spherical image without visible stitching gaps.

[0023] In one example, panoramic image frames are encoded into a video stream and transmitted to the browser, and then decoded on the browser to obtain the panoramic video texture, including: Panoramic image frames are compressed into video streams, and the NAL units of the video streams are encapsulated into RTP data packets and encrypted with SRTP. The data is then transmitted to the browser via a peer-to-peer connection established through WebRTC. On the browser side, RTP data packets are received through the RTCPeerConnection interface, and SRTP decryption and RTP decapsulation are performed sequentially. The data packets are then decoded into video frame data and uploaded to a WebGL texture object to obtain a panoramic video texture.

[0024] In this example, panoramic image frames in equidistant rectangular projection format are acquired from the processing end. These panoramic image frames have a complete 360-degree field of view and are cached in a shared memory area in YUV420P color format. The encoding module calls a hardware video encoding interface, such as NVIDIA NVENC or Intel Quick Sync Video, and sets the encoding parameters to H.264 High Profile, Constant Bit Rate Control (CBR), one IDR frame per second in GOP structure, no B-frames, and disabled look-ahead buffer, etc., to maximize latency compression while meeting network real-time constraints. The encoder encodes the panoramic image frames into a video stream composed of a series of NAL units, each NAL unit being an SPS, PPS, IDR frame, or P frame. The NAL units are RTP-packed according to RFC 6184. Smaller NAL units can be directly encapsulated into a single NAL packet, while larger NAL units (such as keyframes) need to be divided into multiple segments in FU-A format, with each segment including a start bit, sequence number, and end bit for reassembly at the receiving end. All RTP packets must be configured with standard RTP header fields, including payload type, timestamp, sequence number, and synchronization source identifier (SSRC), forming the RTP transmission structure. After RTP packetization, an SRTP secure encapsulation mechanism is introduced. The RTP payload is encrypted using AES-128-CM symmetric encryption, and a 10-byte HMAC-SHA1 authentication code is calculated and appended to the packet tail to ensure the confidentiality and integrity of data transmitted over the public network. Key negotiation is completed through the WebRTC DTLS-SRTP handshake process. Both parties exchange SDPs in the WebSocket signaling channel and negotiate and probe candidate paths for P2P connections through the ICE mechanism. When the optimal transmission path is confirmed (such as a host or srflx direct connection), the SRTP key material is obtained through the DTLS handshake, thus establishing an end-to-end encrypted data channel. The encoding end continuously sends encrypted SRTP data packets to the browser client at a fixed frame rate (e.g., 60fps). On the browser side, the WebRTC RTCPeerConnection interface maintains the end-to-end connection state and automatically receives SRTP encrypted data packets internally. The browser kernel calls the media engine to decrypt the received SRTP data packets, extract the original RTP packets, decapsulate them, restore the NAL unit, and hand them over to the decoder to complete the decoding operation of H.264 video frames, outputting YUV format video frame data. The video frame data is sent into the HTMLVideoElement object as the playback source, and then a video texture is created through the WebGL context interface. The decoded frames are uploaded to the GPU texture unit in texture form, and the gl.texImage2D or gl.texSubImage2D methods are used to complete the texture binding of the frame data, forming a panoramic video texture with real-time updating capabilities.

[0025] When establishing a peer-to-peer connection via WebRTC, signaling negotiation and ICE candidate address connectivity testing are performed to select the optimal transmission path. This includes: the robot client generating an SDP Offer message containing video encoding / decoding capabilities and network candidate addresses, relaying it to the browser client via the signaling server; the browser client parsing the SDP Offer and generating an SDP Answer to return; simultaneously, both parties initiate the ICE process, obtaining public IP addresses as server reflection candidates through the STUN server and relay addresses as relay candidates through the TURN server, resulting in a candidate address set containing local candidates, server reflection candidates, and relay candidates; exchanging the candidate address set between the robot client and the browser client via the signaling server; performing connectivity testing on all candidate address pairs and sending STUN binding request probe packets; selecting the direct connection path with the lowest latency based on response latency and priority scores; if the direct connection path fails to establish, falling back to the TURN relay path to obtain the optimal transmission path; performing a DTLS protocol handshake on the optimal transmission path and completing authentication by exchanging certificate fingerprints; negotiating and generating an SRTP encryption key and salt value; establishing an end-to-end encrypted media transmission channel to obtain a low-latency peer-to-peer connection.

[0026] When compressing panoramic image frames into a video stream, a low-latency coding configuration is used, and the RTP encapsulation mode is adaptively selected based on the NAL unit size. This includes: configuring the hardware encoder to enable the low-latency coding preset mode, disabling lookahead buffering and adaptive B-frame decision-making; setting the GOP structure to insert an IDR keyframe every 60 frames, with the remaining frames being P-frames; limiting the number of reference frames to 1 frame; using a constant bitrate mode and setting a target bitrate; compressing and encoding the panoramic image frames to obtain a video stream composed of IDR frame NAL units and P-frame NAL units; determining the size of each NAL unit in the video stream; when the NAL unit size is less than the MTU minus the RTP header opening... When the threshold is reached, a single NAL unit mode is used to use the complete NAL unit as the RTP packet payload. When the NAL unit size is larger than the threshold, the FU-A fragmentation mode is used to divide the NAL unit into multiple fragments and add an FU indicator and FU header to each fragment to identify the fragment sequence number, thus obtaining the RTP payload data. The RTP payload data is encapsulated with an RTP packet header and filled with a sequence number, timestamp, and SSRC synchronization source identifier. The timestamp increments at a clock frequency of 90kHz with an increment of 1500 per frame. The encapsulated RTP data packet is sent to the SRTP layer for AES encryption and HMAC-SHA1 integrity verification to obtain the encrypted RTP data packet.

[0027] In one example, a panoramic sphere rendering model is constructed by mapping a panoramic video texture onto the inner surface of a sphere and setting a virtual camera at the center of the sphere, including: The vertex position coordinates and texture coordinates of the sphere are generated by subdividing the latitude and longitude grid. The vertex position coordinates and texture coordinates are then uploaded to the vertex buffer object to obtain the sphere grid data. Compile the vertex shader and fragment shader and link them into a shader program. Bind the panoramic video texture to a WebGL texture unit as the sampling source for the fragment shader to obtain a texture mapping shader. The virtual camera is set at the origin of the sphere's center, and the perspective projection matrix and initial view matrix are configured. Combined with the sphere mesh data and texture mapping shader, a panoramic sphere rendering model is obtained.

[0028] In this example, the WebGL or WebGL2 context is obtained within the browser's JavaScript environment, and the appropriate canvas width and height are set during the initialization phase to match the video source texture size. Based on the principle of parametric spherical modeling, the entire sphere is uniformly subdivided into 64 horizontal sectors and 32 vertical latitude layers using a latitude-longitude division method, with each intersection of latitude and longitude being a vertex. For the j-th sector and the i-th latitude layer, its longitude angle θ... j =j·(2π / 64), latitude angle φ i = π / 2 + i·(π / 32), based on which the corresponding three-dimensional position coordinates (x, y, z) can be calculated using the transformation formula from spherical to Cartesian coordinates, where x = R·cosφ i ·cosθ j y=R·sinφ i z = R·cosφ i ·sinθ j R is the unit radius of the sphere. Simultaneously, to achieve the mapping between video texture and spherical coordinates, texture coordinates (u, v) are defined for each vertex, where u = θ. j / (2π), v=(φ iThe texture, with a length of +π / 2) / π, fully covers the longitude range from 0 to 1 horizontally and the latitude range from the South Pole to the North Pole vertically. All generated vertex positions and texture coordinates are packaged into floating-point arrays, and a mesh topology is constructed using triangle indexing. Each small quadrilateral consists of two sets of triangular faces, forming a spherically discretized vertex index array. After mesh generation, vertex buffer objects and index buffer objects are created, and the position and texture data are uploaded to the GPU using the `gl.bufferData` function. WebGL shader programs are written and compiled. The vertex shader receives the position and texture coordinate attributes of each vertex and receives the model-view matrix and projection matrix through uniform variables, implementing geometric transformations from the world coordinate system to clip space. The fragment shader declares a 2D texture sampler `uniform sampler2D` to sample color information from the panoramic video texture and uses the sampling result as the final color output for each fragment. The vertex and fragment shaders are compiled separately and then linked to generate a WebGL program object. After the program is started, vertex attributes and texture sampler positions are bound. Simultaneously, the HTMLVideoElement object is bound to a WebGL texture object as the video frame source, and texture parameters such as LINEAR filtering and CLAMP_TO_EDGE surround mode are configured. Before each frame is rendered, the current video frame is updated to the GPU texture unit via gl.texSubImage2D to synchronize the texture with the video decoding progress. To achieve 3D observation, a virtual camera model is constructed during rendering initialization. Its position is fixed at the origin of the sphere's coordinates (0, 0, 0), facing along the negative Z-axis, with the upward direction being the positive Y-axis. A perspective projection matrix is ​​used to simulate the human eye's imaging process, setting the field of view (FOV) to 90 degrees, the aspect ratio to canvas.width / canvas.height, and the near and far planes to 0.1 and 100.0, respectively. A 4×4 perspective projection matrix is ​​constructed using common projection matrix generation formulas. The initial view matrix is ​​set to an identity matrix to represent no rotation. In VR mode, this matrix is ​​updated in real time using the head-mounted display's pose quaternions to drive the viewpoint transformation. In each frame's rendering loop, the following process is executed: first, update the video texture content; then, clear the color and depth buffers; next, bind the shader program and VBO / EBO resources; and pass the current projection matrix and model-view matrix into the corresponding uniform variables. Finally, call the gl.drawElements function to draw a spherical triangle mesh according to the index, and render the panoramic sphere with the video texture onto the canvas, completing the panoramic rendering process observed from the current virtual camera's perspective.

[0029] In one example, a virtual camera is positioned at the origin of the sphere's center, and a perspective projection matrix and an initial view matrix are configured. Combined with sphere mesh data and a texture mapping shader, a panoramic sphere rendering model is obtained, including: Set the position of the virtual camera to the origin of the sphere's center and set the initial orientation to forward and the upward direction to the zenith to obtain the camera's initial posture; Calculate the perspective projection matrix based on the field of view, canvas aspect ratio, near clipping plane distance, and far clipping plane distance, and set the identity matrix as the initial view matrix; The perspective projection matrix and the initial view matrix are passed to the uniform variable of the texture mapping shader, the vertex array object of the sphere mesh data is bound, and the rendering loop is started to execute the drawing instructions to obtain the panoramic sphere rendering model.

[0030] In this example, a virtual camera model is created in the WebGL context and its position is initialized to the origin of 3D space (0, 0, 0), which is the geometric center of the sphere, to simulate the immersive observer always being in the center of the sphere. Simultaneously, the camera's initial orientation is set to the negative z-axis (facing forward) and the upward direction is set to the positive y-axis (head pointing towards the zenith), forming the camera's initial pose in the spherical coordinate system. Based on the set field of view angle (e.g., 90 degrees), canvas aspect ratio (calculated from the actual pixel width and height of the canvas), and the set near and far planes (e.g., 0.1 and 100.0), a four-dimensional perspective projection matrix is ​​constructed using the projection matrix formula of the right-hand coordinate system, where the matrix element P... 11 = 1 / (canvas aspect ratio × tan(vertical field of view / 2)), P 22 = 1 / tan(vertical field of view / 2), P 33 = (Distant plane distance + Near plane distance) / (Distant plane distance) Near-plane distance), P 34 = (2 × Distance in the far plane × Distance in the near plane) / (Distance in the far plane) Near-plane distance), P 43 = 1, with the rest set to zero, to achieve correct view frustum mapping within clip space. Simultaneously, the initial view matrix is ​​set to an identity matrix to indicate no rotation or displacement, ensuring the viewing direction aligns with the initial orientation of the spherical coordinate system. Two 4×4 uniform matrix variables, `uProjectionMatrix` and `uModelViewMatrix`, are declared in the vertex shader to receive the aforementioned projection and view matrices. During the rendering preparation phase, they are passed to the GPU via `gl.uniformMatrix4fv` to participate in the vertex transformation process. After configuring the rendering parameters, the vertex array object corresponding to the spherical mesh data is bound. This vertex array object contains the spherical vertex coordinates and corresponding texture coordinate data uploaded to the GPU, as well as a mesh index buffer. The rendering loop within the browser's native refresh cycle is started by calling the `requestAnimationFrame` function. In each frame, commands such as `gl.clear` to clear the buffer, `gl.useProgram` to bind the shader, `gl.bindVertexArray` to bind the VAO, and `gl.drawElements` to draw the triangle mesh are executed sequentially, continuously projecting the video sphere mapped with the panoramic texture onto the canvas, dynamically generating a panoramic sphere rendering image that responds to the camera's orientation.

[0031] In each frame rendering loop, pose data is acquired at the VR headset's native refresh rate and executed synchronously with the GPU rendering pipeline to achieve zero network latency for viewpoint updates. This includes: setting the rendering loop frequency of the WebXR session to the VR headset's native refresh rate (11.1 milliseconds for a native refresh rate of 90Hz and 8.3 milliseconds for a native refresh rate of 120Hz); calling the `XRFrame.getViewerPose()` method at the beginning of each rendering frame to read the pose quaternion of the current frame from the headset's inertial measurement unit to obtain frame-synchronized pose data; and immediately converting the frame-synchronized pose data into viewpoint within the current rendering frame. By decoupling the rotation matrix and passing it to the vertex shader of the GPU, the matrix transformation of the sphere vertices, texture sampling of the fragment shader, and rasterization output of the frame buffer are completed within the same frame period. This allows the entire viewpoint update process to be completed within 8.3 to 11.1 milliseconds per frame, resulting in a screen update synchronized with head movement. In the entire process from pose sampling to screen output, no gimbal rotation commands are generated to be sent to the robot dog, no network communication is triggered, and no mechanical response or screen return is waited for from remote devices. The viewpoint switching latency is controlled within the GPU single-frame rendering time and less than the human eye motion-photon delay sensitivity threshold of 20 milliseconds, resulting in a zero-network latency viewpoint update that eliminates motion sickness.

[0032] In one example, the pose quaternion of the VR headset is converted into a gaze-decoupled rotation matrix, and a panoramic sphere rendering model is rendered locally on the GPU based on the gaze-decoupled rotation matrix to obtain a real-time view. Simultaneously, the joystick axis values ​​of the game controller are mapped to control commands and sent to the robot dog terminal, including: In each frame rendering loop, the WebXR pose interface is called to obtain the pose quaternion of head rotation from the VR headset; The three-dimensional rotation matrix is ​​calculated based on the attitude quaternion and extended into a four-dimensional homogeneous transformation matrix to obtain the line-of-sight decoupling rotation matrix. The view decoupled rotation matrix is ​​passed to the shader program to update the view matrix parameters of the virtual camera and vertex transformation is performed on the local GPU, without generating gimbal rotation commands or sending them to the robot dog terminal over the network; The panoramic sphere rendering model is rendered locally using a line-of-sight decoupling rotation matrix to obtain a real-time view. At the same time, the joystick axis values ​​of the game controller are mapped into control commands and sent to the robot dog terminal.

[0033] In this example, after initializing the WebXR immersive session, a `requestAnimationFrame` callback function provided by the `XRSession` object is registered for each frame rendering cycle. Within this function, the `XRFrame` object corresponding to the current frame is retrieved. The `getViewerPose` method of the `XRFrame` is called, passing in a preset reference coordinate system (such as local-floor), to obtain the current viewing posture and pose information of the VR headset. The returned `XRViewerPose` object contains left and right eye views and a unified head pose description. The pose quaternion contained in its `transform` field is extracted from this object. The quaternion is represented by a `DOMPointReadOnly` structure, containing four components (x, y, z, w), representing the user's head rotation state in three-dimensional space. Based on the mathematical transformation relationship from pose quaternion to rotation matrix, it is converted into a 3×3 three-dimensional rotation matrix, where the matrix element calculation formula includes R... 11 =1 2(y 2 +z 2 ), R 12 =2(xy zw), R 13=2(xz+yw), etc., are combined to construct a complete rotation matrix representing the real-time orientation of the user's head. To facilitate vertex transformations in the GPU, the rotation matrix is ​​extended to a 4×4 homogeneous transformation matrix, forming a standard model-view matrix format. The first three rows and three columns are rotation matrix elements, the first three items of the fourth column are zero (indicating no translation), and the fourth row is (0, 0, 0, 1). The resulting matrix is ​​the local camera's view decoupling rotation matrix, driving the virtual camera in WebGL to achieve latency-free view updates. Before each frame is rendered, this rotation matrix is ​​passed as a view matrix parameter to the uModelViewMatrix variable in the bound vertex shader using the gl.uniformMatrix4fv function. This allows the position of each vertex in the shader to change in real-time according to the user's head orientation, achieving visual synchronization with the local camera. Since this gaze change is executed entirely on the browser-side GPU and does not generate any control commands, it does not trigger the movement of the physical gimbal, nor is it sent to the remote robot dog end via the network. This avoids the "motion-photon" mismatch problem caused by the superposition of network latency and motor response latency, eliminating the common causes of motion sickness in traditional VR teleoperation. While performing view matrix updates and GPU rendering, the HTML5 Gamepad API interface is called within the same rendering cycle to poll the status of the currently connected game controller, read the axis values ​​of the left and right joysticks (such as axes[0], axes[1], axes[2]), and perform dead-zone filtering and piecewise linear mapping on them, then map them to the robot dog's linear velocity, lateral velocity, and angular velocity control parameters, respectively. The above control parameters are encapsulated into fixed-length binary control instructions and sent to the robot dog end through the WebRTC RTCDataChannel, where they are parsed and executed by its own control service.

[0034] In one example, a panoramic sphere rendering model is rendered locally on the GPU based on a line-of-sight decoupling rotation matrix to obtain a real-time view. Simultaneously, the joystick axis values ​​of the game controller are mapped to control commands and sent to the robot dog, including: The view decoupling rotation matrix is ​​passed to the vertex shader, the vertex position coordinates of the sphere mesh data are transformed, and the interpolated texture coordinates are output to the fragment shader. In the fragment shader, texture sampling is performed from the corresponding coordinate region of the panoramic video texture based on the interpolated texture coordinates to obtain the texture sampling result; The texture sampling results are output to the frame buffer in the local GPU's rendering pipeline to complete rasterization rendering and obtain the real-time view. Read the joystick axis values ​​of the game controller and map them as forward linear velocity, lateral linear velocity, and yaw angular velocity, then send them to the robot dog terminal.

[0035] In this example, during the WebGL program initialization phase, the view decoupling rotation matrix obtained through quaternion transformation is passed to the GPU in the form of a 4×4 homogeneous matrix and bound to the model-view matrix variable in the vertex shader to simulate the orientation change of the virtual camera. During each frame rendering process, the vertex shader performs matrix transformation operations on the vertex position coordinates of the input spherical mesh. Through multiplication instructions, each vertex is transformed from model coordinate space to clip space, realizing 3D reprojection based on view transformation. At the same time, the vertex shader directly passes the texture coordinates to the varying variable vTexCoord, which is then linearly interpolated and sent to the fragment shader. After receiving the interpolated texture coordinates, the fragment shader performs bilinear interpolation sampling on the bound panoramic video texture object by calling the texture2D function to extract the color value of the current fragment in the equidistant rectangular projection video frame. The GPU automatically uses the color value returned by the fragment shader as the fragment shading output, performs rasterization processing through the local rendering pipeline, and finally writes it to the currently bound frame buffer to generate a complete frame of real-time view. This frame is synchronously refreshed with the refresh cycle of the display or VR headset to achieve low-latency immersive visual feedback. Meanwhile, to achieve remote robot dog movement control, the axis status of the currently connected controller is read in real time through the HTML5 Gamepad API interface during each frame rendering synchronization cycle. The vertical axis value of the left joystick is extracted, inverted, and used as the forward linear velocity input. The horizontal axis value of the left joystick is used as the lateral linear velocity input, and the horizontal axis value of the right joystick is used as the yaw rate input. These three axis values ​​fluctuate within the range of [-1.0, 1.0]. Dead zone filtering and linear remapping operations are first applied to eliminate jitter, and then multiplied by the maximum velocity coefficient (e.g., V1=1.2 m / s, V2=0.5 m / s, Ω=1.5 rad / s) to generate floating-point velocity commands. The velocity values ​​are packaged into a custom binary command frame and sent to the robot dog in unreliable transmission mode through the WebRTC RTCDataChannel. The robot dog's local motion controller parses the command and drives the robot to move or turn, realizing low-latency closed-loop teleoperation between the front-end human-computer interaction and the back-end motion execution.

[0036] In one example, the view decoupling rotation matrix is ​​passed to the vertex shader, the vertex position coordinates of the sphere mesh data are transformed, and the interpolated texture coordinates are output to the fragment shader, including: The view decoupling rotation matrix is ​​passed to the vertex shader through the uniform interface to generate shader matrix parameters; In the vertex shader, matrix multiplication is performed sequentially on the vertex position coordinates of the sphere mesh data with the shader matrix parameters and the perspective projection matrix to obtain the transformed vertex positions. Based on the transformed vertex positions, the texture coordinates of the sphere mesh data are output from the vertex shader to the fragment shader via varying variables.

[0037] In this example, the web-based graphical interface program converts the pose quaternion obtained from the virtual reality headset into a 3D rotation matrix, and then expands the 3D rotation matrix into a 4D homogeneous transformation matrix as the view decoupling rotation matrix for the current frame. This matrix represents the pose transformation of the virtual camera, simulating the viewpoint offset inside the sphere that should be observed after the user's head rotates. The matrix position acquisition function is called to obtain the position of the uniform variable used to receive matrix data in the vertex shader, such as the view matrix variable. The matrix write function is then used to write the calculated view decoupling rotation matrix into the graphics processor in column-major order as the model and view composite matrix parameters for the current rendering frame. During the execution of the vertex shader, for each input sphere vertex position vector, a matrix-vector multiplication operation is performed with the view decoupling matrix to achieve rotational alignment from the object coordinate system to the view coordinate system. This result is then multiplied with the perspective projection matrix to complete the 3D projection transformation from the view coordinate system to clip space, and the final result is written to the built-in output variable. Simultaneously, each vertex in the sphere mesh also contains a set of 2D texture coordinates, representing the vertex's mapped position on the panoramic video texture. The vertex shader directly assigns texture coordinates to the fragment interpolation variable, which is used to pass the texture coordinates to the fragment shader. The graphics pipeline then automatically performs spatial interpolation, enabling each fragment to obtain continuous and smooth texture coordinates.

[0038] In one example, the joystick axis values ​​of the game controller are read and mapped to forward linear velocity, lateral linear velocity, and yaw angular velocity, and then sent to the robot dog. This includes: In the rendering loop, the HTML5 gamepad interface is called to read the axes array of the gamepad, obtain the joystick axis values, perform dead zone filtering on the joystick axis values, and set the axis values ​​with absolute values ​​less than the dead zone threshold to zero to obtain the filtered axis values. The left stick longitudinal axis value in the filtered axis values ​​is converted into forward linear velocity using a velocity mapping function, the left stick lateral axis value is converted into lateral linear velocity, and the right stick lateral axis value is converted into yaw rate. The forward linear velocity, lateral linear velocity, and yaw rate are serialized into data packets according to a binary protocol format and sent to the robot dog terminal through the WebRTC data channel.

[0039] In this example, the HTML5 gamepad interface, specifically the `navigator.getGamepads` function in the browser, is called within each frame rendering cycle. This function returns an array of gamepad objects, where the first non-empty gamepad object is the currently active master gamepad. This object contains an array of `axes` properties, which record the current normalized value for each analog axis, with values ​​ranging from [value range missing]. 1.0 to +1.0 represent the maximum offset direction and neutral state of the joystick, respectively. The longitudinal and lateral axis values ​​of the left joystick (corresponding to indices 1 and 0), and the lateral axis value of the right joystick (corresponding to index 2 or 3) are extracted from the axes attribute array. Dead-zone filtering is then performed on these raw axis values. This involves setting a dead-zone threshold constant (e.g., 0.15). Any axis value whose absolute value is less than this threshold is considered invalid input and reset to zero. If the absolute value is greater than the threshold, the dead-zone effect is removed, and the axis is linearly remapped to the full-scale range, resulting in the filtered valid axis values. The three filtered axis values ​​are then mapped to the robot dog's three motion control variables: the longitudinal axis value of the left joystick is mapped to forward linear velocity after sign inversion, the lateral axis value to lateral linear velocity, and the lateral axis value of the right joystick to yaw rate. A nonlinear velocity mapping function, such as a piecewise linear function based on three slopes or a cubic spline curve, is employed to enable delicate control of small offsets and rapid response to large offsets. Three floating-point velocity parameters are calculated: V1 represents the forward linear velocity, V2 represents the lateral linear velocity, and Ω represents the yaw rate. These three velocity values ​​are used as the core parameters for control commands and packaged into fixed-length data frames according to a preset binary protocol format. For example, the data frame begins with a two-byte fixed header, followed by a one-byte message type identifier, then three floating-point numbers, each occupying 4 bytes in IEEE 754 standard format. An additional byte is appended for the velocity mode identifier or control flag, and finally, a two-byte cyclic redundancy check (CRC) code is appended as the frame tail for the receiver to verify integrity. The entire data packet length is fixed at a certain number of bytes (e.g., 20 bytes). After data frame serialization, the data packet is sent to the remote robot dog control terminal in an unreliable transmission mode (i.e., no ordering, no retransmission) through the established WebRTC data channel object, ensuring low-latency command delivery. A multi-frame redundancy transmission strategy is adopted during transmission, such as repeatedly transmitting the same control packet within three consecutive frame periods, to improve the command delivery success rate. After receiving the command packet, the robot dog parses the values ​​of V1, V2, and Ω, and drives the motion controller to execute the corresponding forward, lateral, or turning actions.

[0040] After receiving the data packet, the robot dog converts the velocity parameters into joint torque commands to drive the quadrupedal movement through inverse kinematics and gait generation. This includes: deserializing and parsing the received data packet to extract the forward linear velocity, lateral linear velocity, and yaw rate; passing the velocity parameters to the inverse kinematics solver; combining the robot dog's current center of mass position, fuselage pitch and roll angles, and leg joint angles; calculating the desired position and velocity of each leg end in the world coordinate system using the Jacobian matrix to obtain the foot trajectory; and calling the gait generator according to the current gait mode, setting the diagonal legs to move in phase with a phase difference of 18 for diagonal trotting gait. For a slow walking gait, the four legs are raised sequentially with a 90-degree phase difference. Periodic swing phase and support phase timing are generated for each leg. In the swing phase, the leg lifts and steps forward, and in the support phase, the leg touches the ground and propels the body to move, thus obtaining the gait phase planning. The foot movement trajectory and gait phase planning are input into the leg controller. The desired foot position is converted into the desired angles of the hip, knee and ankle joints through inverse kinematics. The PD controller calculates the target torque command for each joint and sends it to the motor driver. The motor driver executes FOC field-oriented control to drive the three-phase windings of the brushless motor, thus obtaining the joint movement execution.

[0041] When performing dead-zone filtering on the joystick axis values, zeroing and remapping transformations are executed. A piecewise velocity mapping function is used to achieve fine control at low speeds and fast response at high speeds. This includes: setting a dead-zone threshold of 0.15; checking if the absolute value of the read joystick axis value is less than the dead-zone threshold; if the absolute value is less than 0.15, the axis value is forced to zero to eliminate mechanical jitter in the joystick's neutral position; if the absolute value is greater than or equal to 0.15, a remapping transformation is performed, subtracting the sign of the axis value from the dead-zone threshold, dividing by 1 and subtracting the dead-zone threshold, so that the axis values ​​outside the dead zone are linearly remapped to the full-scale range from -1 to +1, obtaining the filtered axis values; a piecewise cubic spline mapping function is used for velocity conversion on the filtered axis values, and a slope is used when the absolute value of the axis value is less than 0.3 in the low-speed region. A coefficient of 0.4 enables fine-tuning. In the medium-speed range, when the absolute value of the axis value is between 0.3 and 0.7, a slope coefficient of 0.8 is used. In the high-speed range, when the absolute value of the axis value is greater than 0.7, a slope coefficient of 1.2 is used to achieve rapid response. The speed contribution of each range is calculated, accumulated, and multiplied by the maximum speed limit to obtain the segmented mapped speed value. The forward linear velocity in the segmented mapped speed value is limited to the range of -0.8 to +1.2 m / s, the lateral linear velocity is limited to the range of -0.5 to +0.5 m / s, and the yaw rate is limited to the range of -1.5 to +1.5 radians / s. This allows the operator to obtain high-precision control when inputting small joystick movements and rapid response when inputting large movements, resulting in speed control parameters that are suitable for different operating scenarios.

[0042] Reference Figure 2This embodiment provides a WebXR-based panoramic immersive robot teleoperation system, including: Image fusion module 1 is used to perform distortion correction and stitching fusion on the original images captured by the double fisheye lens to obtain panoramic image frames; Transmission decoding module 2 is used to encode panoramic image frames into video streams and transmit them to the browser, and decode them on the browser to obtain panoramic video textures. Texture mapping module 3 is used to map panoramic video textures onto the inner surface of the sphere and set the virtual camera to the center of the sphere to build a panoramic sphere rendering model. The real-time rendering module 4 is used to convert the pose quaternion of the VR headset into a gaze decoupling rotation matrix, and perform local GPU rendering on the panoramic sphere rendering model based on the gaze decoupling rotation matrix to obtain a real-time view. At the same time, it maps the joystick axis values ​​of the game controller into control commands and sends them to the robot dog terminal.

[0043] In this embodiment, the specific implementation of each unit in the above system embodiment is described in the above method embodiment, and will not be repeated here.

[0044] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, system, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, system, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, system, article, or method that includes that element.

[0045] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A WebXR-based panoramic immersive robot teleoperation method, characterized in that, include: The original images captured by the double fisheye lens are subjected to distortion correction and stitching fusion to obtain panoramic image frames; The panoramic image frames are encoded into a video stream and transmitted to the browser, and the panoramic video texture is obtained by decoding on the browser. The panoramic video texture is mapped onto the inner surface of the sphere and a virtual camera is set at the center of the sphere to construct a panoramic sphere rendering model. The VR headset's pose quaternion is converted into a gaze decoupling rotation matrix, and the panoramic sphere rendering model is rendered locally on the GPU based on the gaze decoupling rotation matrix to obtain a real-time view. At the same time, the joystick axis values ​​of the game controller are mapped into control commands and sent to the robot dog terminal.

2. The WebXR-based panoramic immersive robot teleoperation method according to claim 1, characterized in that, The original images captured by the double fisheye lens are subjected to distortion correction and stitching fusion to obtain panoramic image frames, including: Distortion correction is performed on the original image acquired by the double fisheye lens based on the camera intrinsic parameter matrix and distortion coefficients to obtain the distortion-corrected image; Feature points are extracted in the overlapping region of the distortion-corrected image and image registration is performed using a homography matrix to obtain a registered image. The registered image is then seamlessly stitched together in the overlapping region to obtain a spherical image. The spherical coordinates of the spherical image are converted into the horizontal and vertical pixel coordinates of the planar image to obtain a panoramic image frame.

3. The WebXR-based panoramic immersive robot teleoperation method according to claim 1, characterized in that, Encoding the panoramic image frames into a video stream and transmitting it to the browser, then decoding the panoramic video texture on the browser side, includes: The panoramic image frames are compressed into a video stream, the NAL units of the video stream are encapsulated into RTP data packets and encrypted with SRTP, and transmitted to the browser through a peer-to-peer connection established by WebRTC. The browser receives the RTP data packet through the RTCPeerConnection interface and sequentially performs SRTP decryption and RTP decapsulation to decode it into video frame data. The video frame data is then uploaded to a WebGL texture object to obtain a panoramic video texture.

4. The WebXR-based panoramic immersive robot teleoperation method according to claim 1, characterized in that, Mapping the panoramic video texture onto the inner surface of the sphere and setting the virtual camera at the center of the sphere, a panoramic sphere rendering model is constructed, including: The vertex position coordinates and texture coordinates of the sphere are generated by subdividing the latitude and longitude grid. The vertex position coordinates and texture coordinates are then uploaded to the vertex buffer object to obtain the sphere grid data. Compile the vertex shader and fragment shader and link them into a shader program. Bind the panoramic video texture to a WebGL texture unit as the sampling source for the fragment shader to obtain a texture mapping shader. The virtual camera is set at the origin of the sphere's center, and the perspective projection matrix and initial view matrix are configured. By combining the sphere mesh data and the texture mapping shader, a panoramic sphere rendering model is obtained.

5. The WebXR-based panoramic immersive robot teleoperation method according to claim 4, characterized in that, The virtual camera is positioned at the origin of the sphere's center, and the perspective projection matrix and initial view matrix are configured. Combining the sphere mesh data and the texture mapping shader, a panoramic sphere rendering model is obtained, including: Set the position of the virtual camera to the origin of the sphere's center and set the initial orientation to forward and the upward direction to the zenith to obtain the camera's initial posture; Calculate the perspective projection matrix based on the field of view, canvas aspect ratio, near clipping plane distance, and far clipping plane distance, and set the identity matrix as the initial view matrix; The perspective projection matrix and the initial view matrix are passed to the uniform variable of the texture mapping shader, the vertex array object of the sphere mesh data is bound, and the rendering loop is started to execute the drawing instructions to obtain the panoramic sphere rendering model.

6. The WebXR-based panoramic immersive robot teleoperation method according to claim 5, characterized in that, The VR headset's pose quaternion is converted into a gaze-decoupled rotation matrix, and the panoramic sphere rendering model is rendered locally using the gaze-decoupled rotation matrix to obtain a real-time view. Simultaneously, the joystick axis values ​​of the game controller are mapped to control commands and sent to the robot dog terminal, including: In each frame rendering loop, the WebXR pose interface is called to obtain the pose quaternion of head rotation from the VR headset; The three-dimensional rotation matrix is ​​calculated based on the attitude quaternion and expanded into a four-dimensional homogeneous transformation matrix to obtain the line-of-sight decoupling rotation matrix. The view decoupling rotation matrix is ​​passed to the shader program to update the view matrix parameters of the virtual camera and vertex transformation is performed on the local GPU, while no gimbal rotation command is generated and no data is sent to the robot dog terminal via the network; Based on the line-of-sight decoupling rotation matrix, the panoramic sphere rendering model is rendered locally using GPU to obtain a real-time view. At the same time, the joystick axis values ​​of the game controller are mapped into control commands and sent to the robot dog terminal.

7. The WebXR-based panoramic immersive robot teleoperation method according to claim 6, characterized in that, Based on the aforementioned line-of-sight decoupling rotation matrix, the panoramic sphere rendering model is rendered locally using GPU to obtain a real-time view. Simultaneously, the joystick axis values ​​of the game controller are mapped to control commands and sent to the robot dog terminal, including: The line-of-sight decoupling rotation matrix is ​​passed to the vertex shader, the vertex position coordinates of the sphere mesh data are transformed, and the interpolated texture coordinates are output to the fragment shader. In the fragment shader, texture sampling is performed from the corresponding coordinate region of the panoramic video texture according to the interpolated texture coordinates to obtain the texture sampling result; The texture sampling results are output to the frame buffer in the local GPU's rendering pipeline to complete rasterization rendering and obtain a real-time view. Read the joystick axis values ​​of the game controller and map them as forward linear velocity, lateral linear velocity, and yaw angular velocity, then send them to the robot dog terminal.

8. The WebXR-based panoramic immersive robot teleoperation method according to claim 7, characterized in that, The line-of-sight decoupling rotation matrix is ​​passed to the vertex shader, the vertex position coordinates of the sphere mesh data are transformed, and the interpolated texture coordinates are output to the fragment shader, including: The line-of-sight decoupling rotation matrix is ​​passed to the vertex shader through the uniform interface to generate shader matrix parameters; In the vertex shader, matrix multiplication is performed sequentially on the vertex position coordinates of the sphere mesh data with the shader matrix parameters and the perspective projection matrix to obtain the transformed vertex positions. Based on the transformed vertex positions, the texture coordinates of the sphere mesh data are output from the vertex shader to the fragment shader via varying variables.

9. The WebXR-based panoramic immersive robot teleoperation method according to claim 7, characterized in that, The joystick axis values ​​of the game controller are read and mapped to forward linear velocity, lateral linear velocity, and yaw angular velocity, which are then sent to the robot dog terminal, including: In the rendering loop, the HTML5 gamepad interface is called to read the axes array of the gamepad, obtain the joystick axis values, perform dead zone filtering on the joystick axis values, and set the axis values ​​with absolute values ​​less than the dead zone threshold to zero to obtain the filtered axis values. The left joystick longitudinal axis value in the filtered axis values ​​is converted into forward linear velocity through a velocity mapping function, the left joystick lateral axis value is converted into lateral linear velocity, and the right joystick lateral axis value is converted into yaw rate. The forward linear velocity, the lateral linear velocity, and the yaw rate are serialized into data packets according to a binary protocol format and sent to the robot dog terminal through the WebRTC data channel.

10. A WebXR-based panoramic immersive robot teleoperation system, characterized in that, The steps for implementing the WebXR-based panoramic immersive robot teleoperation method according to any one of claims 1 to 9 include: The image fusion module is used to perform distortion correction and stitching fusion on the original images captured by the double fisheye lens to obtain panoramic image frames; The transmission decoding module is used to encode the panoramic image frame into a video stream and transmit it to the browser, and decode it on the browser to obtain the panoramic video texture. The texture mapping module is used to map the panoramic video texture onto the inner surface of the sphere and set the virtual camera to the center of the sphere to construct a panoramic sphere rendering model. The real-time rendering module is used to convert the pose quaternion of the VR headset into a gaze decoupling rotation matrix, and perform local GPU rendering on the panoramic sphere rendering model based on the gaze decoupling rotation matrix to obtain a real-time view. At the same time, it maps the joystick axis values ​​of the game controller into control commands and sends them to the robot dog terminal.

Images