Teleoperated augmented reality method, apparatus, and storage medium

Abstract

This application relates to the field of robotics technology and discloses a teleoperated augmented reality method, device, and storage medium. The method includes: constructing and updating a 3D semantic map containing the real-time pose of a target object based on RGB images and depth images acquired by an acquisition device mounted on a remote robot, combined with the robot's own 3D model; generating an ideal mask for the target object in an unoccluded state based on the real-time pose of the target object's 3D mesh model in the dynamic 3D semantic map and the current pose of the acquisition device; comparing the ideal mask with the actual mask of the target object obtained by instance segmentation of the RGB image to obtain an occluded region mask, and extracting contour polygons representing the boundary of the occluded region; performing depth value adjustment operations on the 3D mesh model of the target object within the pixel area formed by the contour polygons, and rendering it on the RGB image to present the perspective effect of the occluded region.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and more particularly to a teleoperated augmented reality method, apparatus, and storage medium. Background Technology

[0002] In remote control (teleoperation) systems, such as remote surgery, on-orbit space services, or deep-sea exploration, operators heavily rely on transmitted visual images for operational decisions. Existing visual enhancement solutions are generally based on single-modal RGB images and static prior models, and their core deficiency lies in their severely insufficient dynamic occlusion handling capabilities: when operating tools (such as mechanical grippers or surgical forceps) or environmental obstacles (such as intravehicular supports or human tissue) partially obscure key targets (such as parts to be repaired or lesion boundaries), existing systems cannot recover the visual information of the obscured portion, leading to blind operation by the operator, which greatly increases the operational risk and difficulty. Summary of the Invention

[0003] In view of this, embodiments of this application provide a teleoperation augmented reality method, apparatus and storage medium, which can achieve accurate visual compensation for key targets that are obscured during teleoperation, thereby significantly improving the safety of operation and the efficiency of human-machine collaboration in dynamic and complex environments.

[0004] In a first aspect, embodiments of this application provide a teleoperated augmented reality method, including: Based on the RGB and depth images collected by the acquisition device mounted on the remote robot, a 3D semantic map containing the real-time pose of the target object is constructed and updated by combining the robot's own 3D model. Based on the real-time pose of the target object's 3D mesh model in the 3D semantic map and the current pose of the acquisition device that acquired the RGB image, an ideal mask for the target object under unobstructed conditions is generated. The ideal mask is compared with the actual mask of the target object obtained by instance segmentation of the RGB image to obtain the mask of the occluded area, and the contour polygon representing the boundary of the occluded area is extracted. Within the pixel area formed by the outline polygon, a depth value adjustment operation is performed on the three-dimensional mesh model of the target object, and based on the adjusted three-dimensional mesh model, rendering is performed on the RGB image to present the perspective effect of the occluded area.

[0005] In an optional implementation, the construction and updating of a 3D semantic map containing the real-time pose of the target object, based on the RGB and depth images acquired by the acquisition device mounted on the remote robot and combined with the robot's own 3D model, includes: Based on the RGB and depth images synchronously acquired by the RGB and depth cameras mounted on the remote robot, a 3D point cloud with color information is generated, and the point cloud of the robot's own 3D model is removed from the 3D point cloud with color information to form the target point cloud. The target point cloud is fused with the global 3D map to update the 3D semantic map, and the 3D mesh model of the target object is registered with the updated 3D semantic map to obtain the real-time pose of the 3D mesh model.

[0006] In an optional implementation, a 3D point cloud with color information is generated based on the acquired RGB and depth images, and the point cloud of the robot's own 3D model is removed from the 3D point cloud with color information to form the target point cloud, including: The robot's own three-dimensional model is obtained by solving the data from the robot joint encoder. Using the camera intrinsic parameters of the camera, the extrinsic parameter matrix obtained by joint calibration of the camera and the depth camera, and the depth image aligned with the pixels of the RGB image, the 3D point cloud with color information corresponding to the RGB image is generated; The robot's own 3D model is projected onto the imaging view of the RGB image, and the point cloud corresponding to the robot's own 3D model is removed from the 3D point cloud with color information to obtain the target point cloud.

[0007] In an optional implementation, generating an ideal mask for the target object under unobstructed conditions based on the real-time pose of the target object's 3D mesh model in the 3D semantic map and the current pose of the acquisition device acquiring the RGB image includes: The three-dimensional mesh model of the target object is mapped onto the RGB image plane through perspective projection transformation based on the camera intrinsic parameters of the RGB camera and the current pose of the RGB camera. Rasterization filling is performed on the triangular facets of the projected 3D mesh model to generate a binary image with the same resolution as the RGB image; The pixel region with a pixel value of 1 in the binary image is used as the ideal mask for the target object under unobstructed conditions.

[0008] In an optional implementation, the process of performing instance segmentation on the RGB image to obtain the actual mask of the target object includes: Semantic segmentation is performed on the RGB image using a lightweight backbone network, and a semantic segmentation map including the target object category is output. Based on the semantic segmentation map, the connected regions of the category to which the target object belongs are extracted to generate the region of interest of the target object; Perform fine instance segmentation on the region of interest and output the actual mask of the target object.

[0009] In an optional implementation, comparing the ideal mask with the actual mask of the target object obtained by instance segmentation of the RGB image to obtain the mask of the occluded region includes: Perform a logical difference operation between the ideal mask and the actual mask to identify the set of pixels that are foreground pixels in the ideal mask but background pixels in the actual mask, and determine the set of pixels as the mask of the occluded area.

[0010] In an optional implementation, the lightweight backbone network includes an encoder and a decoder; The step of performing semantic segmentation on the RGB image using a lightweight backbone network to output a semantic segmentation map including the target object category includes: The encoder performs hierarchical feature extraction on the RGB image and outputs a multi-scale feature map. The multi-scale feature map is input into the decoder to process the multi-scale feature map through three parallel paths: global average pooling, dilated convolution, and identity mapping. The output results of each path are then fused to obtain a fused semantic feature map. After performing dimensionality reduction, upsampling, and normalization operations on the fused semantic feature map, the class probability distribution of each pixel in the fused semantic feature map is obtained; The category with the highest probability corresponding to each pixel is used as the label of the current pixel, thereby generating the semantic segmentation map.

[0011] In an optional implementation, performing fine instance segmentation on the region of interest and outputting the actual mask of the target object includes: The RGB image block corresponding to the region of interest is concatenated with the feature block at the corresponding position in the multi-scale feature map to obtain the concatenated feature block, and the concatenated feature is input into the instance controller sub-network to output dynamic filter weights; The dynamic filter weights are input into a mask generation network with fixed weights, and the mask generation network performs modulation operations on the spliced ​​feature blocks based on the dynamic filter weights to generate a mask response map. The mask response map is subjected to activation operation and fixed threshold binarization processing in sequence to generate the actual mask of the target object.

[0012] In an optional implementation, performing a depth value adjustment operation on the three-dimensional mesh model of the target object includes: Within the pixel region defined by the outline polygon, the depth value of the corresponding pixel of the three-dimensional mesh model of the target object is reduced to a value lower than the depth value of the real scene; Outside the pixel region defined by the outline polygon, the standard depth value of the corresponding pixel of the three-dimensional mesh model of the target object is maintained.

[0013] In an optional implementation, rendering the RGB image based on the adjusted 3D mesh model to present a perspective effect of the occluded area includes: Perform a template buffer setting operation to load the outline polygon as a template into the template buffer of the graphics rendering pipeline, so as to limit subsequent rendering only in the RGB image and within the pixel area defined by the outline polygon. Perform a depth test configuration operation, setting the depth comparison function of the graphics rendering pipeline to a less than comparison function, so that the depth value of the newly rendered pixel is less than the existing depth value at the corresponding position in the depth buffer; Within the pixel area defined by the outline polygon, the three-dimensional mesh model is rendered with semi-transparent blending to present the perspective effect of the occluded area on the RGB image.

[0014] In an optional implementation, when presenting the perspective effect of the occluded area on the RGB image, the rendering of a navigation guide curve is also included; wherein the navigation guide curve is a smooth two-dimensional curve located in the plane of the RGB image, the navigation guide curve is determined by the collision-free path between the current position of the robot end effector and the task target point, and the visual representation includes a color distribution that changes continuously along the length of the curve.

[0015] Secondly, embodiments of this application provide a teleoperated augmented reality device, comprising: The 3D semantic map acquisition module is used to construct and update a 3D semantic map containing the real-time pose of the target object based on the RGB images and depth images collected by the acquisition device mounted on the remote robot and combined with the robot's own 3D model. An ideal mask generation module is used to generate an ideal mask for the target object under unobstructed conditions based on the real-time pose of the target object's 3D mesh model in the 3D semantic map and the current pose of the acquisition device that acquires the RGB image. The contour polygon acquisition module is used to compare the ideal mask with the actual mask of the target object obtained by instance segmentation of the RGB image to obtain the mask of the occluded area and extract the contour polygon representing the boundary of the occluded area. An enhanced display module is used to perform a depth value adjustment operation on the three-dimensional mesh model of the target object within the pixel area defined by the outline polygon, and to render the adjusted three-dimensional mesh model on the RGB image to present the perspective effect of the occluded area.

[0016] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed on a processor, implements the aforementioned teleoperation augmented reality method.

[0017] The embodiments of this application have the following beneficial effects: This application constructs a three-dimensional semantic map that integrates multi-source perception information, providing a unified, stable, and updatable spatial representation basis for visual enhancement in remote operation scenarios; in addition, by registering the three-dimensional mesh model of the target object with the three-dimensional semantic map to obtain its real-time pose, and combining it with the current pose of the RGB camera to generate an ideal mask under unobstructed conditions, the system has the ability to accurately model the theoretical visible range of the target object; by comparing the ideal mask with the actual mask obtained from instance segmentation, the occluded area is identified, and its contour polygon is extracted, thereby providing a spatial constraint basis for subsequent enhanced rendering; furthermore, the depth value adjustment operation is performed on the three-dimensional mesh model within the pixel area defined by the contour polygon, and the adjusted model is rendered on the RGB image, so that the occluded target structure can be selectively presented in the real picture, which not only retains the realism of the visible part in the original image, but also effectively compensates for the loss of visual information caused by physical occlusion, thereby significantly improving the operator's situational awareness and operational reliability in complex occlusion environments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A first flowchart of the teleoperation augmented reality method according to an embodiment of this application is shown; Figure 2 A second flowchart of the teleoperation augmented reality method according to an embodiment of this application is shown; Figure 3 A schematic diagram of the third process of the teleoperation augmented reality method according to an embodiment of this application is shown; Figure 4 A schematic diagram of the fourth process of the teleoperation augmented reality method according to an embodiment of this application is shown; Figure 5 A fifth flowchart of the teleoperation augmented reality method according to an embodiment of this application is shown; Figure 6 A sixth flowchart of the teleoperation augmented reality method according to an embodiment of this application is shown; Figure 7 The seventh flowchart of the teleoperation augmented reality method according to an embodiment of this application is shown; Figure 8 The eighth flowchart of the teleoperation augmented reality method according to an embodiment of this application is shown; Figure 9 The ninth flowchart of the teleoperation augmented reality method according to an embodiment of this application is shown; Figure 10 A schematic diagram of a teleoperated augmented reality device according to an embodiment of this application is shown. Detailed Implementation

[0020] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0021] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0023] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0024] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0025] The following describes the teleoperation augmented reality method using specific examples.

[0026] Figure 1 A schematic flowchart of a teleoperation augmented reality method according to an embodiment of this application is shown. Exemplarily, the teleoperation augmented reality method includes steps S110-S140: Step S110: Based on the RGB images and depth images collected by the acquisition device mounted on the remote robot, and combined with the robot's own 3D model, construct and update a 3D semantic map containing the real-time pose of the target object.

[0027] Exemplary, such as Figure 2 As shown, step S110 specifically includes steps S210-S220: Step S210: Based on the RGB images and depth images synchronously acquired by the RGB camera and depth camera mounted on the remote robot, a 3D point cloud with color information is generated, and the point cloud of the robot's own 3D model is removed from the 3D point cloud with color information to form the target point cloud.

[0028] This embodiment applies to the field of robotics. To acquire RGB and depth images, an RGB camera and a depth camera can be mounted on the robot's head. The depth camera is used to capture depth images of the current scene. This depth camera can be a depth sensing device based on structured light or time-of-flight principles, capable of directly acquiring the distance from each pixel in the scene to the camera. The RGB camera can acquire high-resolution color 2D images, providing rich texture and color information, and is a primary data source for object recognition and segmentation. Therefore, in this embodiment, an RGB camera is used to acquire RGB images of the current scene. To reduce motion blur, a global shutter sensor can be used for the RGB camera.

[0029] For the acquired RGB and depth images, they need to be physically aligned, meaning they have the same image resolution and pixel coordinate mapping relationship.

[0030] Furthermore, synchronous acquisition can be achieved through a hardware triggering mechanism, where a unified master clock signal simultaneously triggers the exposure actions of both the RGB and depth cameras, and synchronously samples the output pulse signals of the encoders at each joint of the robot's body. This hardware triggering mechanism ensures that the RGB images, depth images, and joint encoder data have strictly consistent timestamps.

[0031] In other implementations, synchronous acquisition can also be achieved through software timestamp alignment. That is, each frame of RGB image, depth image and each group of joint encoder data is stamped with a high-precision system timestamp at the acquisition end, and interpolation matching is performed based on the timestamps in subsequent processing to align the three on the logical time axis.

[0032] Exemplary, such as Figure 3 As shown, based on the acquired RGB and depth images, a 3D point cloud with color information is generated, and the point cloud of the robot's own 3D model is removed from the 3D point cloud with color information to form the target point cloud, including steps S310-S330: Step S310: Obtain the robot's own three-dimensional model based on the robot joint encoder data.

[0033] The robot joint encoder data includes the angle values ​​of each joint of the robot, the number of which corresponds to the robot's degrees of freedom. The calculation process employs a forward kinematics model, where the joint angles are substituted into the standard kinematic equations based on the Denavit-Hartenberg parameters (DH parameters) of each link, progressively calculating the 3D position and orientation of the robot's base, torso, upper arm, forearm, wrist, and end effector in the world coordinate system. The final output 3D model of the robot itself is a set of simplified geometric shapes, including a cylinder representing the upper arm, a cuboid representing the torso, and a wireframe model representing the end effector. This model contains no textures or detailed surfaces, retaining only the topological structure sufficient to support spatial projection and occlusion detection.

[0034] Step S320: Using the camera intrinsic parameters of the RGB camera, the extrinsic parameter matrix obtained by joint calibration of the RGB camera and the depth camera, and the depth image aligned with the pixels of the RGB image, generate a 3D point cloud with color information corresponding to the RGB image.

[0035] For example, the camera intrinsics of an RGB camera include focal length ( ), principal point coordinates ( The extrinsic parameters include radial and tangential distortion coefficients; the extrinsic parameter matrix is ​​a 4×4 homogeneous transformation matrix. This matrix represents the rotation and translation relationship between the depth camera coordinate system and the RGB camera coordinate system. It is obtained through joint calibration using a checkerboard calibration board in the overlapping area of ​​the two cameras' fields of view. For any pixel coordinate (u,v) in the RGB image, its corresponding depth value in the depth image is denoted as d(u,v). The coordinates (X,Y,Z) of this pixel in three-dimensional space are determined by the following formula: , , .

[0036] In this step, all pixels (u,v) in the RGB image are back-projected one by one according to the above formula to obtain a set of three-dimensional points. Then, the RGB image pixel color corresponding to each point is added to that point as its attribute, thus forming a three-dimensional point cloud with color information.

[0037] Step S330: Project the robot's own 3D model onto the imaging view of the RGB image, and remove the point cloud corresponding to the robot's own 3D model from the 3D point cloud with color information to obtain the target point cloud.

[0038] The projection process in this step is based on the robot's end-effector pose and the RGB camera pose obtained above. The vertex coordinates of each simplified geometric object in the robot's 3D model are used for perspective projection through the intrinsic and extrinsic parameters of the RGB camera, mapping them onto the RGB image plane to generate the 2D occupied region of each geometric object on the image. Subsequently, a spatial index query is performed on the 3D point cloud with color information: for each 3D point... = By projecting it backwards onto the RGB image plane, we can obtain its corresponding pixel coordinates. If the coordinate lies within the two-dimensional occupied area of ​​any robot geometry, and its depth value... If the difference between the depth value and the corresponding location in the depth map of that area is less than a preset threshold (e.g., 2 cm), the point is determined to belong to the robot itself and is removed. The remaining point cloud after this processing is the target point cloud containing only 3D information of non-robot entities in the environmental scene.

[0039] It should be noted that the removal in this embodiment is a coordinate-level deletion operation performed at the point cloud data level: for each 3D point determined to belong to the robot itself... = This method directly removes the point and its corresponding RGB color attribute from the 3D point cloud set containing color information, without performing indirect processing such as zeroing out its depth value or masking. This approach ensures that the final target point cloud is a set of independent, complete 3D spatial sampling points that do not contain any robot body coordinates.

[0040] Step S220: The target point cloud is fused with the global 3D map to update the 3D semantic map, and the 3D mesh model of the target object is registered with the updated 3D semantic map to obtain the real-time pose of the 3D mesh model.

[0041] The global 3D map is continuously maintained by a visual-inertial SLAM (V-SLAM) system. Its data structure is a sparse 3D occupancy grid based on voxel hashing. Each voxel stores its spatial location in the world coordinate system, whether it is occupied, and an optional semantic label. The coordinates of the target point cloud also need to be uniformly normalized to the same coordinate system as the global 3D map.

[0042] In some implementations, fusing the target point cloud with the global 3D map specifically includes fusing each 3D point in the target point cloud... = The data is mapped to its corresponding voxel index; if the voxel is not yet occupied, it is marked as newly occupied; if it is already occupied, its occupancy probability is updated according to a preset fusion strategy. The fusion strategy can use a weighted average method, such as a weight of 0.7 for the contribution of the new point cloud to the voxel occupancy status and a weight of 0.3 for the retention of the historical map, thus ensuring that the global 3D map can respond to environmental changes while maintaining long-term stability. After fusion, the global 3D map is updated to a dynamic 3D map reflecting the latest state of the current scene.

[0043] A 3D semantic map is an enhanced version built upon a dynamic 3D map. Its construction process includes: projecting the actual mask of the target object obtained from instance segmentation back into 3D space using camera pose and depth information to generate a point cloud cluster of the target object in 3D space; subsequently, injecting this point cloud cluster along with its corresponding category label (e.g., container A, pallet B, etc.) into the corresponding voxel region of the dynamic 3D map to form a semantically labeled 3D point cloud. For dynamic obstacles (e.g., moving AGVs), the system further fits a linear motion model based on their historical positions over multiple frames to predict their trajectory within a preset time (e.g., 1 second), and injects this predicted trajectory as a temporary semantic label into the map, enabling the dynamic 3D semantic map to possess short-term perception capabilities for moving targets.

[0044] Registration of a 3D mesh model of a target object with a dynamic 3D semantic map involves spatially aligning a pre-stored 3D mesh model of the target object with point cloud clusters of the corresponding category in the dynamic 3D semantic map to determine the six-degree-of-freedom pose of the 3D mesh model in the world coordinate system. This registration process employs an iterative nearest-point algorithm. The initial pose is roughly estimated from the robot's current pose and prior task knowledge. Subsequently, the algorithm minimizes the sum of Euclidean distances from vertices on the surface of the 3D mesh model to the same type of point cloud clusters in the dynamic 3D semantic map until convergence, outputting the final rotation matrix R and translation vector t. This registration result represents the real-time pose of the 3D mesh model.

[0045] Step S120: Based on the real-time pose of the target object's 3D mesh model in the dynamic 3D semantic map and the current pose of the acquisition device that acquired the RGB image, generate an ideal mask for the target object under unobstructed conditions.

[0046] Among them, the ideal mask is used to characterize the theoretical visible range of a target object under ideal conditions of no visual occlusion, lighting, and texture.

[0047] In some implementations, such as Figure 4 As shown, step S120 specifically includes steps S410-S430: Step S410: The 3D mesh model of the target object is mapped to the RGB image plane through perspective projection transformation based on the camera intrinsic parameters of the RGB camera and the current pose of the RGB camera.

[0048] The current pose of the RGB camera can be output in real time by the visual-inertial SLAM system, which can be a 4×4 homogeneous transformation matrix. This represents the rotation and translation of the RGB camera coordinate system relative to the world coordinate system; based on the above, it can be known that the camera intrinsic parameters include focal length ( ), principal point coordinates ( The distortion coefficients are obtained through offline calibration. The 3D mesh model is composed of triangular facets, and each vertex has 3D coordinates (X, Y, Z). The perspective projection transformation can be performed in the following order: first, the 3D vertices are transformed from the model coordinate system to the world coordinate system through real-time pose transformation, then transformed to the camera coordinate system through the current pose transformation, and finally projected onto the RGB image plane through the pinhole camera model and distortion correction to obtain 2D pixel coordinates (u, v).

[0049] Step S420: Perform rasterization filling on the triangular facets of the projected 3D mesh model to generate a binary image with the same resolution as the RGB image.

[0050] Step S430: Use the pixel region with a pixel value of 1 in the binary image as the ideal mask for the target object under unobstructed conditions.

[0051] As an example, a standard scanline rasterization algorithm can be used to calculate the axis-aligned bounding rectangle of each projected triangular facet on the RGB image plane. Each pixel within this bounding rectangle is then traversed, and barycentric coordinate interpolation is used to determine whether the pixel is inside the triangular facet. If it is determined to be inside, the corresponding position of that pixel in the output binary image is set to 1; otherwise, it is set to 0. After all triangular facets have been processed, a binary image with the same resolution as the RGB image (e.g., 640×480) is obtained, where pixels with a value of 1 constitute one or more connected regions, representing the theoretically visible contour of the target object under unobstructed conditions, i.e., the ideal mask.

[0052] The ideal mask can be understood as a two-dimensional array with the exact same size as the RGB image, where each element takes a value of 0 or 1. The set of pixels with a value of 1 is the pixel-level definition of the ideal mask. This ideal mask refers to the precise pixel range that the target object should occupy in the RGB image under the assumption that the target object is not occluded by any other object and that the imaging conditions are ideal. It should be noted that the ideal mask itself does not change with the actual image content; it only updates dynamically with the real-time pose of the 3D mesh model and the current pose of the RGB camera.

[0053] Step S130: Compare the ideal mask with the actual mask of the target object obtained by instance segmentation of the RGB image to obtain the mask of the occluded area, and extract the contour polygon representing the boundary of the occluded area.

[0054] This step mainly involves identifying the precise spatial location of missing visual information by using pixel-level differences between the ideal mask and the actual mask.

[0055] In some implementations, such as Figure 5 As shown, the process of obtaining the actual mask of the target object by instance segmentation of the RGB image includes steps S510-S530: Step S510: Perform semantic segmentation on the RGB image through the backbone network and output a semantic segmentation map including the target object category.

[0056] The lightweight backbone network comprises an encoder and a decoder. The encoder extracts multi-level feature representations from the RGB image, while the decoder fuses and upsamples the features output by the encoder to recover spatial details and output pixel-level class predictions. The decoder uses MobileNetV3-Small as its backbone architecture. The semantic segmentation map is a two-dimensional image with the same dimensions as the RGB image, where each pixel has a class label. The label set includes, but is not limited to, background, target objects (such as boxes or pallets), robot body parts (such as grippers or arms), and environmental elements (such as the ground or shelves).

[0057] Specifically, such as Figure 6 As shown, semantic segmentation of RGB images is performed using a lightweight backbone network, outputting a semantic segmentation map including the target object category, including steps S610-S640: Step S610: Perform hierarchical feature extraction on the RGB image through the encoder layer to output a multi-scale feature map. Exemplarily, this encoder is the feature extraction part of the MobileNetV3-Small backbone network. During forward propagation, it sequentially outputs three layers of feature maps with different spatial resolutions and semantic granularities: the first layer is a high-level semantic feature map with a spatial resolution of 1 / 32 of the input RGB image, which contains strong semantic discriminative information about the target object category; the second layer is a mid-level detail feature map with a spatial resolution of 1 / 8 of the input RGB image, which preserves the edge, texture, and local structural information of the target object to support subsequent ROI cropping and fine segmentation; the third layer is a low-level texture feature map with a spatial resolution of 1 / 4 of the input RGB image, which contains the high-frequency texture and color gradient of the original image to assist in the fine restoration of mask boundaries.

[0058] It should be noted that the decoder layer receives the above three layers of feature maps and uses them as deterministic inputs for subsequent processing: the high-level semantic feature map serves as the main input of the LR-ASPP module and is used to construct global context enhancement features; the mid-level detail feature map is used for ROI cropping and feature concatenation of the instance controller in steps S420 and S610; and the low-level texture feature map is fused with the upsampled intermediate features through skip connections during the upsampling stage of the decoder to correct mask boundaries and improve pixel positioning accuracy.

[0059] Step S620: Input the multi-scale feature map into the decoder to process the multi-scale feature map through three parallel paths: global average pooling, dilated convolution, and identity mapping. Then, fuse the outputs of each path to obtain a fused semantic feature map.

[0060] This step integrates a Lite Reduced ASPP (LR-ASPP) module within the decoder to enhance the model's ability to jointly model global scene relationships, large-scale spatial context, and local structural details of target objects. Specifically, the global average pooling path performs global pooling on the high-level semantic feature map to obtain a global vector representing the scene semantics of the entire image. This vector is then subjected to 1×1 convolution and bilinear upsampling to restore the original feature map spatial size, generating a global context feature map. The dilated convolution path applies a dilated convolution operation with an expansion rate of not less than nine to this high-level semantic feature map, enabling it to obtain contextual information covering a larger receptive field without reducing spatial resolution, generating a large receptive field feature map. The identity mapping path directly retains the original high-level semantic feature map as a local semantic benchmark. Finally, the feature maps output by the above three paths are added element-wise to obtain a fused semantic feature map. This fusion operation enables the model to perceive both local appearance and understand the role of pixels in the global scene when performing pixel classification.

[0061] Step S630: After performing dimensionality reduction, upsampling and normalization operations on the fused semantic feature map, the class probability distribution of each pixel in the fused semantic feature map is obtained.

[0062] In this step, the fused semantic feature map is first processed using step 1. 1. A convolution operation is performed to compress the number of channels to match the total number of preset categories (e.g., N). Then, a bilinear upsampling operation is used to gradually restore the spatial resolution to an intermediate resolution (e.g., 1 / 4) that is the same as the RGB image size. At this point, the aforementioned low-level texture feature map is introduced, and it is added element-wise with the intermediate feature map upsampled to the same resolution to form a fused and enhanced feature representation. Afterward, this fused feature is subjected to a second convolution operation. Bilinear upsampling is performed to restore the spatial resolution to be completely consistent with the RGB image. Finally, a softmax operation is performed on the upsampled feature map in the channel dimension, so that an N-dimensional probability vector is output at each pixel position, and the sum of each element in the vector is 1, representing the probability of the pixel belonging to each category.

[0063] In step S640, the category with the highest probability corresponding to each pixel is used as the label of the current pixel, thereby generating a semantic segmentation map.

[0064] In this step, for each pixel position output by Softmax, the category corresponding to the dimension with the largest value in its N-dimensional probability vector is selected as the final semantic label for that pixel; all pixel labels are arranged according to their spatial position to form a semantic segmentation map with the same size as the RGB image and each pixel has a clear category label.

[0065] Step S520: Extract the connected regions of the category to which the target object belongs based on the semantic segmentation map, and generate the region of interest of the target object.

[0066] This step mainly involves locating the spatial position of each independent target object from the global semantic understanding, providing a focused area for subsequent fine instance segmentation, thereby significantly reducing the computational load while ensuring recognition accuracy.

[0067] Specifically, first, a binary image of the same size as the semantic segmentation map is constructed, called the category binary map. For each pixel in the semantic segmentation map, if its category label belongs to a preset target object category (e.g., a cargo box), the value of the corresponding position of that pixel in the category binary map is set to 1; otherwise, it is set to 0. In the resulting category binary map, all pixels with a value of 1 constitute one or more spatially separated pixel clusters, each pixel cluster corresponding to a potential, complete object instance in the image. It should be noted that the generation of the category binary map and the connected component analysis are based on the semantic segmentation accuracy supported by the aforementioned mid-level detail feature map. Since the mid-level detail feature map has 1 / 8 spatial resolution and 96 channels of structural representation capability, its output semantic segmentation map has higher robustness in the continuity of target object edges and local category discrimination, thus ensuring the completeness and accuracy of subsequent connected component extraction.

[0068] Next, connected component analysis is performed on the binary image of the category. This analysis is based on the 8-neighborhood connectivity criterion, which considers a pixel to be adjacent to its eight neighboring pixels (up, down, left, right, and four diagonal directions). The system traverses all pixels in the binary image of the category. For each pixel with a value of 1, it recursively searches along the 8-neighborhood directions for all connected pixels with a value of 1 and marks them as the same connected region. Each independent connected region represents a spatially continuous and semantically singular target object instance in the image. For example, in an image containing two independent cargo boxes, the analysis will output two unconnected connected regions, corresponding to cargo box A and cargo box B, respectively.

[0069] Subsequently, the bounding rectangle calculation is performed for each connected region. The system iterates through the image coordinates (x, y) of all pixels within the connected region and finds the minimum value of the x-coordinate. With the maximum value And, and the minimum value of the y-coordinate. With the maximum value Using these four extreme values, define an axis-aligned rectangle with the coordinates of its top-left corner as ( ). , ), width is ( - +1), height is ( - +1). This rectangle is the region of interest (ROI) corresponding to the connected region. The obtained ROI can accurately define the approximate spatial range of the target object in the image, providing a cropping boundary for subsequent fine segmentation.

[0070] Finally, to improve the efficiency of the entire cascaded segmentation process and suppress noise interference, the system performs heuristic filtering on the generated ROI list. This includes two operations: First, since small connected regions in teleoperation scenarios are often caused by image noise, reflective spots, or false detections of small objects at a distance, they lack reliable grasping or interactive value. Therefore, ROIs with excessively small areas need to be removed (e.g., in a 640×480 resolution teleoperation scenario, ROIs smaller than 32×32 pixels typically need to be removed). Second, since two ROIs with a large intersection-union (IoU) are considered to point to different segments of the same object, they need to be merged into a single ROI, i.e., ROIs with high overlap and the same category. For example, if two ROIs have an IoU exceeding 0.8 and are of the same category, they are merged. After the above filtering, a concise, high-quality ROI coordinate list is finally generated, where each ROI is defined by (…). , It is represented in the form of a quadruple (width, height) and stored according to category labels.

[0071] Step S530: Perform fine instance segmentation on the region of interest and output the actual mask of the target object.

[0072] This step is the second stage of the cascaded segmentation process. Its main purpose is to abandon intensive calculations on the entire RGB image and instead perform high-precision pixel-level segmentation only in the filtered regions of interest, thereby reducing the computational load by more than an order of magnitude while ensuring the quality of the mask.

[0073] In some implementations, such as Figure 7 As shown, step S530 specifically includes steps S710-S730: Step S710: The RGB image block corresponding to the region of interest is concatenated with the feature block at the corresponding position in the multi-scale feature map to obtain the concatenated feature block. The concatenated feature block is then input into the instance controller sub-network to output the dynamic filter weights.

[0074] This step first involves cropping an image patch from the original RGB image that perfectly corresponds to the spatial location of the region of interest, and then scaling it to a fixed size of 128. 128, to adapt to the input requirements of subsequent networks; simultaneously, from the detail feature map output by the middle layer of the lightweight backbone network encoder, a feature block corresponding to the spatial position of the ROI on the image plane is cropped. This feature block retains higher-level semantic information and richer texture details than the RGB image block. Subsequently, the above RGB image block and feature block are concatenated along the channel dimension.

[0075] This step first crops an image patch from the original RGB image that perfectly corresponds to the spatial location of the Region of Interest (ROI), and scales it to a fixed size of 128×128. Simultaneously, from the mid-level detail feature map, a feature patch corresponding to the ROI's spatial location on the image plane is cropped. This feature patch retains higher-level semantic information and richer texture details than the RGB image patch. Then, the RGB image patch and the feature patch are concatenated along the channel dimension to form a concatenated feature block with higher information density. This concatenated feature block is input into the instance controller subnetwork, which consists of two levels of 3D models. The system consists of three convolutional layers followed by a global average pooling layer. Its function is to extract the overall representation of the ROI and output a set of dynamic filter weights. These weights are not preset constants, but parameters generated in real time based on the appearance characteristics of the current ROI.

[0076] Step S720: Input the dynamic filter weights into a mask generation network with fixed weights, and use the mask generation network to perform modulation operations on the spliced ​​feature blocks based on the dynamic filter weights to generate a mask response map.

[0077] In this step, the mask generation network is an instance-independent, weighted convolutional module containing two convolutional layers and one upsampling operation. The network's processing flow involves using dynamic filter weights as the operating parameters for its key convolutional layers; that is, replacing the fixed convolutional kernels in traditional convolutions with these weights to perform convolution operations on the concatenated feature blocks. This instance-determined convolution behavior modulation operation allows the network to adaptively adjust its feature extraction strategy based on the surface reflection, wear condition, or viewing angle distortion of the cargo container, thereby improving its ability to distinguish between similar objects. The output of the modulation operation is a single-channel mask response map with spatial dimensions consistent with the ROI, where each pixel value represents the probability response intensity that the location belongs to the target object instance.

[0078] Step S730: Perform activation operation and fixed threshold binarization on the mask response map in sequence to generate the actual mask of the target object.

[0079] This step first performs a Sigmoid activation operation on the mask response map, mapping the response value of each pixel to the [0,1] interval to give it probabilistic semantics. Then, the Sigmoid output is binarized using a fixed threshold (e.g., 0.5): for each pixel, if its Sigmoid output value is greater than or equal to the fixed threshold, it is set to 1 (foreground, belonging to the target object instance); otherwise, it is set to 0 (background). After this processing, a binary image with the exact same size as the ROI is obtained, which is the actual mask of the target object.

[0080] As an example, comparing the ideal mask with the actual mask of the target object obtained by instance segmentation of the RGB image to obtain the occluded region mask specifically includes: performing a logical difference operation between the ideal mask and the actual mask to identify the set of pixels that are foreground pixels in the ideal mask but background pixels in the actual mask, and determining the set of pixels as the occluded region mask.

[0081] It can be understood that the logical difference operation in this embodiment is a... and Performing a bitwise NAND operation on two binary images can be represented as follows: = AND (NOT );in, and All are binary images with the same resolution as RGB images, and each pixel has a value of 0 or 1; This is the mask for the occluded area, which is also a binary image of the same resolution. The intuitive physical meaning of this operation is: a pixel only becomes visible if it falls within the ideal mask. The value in the middle is 1 (meaning that under the assumption of no occlusion, the target object should be displayed at this location), while in the actual mask... A pixel is considered occluded only when its value is 0 (meaning that the location is not identified as a target object by the instance segmentation network in the current RGB image). The center value is 1; all other cases (such as both being 1, both being 0, or ideally 0 but actually 1) do not constitute occlusion, and the corresponding pixel is in The middle value is 0.

[0082] Therefore, the occluded area mask is a pixel-level map that accurately represents the location of missing visual information. Each pixel with a value of 1 corresponds to a surface point on the target object that cannot be directly observed by the camera because it is occluded by the robot itself, environmental obstacles, or other target objects.

[0083] Furthermore, extracting the contour polygon representing the boundary of the occluded region specifically includes: performing a contour discovery operation on the occluded region mask, and using a standard contour extraction algorithm based on boundary tracking to generate a closed contour polygon composed of a series of sequentially connected vertex coordinates.

[0084] The contour discovery operation takes an occluded region mask as its sole input. This mask is a binary image with the exact same resolution as the RGB image, where pixels with a value of 1 form one or more mutually separated connected regions. Each connected region corresponds to an occluded continuous surface area on the target object. The goal of contour extraction is to automatically identify the precise geometric representation of the outermost boundary of each connected region from its pixel set; that is, a closed polygon that is connected end-to-end, does not self-intersect, and completely encloses the connected region.

[0085] In one embodiment, this operation is implemented using the Suzuki85 boundary tracking algorithm (the findContours function in OpenCV). Specifically, the execution process involves starting from any pixel with a value of 1 in the mask of the occluded region, searching clockwise for the first pixel in its 8-neighborhood that is also 1, and recording its coordinates. Then, using this pixel as a new starting point, the same tracking rule continues until the starting pixel is reached, forming a closed trajectory. All pixel coordinates on this trajectory are arranged in traversal order, forming a set of vertex coordinates [(x1,y1),(x2,y2),…,(xn,yn)] constituting the connected region. For scenarios with multiple occluded regions (e.g., the top and sides of a cargo box are simultaneously occluded), the algorithm will independently perform multiple boundary tracking operations, outputting multiple unrelated closed contour polygons. Each contour polygon is stored in memory as an ordered array of its vertex coordinates for direct use in subsequent steps.

[0086] In the subsequent graphics rendering stage, the system passes the aforementioned vertex coordinate array to the rendering pipeline, where the stencil buffer generates a binary mask of the same size as the RGB image. Pixels within the mask that are inside all the outline polygons are set to 1, while the rest are set to 0. This mask is then used to limit the scope of all subsequent perspective rendering operations, ensuring that the virtual model is only rendered within the occluded areas, thus achieving a true selective perspective effect.

[0087] In some implementations, the process also includes performing spatial relationship calculations and risk assessments. This process is based on the dynamic three-dimensional semantic map obtained in step S110, the current position of the robot's end effector, the pose of the target object's three-dimensional mesh model, and the three-dimensional information of environmental obstacles. It performs real-time geometric relationship quantification analysis to provide data support for safety warnings and task logic.

[0088] Specifically, it includes two parallel computing processes: The first step: calculating the interaction between the operating tool and the target object.

[0089] This process is based on the three-dimensional position of the robot's end effector. Starting from the center point of the predefined graspable surface on the target object in three dimensions. Calculate the Euclidean distance between the two endpoints: Distance = ... Simultaneously, calculate the Z-axis direction vector of the end effector coordinate system and... The angle between the normal vectors of the grabbable surface. This distance and angle together constitute the quantitative criterion for the operation readiness state, which is used to drive task logic judgment and AR guidance accuracy adjustment.

[0090] The second process: calculating the minimum distance between the robot and environmental obstacles.

[0091] This process simplifies the robot into a set of basic geometries (including a cylinder representing the upper arm, a cuboid representing the torso, and a sphere representing the end effector), and represents environmental obstacles as simplified models with known poses (such as cylinders representing shelf uprights and bounding boxes representing AGVs). A hierarchical bounding box algorithm is used for continuous collision detection: first, axis-aligned bounding boxes (AABBs) are used to coarsely filter out obviously non-intersecting object pairs; then, precise geometric distance calculations are performed on candidate object pairs (such as the shortest distance between the surfaces of two cylinders); finally, the minimum value among all calculation results is taken as the global minimum distance between the robot and the environment at the current moment. The system will The system compares the data with preset multi-level safety thresholds. Based on the comparison results, it generates different levels of warning instructions (such as visual warnings, auditory alarms, and sending deceleration or pause commands to the robot controller), and identifies the risky areas, such as when... When the distance falls below the warning threshold (e.g., 0.15 meters), a visual warning instruction is generated; when the distance falls below the emergency stop threshold (e.g., 0.08 meters), a signal is generated to send a pause instruction to the robot controller.

[0092] It should be noted that the spatial relationship calculation and the occlusion analysis in step S130 share the same set of upstream data sources (i.e., the dynamic 3D semantic map and the poses of each object output by S110). While the two calculations are parallel and independent in their computational processes, they are collaboratively output at the decision-making level. For example, when step S130 identifies that the top of the cargo box is occluded and determines... When the distance is 0.10 meters, the system will increase the risk level of the obscured area and simultaneously enable the perspective effect and red highlight warning in the AR rendering.

[0093] In step S140, within the pixel area formed by the outline polygon, a depth value adjustment operation is performed on the three-dimensional mesh model of the target object, and based on the adjusted three-dimensional mesh model, rendering is performed on the RGB image to present the perspective effect of the occluded area.

[0094] This step primarily involves adjusting the depth value in the graphics rendering pipeline to allow the virtual model to overcome the actual occlusion relationship in occluded areas, thereby visually presenting the hidden key structures to the operator. This operation does not globally modify the model's depth; rather, it is strictly limited to the pixel area defined by the contour polygon extracted in step S130, ensuring that virtual information is presented only where necessary and avoiding visual interference.

[0095] In some implementations, such as Figure 8 As shown, the depth value adjustment operation is performed on the 3D mesh model of the target object, including steps S810-S820: Step S810: Within the pixel area defined by the contour polygon, reduce the depth value of the corresponding pixel of the 3D mesh model of the target object to be lower than the depth value of the real scene.

[0096] Exemplary, this step involves dynamically adjusting the depth value of each pixel located inside the outline polygon within the fragment shader of the graphics rendering pipeline. This includes two specific methods: The first method is the depth offset method. It first calculates the depth value that the pixel should have under the standard rendering process (i.e., the standard depth value), and then subtracts it from a preset depth offset. The adjusted depth value is obtained. ;in, The value range is 0.01 to 0.05 (dimensionless value in normalized device coordinate system), which corresponds to a physical spatial distance of 1 cm to 5 cm, that is, when At 0.01, it indicates that the virtual model is still easily occluded by real occlusions; when At 0.05, it indicates that the virtual model will be distorted, disrupting spatial consistency; The second method is the fixed near-value method: This method does not perform subtraction operations, but directly sets the depth value of the pixel to a fixed small value that is very close to the camera's near clipping plane, such as 0.1 (in the normalized device coordinate system, 0.0 represents the near clipping plane, and 1.0 represents the far clipping plane). This value has been tested and can ensure that the virtual model can stably penetrate and display in all common occlusion scenarios, and does not incorrectly overlap with other foreground virtual elements.

[0097] Step S820: Outside the pixel area defined by the outline polygon, maintain the standard depth value of the corresponding pixel of the three-dimensional mesh model of the target object.

[0098] As an example, for all pixels located outside the pixel region defined by the outline polygon, the system does not perform any depth value adjustment operation. Instead, it directly uses the depth value calculated by the 3D mesh model through the standard graphics rendering process. This standard depth value is obtained through the following steps: First, the vertex coordinates of the 3D mesh model are transformed from the real-time pose obtained above in the model coordinate system to the world coordinate system, and then transformed to the camera coordinate system through the current pose of the RGB camera; subsequently, perspective projection is performed through the pinhole camera model, and perspective division is performed to obtain the depth value in the normalized device coordinate system; this depth value is the standard depth value.

[0099] Furthermore, in the graphics rendering pipeline, this standard depth value is used for depth testing. When pixels of the virtual model outside the contour region participate in the depth test, their standard depth value is compared with the real scene depth value at the corresponding position in the depth buffer. If the standard depth value is greater than or equal to the real scene depth value at that position, the depth test fails, and the virtual model's pixels are discarded; if it is less, it passes and is written to the frame buffer. This processing method ensures that the rendering behavior of the virtual model outside the contour region is consistent with the occlusion relationship of real objects: when a part of the target object is not occluded, the virtual model pixels at its corresponding position are discarded because their depth value is not less than the real depth; when that part is occluded by other objects, the real depth value is smaller, and the virtual model pixels are still discarded. Thus, the virtual model only breaks through real occlusion within the pixel area defined by the contour polygon by adjusting the depth value and configuring the depth test in step S820; outside this area, the virtual model does not change its original occlusion state.

[0100] Furthermore, such as Figure 9 As shown, based on the adjusted 3D mesh model, rendering is performed on the RGB image to present the perspective effect of the occluded area, including steps S910-S930: Step S910: Perform a row template buffer setting operation, load the outline polygon as a template into the template buffer of the graphics rendering pipeline, so as to limit subsequent rendering only in the RGB image and within the pixel area defined by the outline polygon.

[0101] This step first passes the vertex coordinate array of the contour polygons extracted in step S130 to the GPU and calls an OpenGL function to initialize the stencil buffer. For pixels located inside all contour polygons, their stencil values ​​are set to 1; for pixels outside the contours and in uncovered areas, the stencil values ​​remain 0. Thereafter, all subsequent rendering operations are only executed when the stencil value of the current pixel is equal to 1, thus strictly limiting the scope of subsequent perspective rendering in space. This method ensures that the virtual model does not generate redundant overlays in the visible area of ​​the target object, which is a key spatial constraint for achieving selective rather than global perspective.

[0102] Step S920: Perform a depth test configuration operation, setting the depth comparison function of the graphics rendering pipeline to a less than comparison function, so that the depth value of the newly rendered pixel is less than the existing depth value at the corresponding position in the depth buffer.

[0103] This step can be achieved by configuring the depth testing mechanism of the graphics rendering pipeline. Specifically, the system calls the OpenGL function `glEnable(GL_DEPTH_TEST)` to enable the depth testing function and calls `glDepthFunc(GL_LESS)` to set the depth comparison function to less than the comparison function. Under this configuration, when the depth value of a new rendered pixel (i.e., the depth value of the pixel mentioned above) is less than the comparison function, the depth testing function will be configured to be less than the comparison function. If the depth value of a pixel is less than the depth value already stored at the same screen position in the depth buffer, the depth test passes, and the color value of that pixel is written to the frame buffer; otherwise, the pixel is discarded. Since step S810 has adjusted the depth value of the virtual model within the area defined by the outline polygon to be lower than the depth value corresponding to the real occluder, the depth test always passes within this area, and the pixels of the virtual model are stably written to the frame buffer. This depth test configuration works in conjunction with the stencil buffer setting in step S910. The stencil buffer determines the pixel space range for rendering, and the depth test determines whether pixels within this range meet the depth conditions and are ultimately retained. Together, they ensure that the virtual model is only effectively rendered within the occluded area.

[0104] Step S930: Within the pixel area defined by the outline polygon, the 3D mesh model is semi-transparently blended and rendered to present the perspective effect of the occluded area on the RGB image.

[0105] This step is achieved by enabling the alpha blending mechanism. Specifically, the system calls the OpenGL function `glEnable(GL_BLEND)` to enable the blending function and calls `glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA)` to set the blending factor. Under this configuration, the source color (virtual model color) is weighted according to its alpha channel value, and the target color (current pixel color of the RGB image) is weighted according to (1 minus the source alpha value). The blending result is written to the frame buffer. The virtual model's material is assigned an alpha value in the range of 0.3 to 0.5. This value was determined through testing; when the alpha value is below 0.3, the virtual model's visibility is insufficient; when it is above 0.5, the underlying information of the RGB image is excessively attenuated. Within the pixel area defined by the outline polygon, the virtual model and the RGB image are calculated pixel-by-pixel according to the above blending formula to generate the final display pixels. This process allows the virtual model to be superimposed on the real image in a partially opaque state, and its pixel value is jointly determined by the source alpha, source color, and target color. The resulting visual effect is that the virtual model of the target object is presented only within the pixel area defined by the outline polygon, while outside this area, the virtual model does not participate in rendering, and the real image remains in its original state.

[0106] Furthermore, when presenting the perspective effect of the occluded area on the RGB image, it also includes rendering a navigation guide curve; wherein, the navigation guide curve is a smooth two-dimensional curve located in the RGB image plane, the navigation guide curve is determined by the collision-free path between the current position of the robot end effector and the task target point, and the visual representation includes a color distribution that changes continuously along the length of the curve.

[0107] The navigation guidance curve is generated based on spatial constraints jointly constituted by a dynamic 3D semantic map, the current position of the robot's end effector, and the position of the task target point. Specifically, the system first performs path planning in the dynamic 3D semantic map, starting from the current position of the robot's end effector and ending at the task target point, using a fast randomized exploration tree algorithm to generate an initial path that avoids all obstacles marked on the map. This initial path consists of a series of discrete 3D coordinate points, and its total length is denoted as... .

[0108] The system then performs geometric fitting on the initial path to generate a smooth two-dimensional curve. In some implementations, this curve is a cubic Bézier curve, with its control points determined by the endpoints and direction information of the initial path, i.e., starting from the current position of the robot's end effector. The endpoint is the location of the mission objective. ;from Starting from the beginning, extend the distance along the tangent of the first segment of the initial path. The first control point was obtained. ;from Starting from the beginning, extend the line along the opposite direction of the tangent at the end of the initial path by the same distance to obtain the second control point. The cubic Bézier curve can be represented as: ,in The system samples t at equal intervals (e.g., step size 0.02) and calculates a set of three-dimensional points. .

[0109] Furthermore, the system uses the current pose of the RGB camera obtained in step S220 to convert the above-mentioned three-dimensional point set... By mapping the image onto the RGB image plane through perspective projection transformation, a set of two-dimensional pixel coordinates is obtained. Then connect adjacent coordinate points in sequence to form a smooth two-dimensional curve located in the RGB image plane.

[0110] Finally, the system applies a color distribution that varies continuously along the length of the curve to the smooth two-dimensional curve. Specifically, for each sampling point... Calculate the cumulative arc length along the curve from the starting point to that point. and normalized to Then, the corresponding color is retrieved from the preset color mapping table, for example, when... At 0, the color is red. (1.0, 0.0, 0.0); when At 1 o'clock, the color is blue. (0.0, 0.0, 1.0); For intermediate values, linear interpolation is used: Color( )= It is understandable that this color distribution follows... Continuous change, without jumps or interruptions.

[0111] It should be noted that the navigation guide curve shares the same graphics rendering pipeline configuration as the perspective effect of the occluded area presented in step S140. They share the same stencil buffer settings (effective only within the outline polygon), the same depth test configuration (less than the comparison function), and the same semi-transparent blending rendering mode. Therefore, the navigation guide curve is always visually positioned at the same rendering level as the perspective effect, neither obscured by the virtual model nor penetrating the real scene, ensuring that the operator receives consistent, stable, and spatially believable augmented reality information.

[0112] In some implementations, security alert rendering is also included.

[0113] This step responds to the collision risk warning instruction output by the spatial relationship calculation unit in step S130. This instruction includes a risk location identifier (e.g., right elbow), a corresponding obstacle identifier (e.g., right upright of a shelf), and a risk level (e.g., warning level or emergency level). The safety warning rendering includes two synchronous sub-processes: virtual ontology rendering and collision risk highlighting. Both share the same underlying configuration of the graphics rendering pipeline.

[0114] Virtual ontology rendering sub-process: The robot's 3D model obtained in step S220 is mapped onto the RGB image plane using its real-time pose in the world coordinate system and the current pose of the RGB camera in step S120 through perspective projection transformation. Subsequently, the model is rendered as a semi-transparent blue wireframe, overlaying it onto the actual robot body. This rendering allows the operator to intuitively perceive the robot's overall spatial footprint and range of motion without switching viewpoints.

[0115] High-risk collision subprocesses: When the spatial relationship calculation unit determines in step S130 that a certain part of the robot (such as the right elbow) has a collision risk, the system first extracts the geometric definition of the part in the robot's own 3D model (e.g., a cylindrical model representing the right elbow), and accurately projects it onto the RGB image plane based on its real-time pose in the world coordinate system and the current pose of the RGB camera, generating its two-dimensional occupied area in the image. Subsequently, a high-frequency visual warning is enabled in this area, and the two-dimensional occupied area is filled with red at a preset frequency. The warning level risk uses a frequency of 2Hz (i.e., one flashing cycle every 500ms), and the emergency level risk uses a frequency of 5Hz (i.e., one flashing cycle every 200ms). The filling duration within each flashing cycle is 40ms, and the rest of the time is transparent. Simultaneously, for the associated obstacles marked in the dynamic 3D semantic map in step S220 (such as the right upright of the shelf), the red outline flashing with the same frequency and duration is performed using its projection boundary in the image as the outline. This high-frequency flashing is precisely controlled by the GPU timer of the graphics rendering pipeline.

[0116] It should be noted that this safety warning rendering strictly shares the same graphics rendering pipeline configuration as the perspective rendering in step S140 and the subsequent navigation guidance curve rendering. That is, they share a template buffer, and all rendering operations use their respective two-dimensional occupied areas or outline polygons as templates to ensure that they only take effect within the target pixel area. They also share a depth testing configuration: the depth comparison function is set to GL_LESS, and the depth writing strategy of the warning layer is consistent with that of the perspective layer, ensuring that its rendering result is above the perspective effect but below the UI information layer. They also share an Alpha blending mode (the red fill uses Alpha=0.7 to ensure visibility). As a result, the operator can synchronously, stably, and unambiguously observe the occluded structure (perspective layer), their own movement path (guidance layer), and the immediate collision risk (warning layer) in the same AR frame, forming a multi-dimensional situational awareness closed loop with physical credibility and cognitive consistency.

[0117] This embodiment solves long-standing technical problems in the field of teleoperated augmented reality, such as visual blind spots, difficulty in balancing real-time performance and accuracy, and lack of contextual adaptation in information presentation, through multimodal perception, occlusion-driven pixel-level localization, lightweight cascaded segmentation, and adaptive rendering. Specifically, this embodiment employs a multimodal fusion-driven dynamic 3D semantic modeling method, uniformly mapping RGB images, depth data, robot pose, and prior 3D models to the same spatiotemporal reference. This enables the system to generate reliable visual compensation based on geometrically consistent scene understanding even when the target is partially occluded, fundamentally mitigating operational risks caused by invisibility. Furthermore, this application adopts a cascaded, task-oriented instance segmentation architecture, significantly reducing computational redundancy while ensuring pixel-level recognition accuracy for operating tools and key targets. This allows the system to stably output high-frame-rate, low-latency enhanced visuals on resource-constrained remote operation terminals, ensuring a high degree of synchronization between visual feedback and physical actions. Further, this application employs a context-aware layered rendering mechanism, differentiating and collaboratively presenting enhanced information such as perspective effects, safety warnings, and navigation guidance based on task stages, spatial relationships, and risk levels. This avoids information overload or the neglect of key prompts, thereby constructing a more natural, reliable, and easily understandable human-computer collaborative interaction method.

[0118] Figure 10 A schematic diagram of a teleoperated augmented reality device according to an embodiment of this application is shown. Exemplarily, the teleoperated augmented reality device includes: The 3D semantic map acquisition module 100 is used to construct and update a 3D semantic map containing the real-time pose of the target object based on the RGB images and depth images acquired by the acquisition device mounted on the remote robot and combined with the robot's own 3D model.

[0119] The ideal mask generation module 200 is used to generate an ideal mask for the target object under unobstructed conditions based on the real-time pose of the target object's three-dimensional mesh model in the three-dimensional semantic map and the current pose of the acquisition device that acquires the RGB image.

[0120] The contour polygon acquisition module 300 is used to compare the ideal mask with the actual mask of the target object obtained by instance segmentation of the RGB image to obtain the mask of the occluded area and extract the contour polygon representing the boundary of the occluded area.

[0121] The enhanced display module 400 is used to perform depth value adjustment operations on the three-dimensional mesh model of the target object within the pixel area defined by the outline polygon, and to render the adjusted three-dimensional mesh model on the RGB image to present the perspective effect of the occluded area.

[0122] It is understood that the device in this embodiment corresponds to the teleoperated augmented reality method in the above embodiments, and the options in the above embodiments are also applicable to this embodiment, so they will not be described again here.

[0123] This application also provides a terminal device, exemplary of which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the functions of the various modules in the above-described teleoperation augmented reality method or the above-described teleoperation augmented reality device.

[0124] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0125] The memory can be, but is not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0126] This application also provides a computer-readable storage medium for storing the computer program used in the aforementioned terminal device. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0128] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0129] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0130] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A teleoperated augmented reality method, characterized in that, include: Based on the RGB and depth images collected by the acquisition device mounted on the remote robot, a 3D semantic map containing the real-time pose of the target object is constructed and updated by combining the robot's own 3D model. Based on the real-time pose of the target object's 3D mesh model in the 3D semantic map and the current pose of the acquisition device that acquired the RGB image, an ideal mask for the target object under unobstructed conditions is generated. The ideal mask is compared with the actual mask of the target object obtained by instance segmentation of the RGB image to obtain the mask of the occluded area, and the contour polygon representing the boundary of the occluded area is extracted. Within the pixel area formed by the outline polygon, a depth value adjustment operation is performed on the three-dimensional mesh model of the target object, and based on the adjusted three-dimensional mesh model, rendering is performed on the RGB image to present the perspective effect of the occluded area.

2. The teleoperation augmented reality method according to claim 1, characterized in that, The RGB and depth images acquired by the acquisition device mounted on the remote robot are combined with the robot's own 3D model to construct and update a 3D semantic map containing the real-time pose of the target object, including: Based on the RGB and depth images synchronously acquired by the RGB and depth cameras mounted on the remote robot, a 3D point cloud with color information is generated, and the point cloud of the robot's own 3D model is removed from the 3D point cloud with color information to form the target point cloud. The target point cloud is fused with the global 3D map to update the 3D semantic map, and the 3D mesh model of the target object is registered with the updated 3D semantic map to obtain the real-time pose of the 3D mesh model.

3. The teleoperated augmented reality method according to claim 2, characterized in that, Based on the acquired RGB and depth images, a 3D point cloud with color information is generated, and the point cloud of the robot's own 3D model is removed from the 3D point cloud with color information to form the target point cloud, including: The robot's own three-dimensional model is obtained by solving the data from the robot joint encoder. Using the camera intrinsic parameters of the RGB camera, the extrinsic parameter matrix obtained by joint calibration of the RGB camera and the depth camera, and the depth image aligned with the pixels of the RGB image, the 3D point cloud with color information corresponding to the RGB image is generated; The robot's own 3D model is projected onto the imaging view of the RGB image, and the point cloud corresponding to the robot's own 3D model is removed from the 3D point cloud with color information to obtain the target point cloud.

4. The teleoperated augmented reality method according to claim 2, characterized in that, The step of generating an ideal mask for the target object in an unobstructed state based on the real-time pose of the target object's 3D mesh model in the 3D semantic map and the current pose of the acquisition device acquiring the RGB image includes: The three-dimensional mesh model of the target object is mapped onto the RGB image plane through perspective projection transformation based on the camera intrinsic parameters of the RGB camera and the current pose of the RGB camera. Rasterization filling is performed on the triangular facets of the projected 3D mesh model to generate a binary image with the same resolution as the RGB image; The pixel region with a pixel value of 1 in the binary image is used as the ideal mask for the target object under unobstructed conditions.

5. The teleoperation augmented reality method according to claim 1, characterized in that, The process of performing instance segmentation on the RGB image to obtain the actual mask of the target object includes: Semantic segmentation is performed on the RGB image using a backbone network, and a semantic segmentation map including the target object category is output. Based on the semantic segmentation map, the connected regions of the category to which the target object belongs are extracted to generate the region of interest of the target object; Perform fine instance segmentation on the region of interest and output the actual mask of the target object.

6. The teleoperation augmented reality method according to claim 5, characterized in that, The step of comparing the ideal mask with the actual mask of the target object obtained by instance segmentation of the RGB image to obtain the mask of the occluded region includes: Perform a logical difference operation between the ideal mask and the actual mask to identify the set of pixels that are foreground pixels in the ideal mask but background pixels in the actual mask, and determine the set of pixels as the mask of the occluded area.

7. The teleoperation augmented reality method according to claim 5, characterized in that, The backbone network includes encoders and decoders; The step of performing semantic segmentation on the RGB image through a backbone network to output a semantic segmentation map including the target object category includes: The encoder performs hierarchical feature extraction on the RGB image and outputs a multi-scale feature map. The multi-scale feature map is input into the decoder to process the multi-scale feature map through three parallel paths: global average pooling, dilated convolution, and identity mapping. The output results of each path are then fused to obtain a fused semantic feature map. After performing dimensionality reduction, upsampling, and normalization operations on the fused semantic feature map, the class probability distribution of each pixel in the fused semantic feature map is obtained; The category with the highest probability corresponding to each pixel is used as the label of the current pixel, thereby generating the semantic segmentation map.

8. The teleoperation augmented reality method according to claim 7, characterized in that, The step of performing fine-grained instance segmentation on the region of interest and outputting the actual mask of the target object includes: The RGB image block corresponding to the region of interest is concatenated with the feature block at the corresponding position in the multi-scale feature map to obtain the concatenated feature block, and the concatenated feature is input into the instance controller sub-network to output dynamic filter weights; The dynamic filter weights are input into a mask generation network with fixed weights, and the mask generation network performs modulation operations on the spliced ​​feature blocks based on the dynamic filter weights to generate a mask response map. The mask response map is subjected to activation operation and fixed threshold binarization processing in sequence to generate the actual mask of the target object.

9. The teleoperation augmented reality method according to claim 1, characterized in that, The process of adjusting the depth value of the 3D mesh model of the target object includes: Within the pixel region defined by the outline polygon, the depth value of the corresponding pixel of the three-dimensional mesh model of the target object is reduced to a value lower than the depth value of the real scene; Outside the pixel region defined by the outline polygon, the standard depth value of the corresponding pixel of the three-dimensional mesh model of the target object is maintained.

10. The teleoperation augmented reality method according to claim 1, characterized in that, The rendering of the RGB image based on the adjusted 3D mesh model to present the perspective effect of the occluded area includes: Perform a template buffer setting operation to load the outline polygon as a template into the template buffer of the graphics rendering pipeline, so as to limit subsequent rendering only in the RGB image and within the pixel area defined by the outline polygon. Perform a depth test configuration operation, setting the depth comparison function of the graphics rendering pipeline to a less than comparison function, so that the depth value of the newly rendered pixel is less than the existing depth value at the corresponding position in the depth buffer; Within the pixel area defined by the outline polygon, the three-dimensional mesh model is rendered with semi-transparent blending to present the perspective effect of the occluded area on the RGB image.

11. The teleoperation augmented reality method according to claim 10, characterized in that, When presenting the perspective effect of the occluded area on the RGB image, the method further includes rendering a navigation guide curve; wherein, the navigation guide curve is a smooth two-dimensional curve located in the plane of the RGB image, the navigation guide curve is determined by the collision-free path between the current position of the robot end effector and the task target point, and the visual representation includes a color distribution that changes continuously along the length of the curve.

12. A remotely operated augmented reality device, characterized in that, include: The 3D semantic map acquisition module is used to construct and update a 3D semantic map containing the real-time pose of the target object based on the RGB images and depth images collected by the acquisition device mounted on the remote robot and combined with the robot's own 3D model. An ideal mask generation module is used to generate an ideal mask for the target object under unobstructed conditions based on the real-time pose of the target object's 3D mesh model in the 3D semantic map and the current pose of the acquisition device that acquires the RGB image. The contour polygon acquisition module is used to compare the ideal mask with the actual mask of the target object obtained by instance segmentation of the RGB image to obtain the mask of the occluded area and extract the contour polygon representing the boundary of the occluded area. An enhanced display module is used to perform a depth value adjustment operation on the three-dimensional mesh model of the target object within the pixel area defined by the outline polygon, and to render the adjusted three-dimensional mesh model on the RGB image to present the perspective effect of the occluded area.

13. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed on a processor, implements the teleoperated augmented reality method according to any one of claims 1-11.

Images