A substation screen cabinet robot pressing operation method and system fusing three-dimensional semantic information
By using a three-branch coupled network structure and a confidence gating mechanism, the problems of button differentiation and deep failure in substation cabinets were solved, achieving high-precision and robust robotic pressing operation.
Patent Information
- Application Number
- CN202611584734.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-14
- Publication Date
- 2026-11-10
AI Technical Summary
Existing VLA models lack the ability to actively perceive three-dimensional geometry in substation secondary cabinet scenarios, making it difficult to distinguish adjacent buttons with similar appearances. Furthermore, they fail to effectively address depth measurement failures caused by transparent protective panels and reflective indicator lights, resulting in insufficient accuracy and robustness of robot pressing operations.
A three-branch coupled network structure is adopted. The RGB image features are extracted by a two-dimensional visual encoder, the three-dimensional geometric features of the depth image are constructed by a three-dimensional geometric encoder, and the confidence of the depth image is evaluated by a confidence prediction network. The cross-modal attention mechanism of confidence gating is used for adaptive fusion to generate an accurate robot pressing operation sequence.
It significantly improves the model's ability to distinguish between adjacent buttons and its operational accuracy. It can automatically identify deep failure areas, improve the accuracy and robustness of robot pressing operations, and maintain a high success rate and low error, especially in the environment of transparent panels and reflective indicator lights.
Smart Images

Figure CN123071784A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot intelligent control technology, and in particular relates to a method and system for operating a substation cabinet robot by integrating three-dimensional semantic information. Background Technology
[0002] Current Vision-Language-Action (VLA) models for precise robot manipulation typically use RGB images and natural language commands as input. Visual features are extracted using a visual Transformer and aligned cross-modal with the language features to ultimately regress the action sequence of the end effector. When incorporating depth information, the mainstream approach is to directly concatenate the depth map as the fourth channel of the RGB image (RGB-D four-channel input), feeding it along with the RGB data into the same ViT backbone network for joint encoding. Some solutions also use early fusion or late concatenation to simply add or cascade depth features with RGB features before uniformly inputting them into the policy head to predict actions. These methods have achieved some success in general desktop grasping and object pushing tasks.
[0003] The existing technologies mentioned above have two significant shortcomings: First, treating the depth map as a fourth channel and stitching it with RGB essentially still relies on 2D convolution or 2D attention for feature extraction, failing to explicitly model the 3D geometric structure inherent in the depth. This results in insufficient perception of geometric information such as height differences, protrusions, and relative spatial positions of adjacent objects. In substation secondary cabinet scenarios, dozens of buttons with similar appearances are densely arranged on the cabinet panel, with each button protruding only a few millimeters. RGB features are insensitive to such subtle depth differences, easily leading to misidentification of adjacent buttons and deviations in press pose prediction. Second, existing methods assume the depth map output by the depth sensor is globally reliable, failing to consider the practical deployment challenges of consumer-grade depth cameras on surfaces such as transparent protective panels, reflective indicator lights, and mirrored metal, where depth noise is severely compromised. Once depth noise enters the feature space, it contaminates downstream decision-making, causing a sharp drop in the model's robustness in real substation environments.
[0004] In summary, existing VLA models lack both the ability to actively perceive 3D geometric structures and an adaptive discrimination mechanism for sensor data quality in complex industrial scenarios such as substations, making it difficult to meet the dual requirements of accuracy and robustness in practical deployments. Therefore, this invention proposes a substation cabinet robot pressing operation method and system that integrates 3D semantic information, fundamentally solving the above problems. Summary of the Invention
[0005] To address the above technical problems, this invention provides a method and system for operating a substation cabinet robot by integrating three-dimensional semantic information.
[0006] The technical solution adopted by this invention to solve its technical problem is: In the first aspect, a method for operating a substation cabinet robot by integrating three-dimensional semantic information is provided, including the following steps: S100: Acquire multimodal input data of the robot operation scene, including RGB image, depth image, natural language operation instructions and the current pose of the end effector of the robotic arm, preprocess the RGB image and depth image to obtain the preprocessed RGB image and depth image; S200: A two-dimensional visual token sequence is obtained by extracting features from the preprocessed RGB image using a two-dimensional visual encoder; S300: Based on the preprocessed depth image, perform 3D spatial back projection to construct a 3D geometric feature representation and obtain a 3D geometric token sequence; S400: The confidence prediction network evaluates the confidence of each pixel in the preprocessed depth image to obtain pixel-level confidence information that reflects the reliability of depth measurement. S500: A cross-modal attention mechanism is constructed using a two-dimensional visual token sequence as the guide and a three-dimensional geometric token sequence as the guided feature. The confidence information is used to suppress the attention response from the untrusted region in the cross-modal attention mechanism. Based on the confidence information, the suppressed attention output is adaptively weighted and fused with the two-dimensional visual token sequence to obtain fused multimodal features. S600: Encodes natural language operation instructions into language feature sequences, performs cross-modal alignment between fused multimodal features and language feature sequences, and generates robot motion sequences based on the aligned features and the current pose of the end effector.
[0007] Preferably, S100 includes: S110: Acquire multimodal input data of the robot operation scenario, including RGB images and depth images captured by the robotic arm wrist or a third-view RGB-D camera, and natural language operation commands issued by the substation dispatch system. and the current pose of the robotic arm's end effector. ; S120: Normalize the RGB image, perform invalid value masking on the depth image to remove invalid depth pixels, and scale the normalized RGB image and the masked depth image to a uniform network input resolution of 224×224 to obtain the preprocessed RGB image. and preprocessed depth map .
[0008] Preferably, S200 includes: The preprocessed RGB image is divided into segments by a 2D visual encoder. Each image block is divided into non-overlapping image blocks, and after linear projection, a two-dimensional visual token sequence is obtained, specifically: in, For token feature dimensions, Represents a two-dimensional visual encoder. The total number of image blocks. The height of the image. The width of the image. The side length of each image block. It is a two-dimensional visual token sequence used to encode the texture information, color information, button label text information, and indicator light status information of the display panel in the robot operation scenario.
[0009] Preferably, S300 includes: S310: Utilizing the intrinsic parameter matrix of an RGB-D camera The preprocessed depth image Each valid pixel in The back projection onto a 3D point in the camera coordinate system is as follows: This yields the original point cloud. Specifically: in, These are the x and y coordinates of the pixel, respectively. For the effective depth pixels, , Let these be the coordinates of the camera's principal point. , To normalize the focal length for the camera, , , These represent the x-coordinate, y-coordinate, and depth coordinates of the current pixel's 3D spatial point in the camera coordinate system. These are the x-coordinate, y-coordinate, and depth coordinates of the i-th effective depth pixel in the original point cloud, respectively. S320: Add color features sampled from the corresponding positions in the preprocessed RGB image as an extra channel to each point in the original point cloud, resulting in an original point cloud with color information. To control computational load and balance spatial distribution, the farthest point of the original point cloud containing color information is sampled to obtain a fixed number of points. Standardized point cloud ; S330: Standardizing point clouds Input a lightweight 3D encoder, at each level Grouping queries by ball with the query point as the center and radius The local geometry is constructed by sampling neighborhood points, and then local features are extracted through a shared multilayer perceptron and max pooling, specifically: in, Indicates the first In the hierarchy, query point Local features, Indicates query point Center, radius The set of neighboring points within, For neighborhood point index, For neighborhood points Three-dimensional spatial coordinates, For neighborhood points In the previous level Features Represents a multilayer perceptron mapping; S340: After multi-level feature extraction, the point cloud features are aligned to the same dimension as the 2D visual token through linear projection. The resulting three-dimensional geometric token sequence is as follows: in, This represents the final number of point cloud tokens. It is a sequence of three-dimensional geometric tokens used to represent the convex height information of objects in the scene, the distance information between adjacent objects, and the surface normal vector information.
[0010] Preferably, S400 includes: S410: The confidence prediction network employs a lightweight encoder-decoder architecture, with input consisting of a channel-level concatenation of the preprocessed RGB image and the preprocessed depth image. The output is a single-channel pixel-level confidence map, specifically: in, This is the output pixel-level confidence map; S420: The training supervision signal for the confidence prediction network is constructed as follows: Simultaneously acquire the ground truth depth in the simulation environment. and simulated sensor depth The pixel-level confidence ground truth is constructed based on the relative error between the two at the same pixel location, specifically: in, For temperature coefficient, To prevent small constants with a denominator of zero, pixel position The simulated sensor depth at the location, pixel position The truth depth at that point pixel position The confidence level is the true value at the given position. This construction method makes the confidence level of pixels with larger errors lower. Using binary cross-entropy loss The confidence prediction network is trained as follows: S430: Aligns with the 2D visual token structure, and displays pixel-level confidence maps. The image is divided into blocks according to the block partitioning method of the 2D visual encoder. Average pooling is then performed on the pixel confidence scores within each block to obtain the block-level confidence vector corresponding to the 2D visual token. Specifically: in, Indicates pixel position The reliability of the depth measurement Indicates the first Image blocks corresponding to each two-dimensional visual token This represents the side length of an image block, and each image block contains... 1 pixel, This represents the total number of two-dimensional visual tokens. Indicates the index of the two-dimensional visual token. Indicates the first Block-level confidence vectors corresponding to each two-dimensional visual token; S440: Project the 3D point cloud token back onto the image plane using camera extrinsic parameters, sample the confidence scores at corresponding locations, and obtain the confidence vector of the 3D geometric token. .
[0011] Preferably, S500 includes: S510: Using a 2D visual token sequence as the query and a 3D geometric token sequence as the key and value, a cross-modal attention mechanism is constructed, specifically as follows: in, , , It is a learnable linear projection matrix. For feature dimension, For querying the matrix, The key matrix, For value matrices, This is the original attention weight matrix. This is the scaling factor; S520: To introduce confidence gating, the confidence vector corresponding to the 3D geometric token sequence is... As a soft mask, attention weights are injected to suppress the attention weights of key-value pairs corresponding to depth-untrusted regions: in, For the suppressed attention weights, Added as a bias term In the original attention scores (logits) before normalization, the 3D geometric tokens with low confidence are... The weights are then close to zero, so in deep failure regions, the model automatically ignores unreliable 3D geometric information, ultimately obtaining fused multimodal features. for: in, This is element-wise multiplication weighted by token. The block-level weights control the relative contribution of the two-dimensional features and the fused three-dimensional features at each token location.
[0012] Preferably, S600 includes: S610: Enables natural language operation commands The language token sequence is encoded by the pre-trained language encoder Llama. ,in, The number of language tokens; S620: Will fuse multimodal features With language token sequence Cross-modal alignment is performed using a multi-layer Transformer decoder to obtain the aligned joint feature representation; S630: Combine the joint feature representation with the current pose input action head of the end effector to regress and generate the target pose action sequence of the end effector. Specifically: in For the current moment, To predict the length of the time domain, For action generator; The motion generator uses a diffusion strategy or flow matching as its motion generator, outputting a target pose motion sequence. Including the target position of the end effector The posture quaternion and gripper / finger-pressing state, the action sequence represents the complete trajectory of the corresponding robot end effector from the current pose to pressing the target button and then retracting.
[0013] Preferably, after S600, end-to-end training of the entire network is also included. The training loss of the entire network consists of three parts, specifically: in, For the total loss, The motion regression loss is used to monitor the prediction accuracy of the robotic arm's motion sequences. To predict loss based on confidence level, A contrast loss for visual-language alignment, used to encourage the fusion of visual features and linguistic instructions for alignment in the embedding space; and These are the weight hyperparameters for the confidence prediction loss and the contrastive loss for visual-language alignment, respectively. The network is trained end-to-end by collecting demonstration data of pressing operations in a simulation environment, then fine-tuned using real machine data, and finally deployed to the target robotic arm to complete the pressing operation of the substation secondary cabinet.
[0014] In a second aspect, a substation cabinet robot pressing operating system integrating three-dimensional semantic information is provided, including: The multimodal acquisition module is used to acquire multimodal input data of the robot operation scene. The multimodal input data includes RGB images, depth images, natural language operation commands and the current pose of the robot arm end effector. The RGB images and depth images are preprocessed to obtain preprocessed RGB images and depth images. The 2D visual token sequence extraction module is used to extract features from the preprocessed RGB image using a 2D visual encoder to obtain a 2D visual token sequence. The 3D geometric token sequence extraction module is used to construct a 3D geometric feature representation based on the preprocessed depth image through 3D spatial back projection, and obtain a 3D geometric token sequence. The confidence prediction module is used to evaluate the confidence of each pixel in the preprocessed depth image through a confidence prediction network, and obtain pixel-level confidence information that reflects the reliability of depth measurement. The adaptive fusion module is used to construct a cross-modal attention mechanism by using a two-dimensional visual token sequence as the guide and a three-dimensional geometric token sequence as the guided feature. It uses confidence information to suppress the attention response from untrusted regions in the cross-modal attention mechanism, and adaptively weights and fuses the suppressed attention output with the two-dimensional visual token sequence according to the confidence information to obtain fused multimodal features. The motion generation module is used to encode natural language operation instructions into language feature sequences, perform cross-modal alignment between the fused multimodal features and the language feature sequences, and generate robot motion sequences based on the aligned features and the current pose of the end effector.
[0015] In a third aspect, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a substation cabinet robot pressing operation method that integrates three-dimensional semantic information.
[0016] This invention fundamentally solves the problem of existing VLA models' lack of perception of 3D geometric structure by explicitly converting depth maps into 3D geometric feature representations, enabling the model to accurately distinguish adjacent buttons that look similar but have different spatial structures. Simultaneously, a confidence prediction network is introduced to evaluate the reliability of the depth data. Combined with an adaptive fusion mechanism based on confidence information, 3D geometric information dominates decision-making in reliable depth regions to ensure operational accuracy, while automatically reverting to 2D visual information in unreliable depth regions to avoid noise contamination, thus achieving a balance between accuracy and robustness. Attached Figure Description
[0017] Figure 1 This is a flowchart of a substation cabinet robot pressing operation method that integrates three-dimensional semantic information in one embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the principle of a substation cabinet robot pressing operation method that integrates three-dimensional semantic information in one embodiment of the present invention. Detailed Implementation
[0018] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.
[0019] This invention aims to address two key technical problems of existing VLA models in the scenario of precise press-operated secondary switchgear in substations: First, existing methods directly input the depth map as the fourth channel of RGB into a 2D vision Transformer without explicitly modeling the 3D geometry inherent in the depth. This results in the model's inability to accurately distinguish between adjacent buttons on the switchgear panel that are densely packed, highly similar in appearance, but only protrude a few millimeters, leading to biases in the prediction of press poses. Second, substation secondary switchgear is equipped with transparent protective panels and numerous reflective indicator lights. Consumer-grade depth cameras are severely ineffective at depth measurement on such surfaces. Existing methods assume global depth reliability and directly input noisy depth into the model, contaminating downstream decision-making and drastically reducing the system's robustness in real-world deployment environments. This invention fundamentally solves these problems through an explicit 3D geometric prior modeling and uncertainty perception fusion mechanism.
[0020] This invention employs a three-branch coupled network structure: a 2D vision branch extracts semantic and texture features from RGB images; a 3D geometry branch back-projects the depth image into a point cloud and extracts 3D geometric prior features using a lightweight point cloud encoder; and a confidence prediction branch takes the RGB image and the original depth map as input and outputs a pixel-level depth confidence map. The outputs of the three branches are adaptively fused using a confidence-gated cross-modal attention mechanism: in high-confidence regions, 3D geometric features dominate the decision-making, enabling the model to accurately perceive the button's 3D position and height difference; in low-confidence regions, the fusion weights automatically fall back, prioritizing 2D visual features to avoid noise-induced depth contamination of downstream strategies. The fused multimodal features are cross-modal aligned with the natural language instruction encoding, ultimately regressing the pressing action sequence of the end effector by the action head. Simultaneously, the confidence prediction branch uses the error between the real depth in the simulation environment and the simulated sensor depth as a supervisory signal for training, enabling the model to automatically learn and identify depth failure areas such as transparency and reflection.
[0021] The overall flowchart of the present invention is as follows: Figure 1 As shown in the schematic diagram, the principle is as follows: Figure 2 As shown, during the perception stage, two-dimensional visual feature extraction, three-dimensional geometric prior modeling, and depth confidence prediction are performed simultaneously. Adaptive fusion is achieved through cross-modal attention with confidence gating, ultimately outputting a pressing action sequence. This improves the geometric perception capability and robustness of the VLA model in the pressing operation of substation cabinets.
[0022] (I) Data Acquisition and Preprocessing S100: Acquire multimodal input data of the robot operation scene, including RGB image, depth image, natural language operation instructions and the current pose of the end effector of the robotic arm. Preprocess the RGB image and depth image to obtain the preprocessed RGB image and depth image.
[0023] In one embodiment, S100 includes: S110: Acquire multimodal input data of the robot operation scenario, including RGB images and depth images captured by the robotic arm wrist or a third-view RGB-D camera, and natural language operation commands issued by the substation dispatch system. and the current pose of the robotic arm's end effector. ; S120: Normalize the RGB image, perform invalid value masking on the depth image to remove invalid depth pixels, and scale the normalized RGB image and the masked depth image to a uniform network input resolution of 224×224 to obtain the preprocessed RGB image. and preprocessed depth map .
[0024] Specifically, this step provides input data with a uniform format and consistent scale for subsequent multi-branch feature extraction.
[0025] (ii) Two-dimensional visual feature extraction branch S200: A two-dimensional visual token sequence is obtained by extracting features from the preprocessed RGB image using a two-dimensional visual encoder.
[0026] In one embodiment, S200 includes: The preprocessed RGB image is divided into segments by a 2D visual encoder. Each image block is divided into non-overlapping image blocks, and after linear projection, a two-dimensional visual token sequence is obtained, specifically: in, For token feature dimensions, Represents a two-dimensional visual encoder. The total number of image blocks. The height of the image. The width of the image. The side length of each image block. It is a two-dimensional visual token sequence used to encode the texture information, color information, button label text information, and indicator light status information of the display panel in the robot operation scenario.
[0027] Specifically, in this embodiment, the two-dimensional visual encoder uses a pre-trained visual SigLIP as the backbone network.
[0028] (III) Three-dimensional geometric prior modeling branch S300: Based on the preprocessed depth image, a 3D spatial backprojection is performed to construct a 3D geometric feature representation, resulting in a 3D geometric token sequence.
[0029] Specifically, the purpose of this step is to explicitly convert depth information into a three-dimensional geometric representation and extract geometric features, so that the model can obtain an accurate perception of the raised structure of the cabinet buttons and the spatial relationship between adjacent buttons.
[0030] In one embodiment, S300 includes: S310: Utilizing the intrinsic parameter matrix of an RGB-D camera The preprocessed depth image Each valid pixel in The back projection onto a 3D point in the camera coordinate system is as follows: This yields the original point cloud. Specifically: in, These are the x and y coordinates of the pixel, respectively. For the effective depth pixels, , Let these be the coordinates of the camera's principal point. , To normalize the focal length for the camera, , , These represent the x-coordinate, y-coordinate, and depth coordinates of the current pixel's 3D spatial point in the camera coordinate system. These are the x-coordinate, y-coordinate, and depth coordinates of the i-th effective depth pixel in the original point cloud, respectively. S320: Add color features sampled from the corresponding positions in the preprocessed RGB image as an extra channel to each point in the original point cloud, resulting in an original point cloud with color information. To control computational load and balance spatial distribution, the farthest point of the original point cloud containing color information is sampled to obtain a fixed number of points. Standardized point cloud ; S330: Standardizing point clouds Input a lightweight 3D encoder, at each level Grouping queries by ball with the query point as the center and radius The local geometry is constructed by sampling neighborhood points, and then local features are extracted through a shared multilayer perceptron and max pooling, specifically: in, Indicates the first In the hierarchy, query point Local features, Indicates query point Center, radius The set of neighboring points within, For neighborhood point index, For neighborhood points Three-dimensional spatial coordinates, For neighborhood points In the previous level Features Represents a multilayer perceptron mapping; S340: After multi-level feature extraction, the point cloud features are aligned to the same dimension as the 2D visual token through linear projection. The resulting three-dimensional geometric token sequence is as follows: in, This represents the final number of point cloud tokens. This is a sequence of 3D geometric tokens used to represent the protrusion height information of objects in the scene, the distance information between adjacent objects, and the surface normal vector information. It enables the model to distinguish adjacent buttons that look similar but have different depths.
[0031] Specifically, in this embodiment, the 3D encoder uses PointNet++ as the backbone to extract layered geometric features. This invention does not employ the conventional approach of directly stitching together depth images as the fourth RGB channel. Instead, it backprojects the original depth map into a colored point cloud using camera intrinsics, and then uses a lightweight PointNet++ encoder to extract layered geometric features, outputting a 3D geometric token sequence aligned with the dimensions of the 2D visual tokens. This method allows the model to explicitly obtain 3D geometric structure information such as the protrusion height of the cabinet buttons, the spacing between adjacent buttons, and the panel normal vector, fundamentally solving the problem of existing VLA models having difficulty distinguishing densely adjacent buttons and having large prediction errors in press poses.
[0032] Furthermore, besides using point clouds and PointNet++ encoders to achieve 3D geometric prior modeling, other 3D representation methods can be used as alternatives. One is voxel representation based on truncated signed distance fields (TSDF): the depth map is fused into a predefined 3D voxel grid, each voxel stores its signed distance to the nearest surface, and geometric features are extracted through a sparse convolutional network. This approach has memory advantages in large-scale scenes, but requires high voxel resolution for millimeter-level protrusions such as cabinet buttons, resulting in significant computational overhead. A second approach is binary voxel representation based on an occupied grid, using a 3D U-Net encoder to extract features. This approach is simple in structure, but geometric accuracy is limited by voxel resolution. A third approach is to use Point Transformer instead of PointNet++ as the point cloud backbone network, leveraging a self-attention mechanism to model the global geometric relationships of the point cloud. This can achieve better feature representation in button-dense scenes, but the number of parameters and computational overhead are relatively large.
[0033] (iv) Deep Confidence Prediction Branch S400: The confidence prediction network evaluates the confidence of each pixel in the preprocessed depth image to obtain pixel-level confidence information that reflects the reliability of depth measurement.
[0034] Specifically, to address the practical deployment pain point of deep failure caused by substation panels and reflective indicator lights, a pixel-level depth confidence map is predicted to provide gating signals for subsequent fusion.
[0035] In one embodiment, S400 includes: S410: The confidence prediction network employs a lightweight encoder-decoder architecture, with input consisting of a channel-level concatenation of the preprocessed RGB image and the preprocessed depth image. The output is a single-channel pixel-level confidence map, specifically: in, The output is a pixel-level confidence map, which includes pixel locations. The reliability of the depth measurement is indicated by 1, which represents complete reliability, and 0, which represents complete unreliability. S420: The training supervision signal for the confidence prediction network is constructed as follows: Simultaneously acquire the ground truth depth in the simulation environment. and simulated sensor depth The pixel-level confidence ground truth is constructed based on the relative error between the two at the same pixel location, specifically: in, For temperature coefficient, To prevent small constants with a denominator of zero, pixel position The simulated sensor depth at the location, pixel position The truth depth at that point pixel position The confidence level is the true value at the given position. This construction method makes the confidence level of pixels with larger errors lower. Using binary cross-entropy loss The confidence prediction network is trained as follows: S430: Aligns with the 2D visual token structure, and displays pixel-level confidence maps. The image is divided into blocks according to the block partitioning method of the 2D visual encoder. Average pooling is then performed on the pixel confidence scores within each block to obtain the block-level confidence vector corresponding to the 2D visual token. Specifically: in, Indicates pixel position The reliability of the depth measurement Indicates the first Image blocks corresponding to each two-dimensional visual token This represents the side length of an image block, and each image block contains... 1 pixel, This represents the total number of two-dimensional visual tokens. Indicates the index of the two-dimensional visual token. Indicates the first Block-level confidence vectors corresponding to each two-dimensional visual token; S440: Project the 3D point cloud token back onto the image plane using camera extrinsic parameters, sample the confidence scores at corresponding locations, and obtain the confidence vector of the 3D geometric token. .
[0036] Specifically, this invention addresses the practical deployment challenge of severe depth failure caused by secondary panel panels and reflective indicator lights in substations. It designs an independent, lightweight confidence prediction network that uses a pre-processed RGB channel concatenation with the original depth channel as input to output a pixel-level confidence map. Simultaneously, it automatically constructs a pixel-level confidence monitoring signal by utilizing the relative error between the ground truth depth and the simulated sensor depth in the simulation environment. This allows the model to identify depth failure areas such as transparent and reflective surfaces without manual annotation, achieving a high degree of correspondence with real substation deployment scenarios.
[0037] (v) Cross-modal attention fusion with confidence gating S500: A cross-modal attention mechanism is constructed using a two-dimensional visual token sequence as the guide and a three-dimensional geometric token sequence as the guided feature. The confidence information is used to suppress the attention response from the untrusted region in the cross-modal attention mechanism. Based on the confidence information, the suppressed attention output is adaptively weighted and fused with the two-dimensional visual token sequence to obtain fused multimodal features.
[0038] In one embodiment, S500 includes: S510: Using a 2D visual token sequence as the query and a 3D geometric token sequence as the key and value, a cross-modal attention mechanism is constructed, specifically as follows: in, , , It is a learnable linear projection matrix. For feature dimension, For querying the matrix, The key matrix, For value matrices, This is the original attention weight matrix. This is the scaling factor; S520: To introduce confidence gating, the confidence vector corresponding to the 3D geometric token sequence is... As a soft mask, attention weights are injected to suppress the attention weights of key-value pairs corresponding to depth-untrusted regions: in, For the suppressed attention weights, Added as a bias term In the original attention scores (logits) before normalization, the 3D geometric tokens with low confidence are... The weights are then close to zero, so in deep failure regions, the model automatically ignores unreliable 3D geometric information, ultimately obtaining fused multimodal features. for: in, This is element-wise multiplication weighted by token. The block-level weights control the relative contribution of the two-dimensional features and the fused three-dimensional features at each token location.
[0039] The physical meaning of this formula is that when the depth of the image region corresponding to a certain patch is reliable, the fused features are dominated by three-dimensional geometric information to provide accurate button spatial positioning; when the depth of the region is unreliable, the fused features automatically fall back to two-dimensional visual features to avoid depth noise from ruining the model's decision.
[0040] Specifically, this invention proposes dividing the pixel-level confidence map into patch-level confidence vectors through average pooling. These vectors are then used as the log-biased soft mask for cross-modal attention logits and as the gating weights for the final feature fusion. In regions with high confidence, the fused features are dominated by 3D geometric information, providing accurate button spatial positioning; in regions with low confidence, the fused weights automatically revert to 2D visual features to avoid noise and depth contamination. This mechanism achieves adaptive behavior of "3D dominance when depth is reliable, and 2D fallback when depth is unreliable."
[0041] Furthermore, in addition to employing pixel-level confidence prediction combined with patch-level token soft-gating fusion, the fusion granularity can be flexibly adjusted. One approach is a pure patch-level confidence prediction scheme: directly outputting confidence scores in patch units, omitting the pixel-to-patch pooling mapping step, resulting in a simpler structure but sacrificing spatial precision. Another approach is a pure pixel-level feature fusion scheme: before patching, performing pixel-by-pixel weighted fusion of RGB and point cloud projection features in pixel space, then uniformly feeding them into ViT to extract tokens. This method offers high spatial precision but has weak compatibility with existing ViT pre-trained weights. A third approach is a multi-scale joint fusion scheme: injecting confidence signals of different granularities into multiple layers of ViT, adjusting feature fusion layer by layer. Theoretically, this can balance local details and global semantics, but training complexity and computational overhead increase significantly. Furthermore, the roles of Query / Key / Value in cross-modal attention fusion can be interchanged, i.e., using 3D tokens as Query and 2D tokens as Key / Value, forming a dual fusion branch that complements this invention.
[0042] (vi) Visual-language alignment and action prediction S600: Encodes natural language operation instructions into language feature sequences, performs cross-modal alignment between fused multimodal features and language feature sequences, and generates robot motion sequences based on the aligned features and the current pose of the end effector.
[0043] In one embodiment, S600 includes: S610: Enables natural language operation commands The language token sequence is encoded by the pre-trained language encoder Llama. ,in, The number of language tokens; S620: Will fuse multimodal features With language token sequence Cross-modal alignment is performed using a multi-layer Transformer decoder to obtain the aligned joint feature representation; S630: Combine the joint feature representation with the current pose input action head of the end effector to regress and generate the target pose action sequence of the end effector. Specifically: in For the current moment, To predict the length of the time domain, For action generator; The motion generator uses a diffusion strategy or flow matching as its motion generator, outputting a target pose motion sequence. Including the target position of the end effector The posture quaternion and gripper / finger-pressing state, the action sequence represents the complete trajectory of the corresponding robot end effector from the current pose to pressing the target button and then retracting.
[0044] (vii) Loss Function and End-to-End Optimization In one embodiment, S600 is followed by end-to-end training of the entire network, where the training loss of the entire network consists of three parts, specifically: in, For the total loss, For the action regression loss, either L1 loss or the denoising loss of the diffusion model is used to supervise the prediction accuracy of the robotic arm's action sequence. To predict loss based on confidence level, A contrast loss for visual-language alignment, used to encourage the fusion of visual features and linguistic instructions for alignment in the embedding space; and These are the weight hyperparameters for the confidence prediction loss and the contrastive loss for visual-language alignment, respectively. In a simulation environment, 50-100 demonstration data points of pressing operations are collected through expert teleoperation or reinforcement learning to train the overall network end-to-end. Then, fine-tuning is performed using real machine data, and finally, the network is deployed to the Franka or UR5 target robotic arm to complete the pressing operation of the substation secondary cabinet.
[0045] The core improvements of this invention are reflected in the following three aspects: 1. Explicit 3D Geometric Prior Modeling: Traditional VLA models feed the depth map as the fourth channel of RGB into a 2D ViT, where depth information is only processed through 2D convolution / attention, resulting in implicit loss of 3D spatial structure. This invention backprojects the depth into a point cloud and extracts layered geometric features using PointNet++, enabling the model to explicitly obtain 3D geometric information such as button protrusion height and spacing between adjacent buttons. The technical effect is that in scenarios with densely packed buttons on a display cabinet, the model can distinguish adjacent buttons with similar appearance but different depths through geometric features, significantly improving the accuracy of press pose prediction.
[0046] 2. Explicit Prediction and Supervision of Depth Uncertainty: Existing methods assume global depth reliability and do not consider depth failure issues caused by transparent panels and reflective surfaces. This invention automatically constructs a confidence supervision signal by using the relative error between the ground truth depth and the simulated sensor depth in the simulation environment, and trains an independent confidence prediction network. The technical effect is that the model can automatically identify depth failure areas such as panels and reflective indicator lights without manual annotation, perfectly corresponding to the real deployment environment of substations.
[0047] 3. Adaptive Cross-Modal Fusion with Confidence Gating: This invention maps pixel-level confidence scores to patch-level weights, which serve as soft masks for cross-modal attention and gating signals for final feature weighting, respectively. This achieves adaptive behavior where "3D features dominate when the depth is reliable, and revert to 2D features when the depth is unreliable." The technical benefits are: in complex scenarios involving transparent panels in substation cabinets, the model can benefit from the improved accuracy brought by 3D geometry in areas with high confidence, such as stainless steel panels, while avoiding noise contamination in the panel area, achieving a balance between accuracy and robustness.
[0048] To verify the effectiveness of the technical solution of this invention, experiments were conducted in a substation secondary switchgear simulation environment built with Isaac Sim and on a self-built real machine platform. In the simulation environment, the switchgear was modeled according to the actual dimensions of a substation relay protection panel, with 10 industrial buttons, several indicator lights, and a transparent protective panel distributed on the panel. The real machine platform used a Franka robotic arm paired with an Intel RealSense D435i depth camera, and a simple switchgear was built using industrial buttons and a 3mm panel. Eighty press operation demonstration data points were collected via teleoperation, covering 10 types of operation commands, including "pressing the reset button" and "pressing the fault recorder start button."
[0049] The experimental results are shown in Table 1. In the simulation environment, the overall success rate of this invention reached 87.7%, which is an improvement of 3.2, 11.0, and 16.3 percentage points compared to the three mainstream baseline methods: OpenVLA, ACT, and DiffusionPolicy, respectively. On a real device platform, this invention achieved a success rate of 82.9% in 200 press tests, an improvement of 3.4, 10.9, and 15.4 percentage points compared to the baseline methods, verifying the effectiveness of the method in real deployment scenarios. Regarding press accuracy, the average press error of this invention is only 2.8 mm, significantly better than OpenVLA's 4.5 mm, ACT's 6.2 mm, and DiffusionPolicy's 8.7 mm. Especially in button press tasks with transparent panel obstruction and indicator light reflection areas, the baseline methods suffer from significant depth noise leading to large press position deviations. This invention, however, automatically identifies depth failure areas through a confidence gating mechanism and reverts to RGB visual features, stabilizing the press position error within 3 mm. Ablation experiments further show that the overall success rate drops significantly after removing the three-dimensional geometric prior branches, and the accuracy loss is particularly obvious in transparent panel scenarios after removing confidence-gated fusion, proving that the two core innovations of this invention are indispensable to the final performance.
[0050] Table 1. Experimental results on the pressing operation task of the secondary switchgear in the substation. Experimental results fully demonstrate the effectiveness and robustness of the proposed technical solution in the pressing operation task of substation secondary switchgear. This invention not only solves the problem of distinguishing adjacent buttons through explicit three-dimensional geometric prior modeling, but also effectively addresses the deep failure problem caused by panels and reflective indicator lights through an uncertainty perception fusion mechanism, providing an innovative and practical solution for the engineering implementation of remote switchgear operation in unattended substations.
[0051] In one embodiment, a substation cabinet robot pressing operating system integrating three-dimensional semantic information is also provided, comprising: The multimodal acquisition module is used to acquire multimodal input data of the robot operation scene. The multimodal input data includes RGB images, depth images, natural language operation commands and the current pose of the robot arm end effector. The RGB images and depth images are preprocessed to obtain preprocessed RGB images and depth images. The 2D visual token sequence extraction module is used to extract features from the preprocessed RGB image using a 2D visual encoder to obtain a 2D visual token sequence. The 3D geometric token sequence extraction module is used to construct a 3D geometric feature representation based on the preprocessed depth image through 3D spatial back projection, and obtain a 3D geometric token sequence. The confidence prediction module is used to evaluate the confidence of each pixel in the preprocessed depth image through a confidence prediction network, and obtain pixel-level confidence information that reflects the reliability of depth measurement. The adaptive fusion module is used to construct a cross-modal attention mechanism by using a two-dimensional visual token sequence as the guide and a three-dimensional geometric token sequence as the guided feature. It uses confidence information to suppress the attention response from untrusted regions in the cross-modal attention mechanism, and adaptively weights and fuses the suppressed attention output with the two-dimensional visual token sequence according to the confidence information to obtain fused multimodal features. The motion generation module is used to encode natural language operation instructions into language feature sequences, perform cross-modal alignment between the fused multimodal features and the language feature sequences, and generate robot motion sequences based on the aligned features and the current pose of the end effector.
[0052] Specific limitations regarding the substation cabinet robot pressing operating system incorporating 3D semantic information can be found in the above-described limitations regarding the pressing operation method of the substation cabinet robot incorporating 3D semantic information, and will not be repeated here. Each module in the aforementioned substation cabinet robot pressing operating system incorporating 3D semantic information can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0053] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a substation cabinet robot pressing operation method that integrates three-dimensional semantic information.
[0054] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0055] The foregoing has provided a detailed description of the substation cabinet robot pressing operation method and system integrating three-dimensional semantic information provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention, and the descriptions of the embodiments above are only for the purpose of helping to understand the core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.
Claims
1. A method for robotic pressing operation of substation cabinets integrating three-dimensional semantic information, characterized in that, Includes the following steps: S100: Acquire multimodal input data of the robot operation scene, including RGB image, depth image, natural language operation instructions and the current pose of the end effector of the robotic arm, preprocess the RGB image and depth image to obtain the preprocessed RGB image and depth image; S200: A two-dimensional visual token sequence is obtained by extracting features from the preprocessed RGB image using a two-dimensional visual encoder; S300: Based on the preprocessed depth image, perform 3D spatial back projection to construct a 3D geometric feature representation and obtain a 3D geometric token sequence; S400: The confidence prediction network evaluates the confidence of each pixel in the preprocessed depth image to obtain pixel-level confidence information that reflects the reliability of depth measurement. S500: A cross-modal attention mechanism is constructed using a two-dimensional visual token sequence as the guide and a three-dimensional geometric token sequence as the guided feature. The confidence information is used to suppress the attention response from the untrusted region in the cross-modal attention mechanism. Based on the confidence information, the suppressed attention output is adaptively weighted and fused with the two-dimensional visual token sequence to obtain fused multimodal features. S600: Encodes natural language operation instructions into language feature sequences, performs cross-modal alignment between fused multimodal features and language feature sequences, and generates robot motion sequences based on the aligned features and the current pose of the end effector.
2. The method according to claim 1, characterized in that, S100 includes: S110: Acquire multimodal input data of the robot operation scenario, including RGB images and depth images captured by the robotic arm wrist or a third-view RGB-D camera, and natural language operation commands issued by the substation dispatch system. and the current pose of the robotic arm's end effector. ; S120: Normalize the RGB image, perform invalid value masking on the depth image to remove invalid depth pixels, and scale the normalized RGB image and the masked depth image to a uniform network input resolution of 224×224 to obtain the preprocessed RGB image. and preprocessed depth map .
3. The method according to claim 2, characterized in that, S200 includes: The preprocessed RGB image is divided into segments by a 2D visual encoder. Each image block is divided into non-overlapping image blocks, and after linear projection, a two-dimensional visual token sequence is obtained, specifically: in, For token feature dimensions, Represents a two-dimensional visual encoder. The total number of image blocks. The height of the image. The width of the image. The side length of each image block. It is a two-dimensional visual token sequence used to encode the texture information, color information, button label text information, and indicator light status information of the display panel in the robot operation scenario.
4. The method according to claim 3, characterized in that, The S300 includes: S310: Utilizing the intrinsic parameter matrix of an RGB-D camera The preprocessed depth image Each valid pixel in The back projection onto a 3D point in the camera coordinate system is as follows: This yields the original point cloud. Specifically: in, These are the x and y coordinates of the pixel, respectively. For the effective depth pixels, , Let these be the coordinates of the camera's principal point. , To normalize the focal length for the camera, , , These represent the x-coordinate, y-coordinate, and depth coordinates of the current pixel's 3D spatial point in the camera coordinate system. These are the x-coordinate, y-coordinate, and depth coordinates of the i-th effective depth pixel in the original point cloud, respectively. S320: Add color features sampled from the corresponding positions in the preprocessed RGB image as an extra channel to each point in the original point cloud, resulting in an original point cloud with color information. To control computational load and balance spatial distribution, the farthest point of the original point cloud containing color information is sampled to obtain a fixed number of points. Standardized point cloud ; S330: Standardizing point clouds Input a lightweight 3D encoder, at each level Grouping queries by ball with the query point as the center and radius The local geometry is constructed by sampling neighborhood points, and then local features are extracted through a shared multilayer perceptron and max pooling, specifically: in, Indicates the first In the hierarchy, query point Local features, Indicates query point Center, radius The set of neighboring points within, For neighborhood point index, For neighborhood points Three-dimensional spatial coordinates, For neighborhood points In the previous level Features Represents a multilayer perceptron mapping; S340: After multi-level feature extraction, the point cloud features are aligned to the same dimension as the 2D visual token through linear projection. The resulting three-dimensional geometric token sequence is as follows: in, This represents the final number of point cloud tokens. It is a sequence of three-dimensional geometric tokens used to represent the convex height information of objects in the scene, the distance information between adjacent objects, and the surface normal vector information.
5. The method according to claim 4, characterized in that, The S400 includes: S410: The confidence prediction network employs a lightweight encoder-decoder architecture, with input consisting of a channel-level concatenation of the preprocessed RGB image and the preprocessed depth image. The output is a single-channel pixel-level confidence map, specifically: in, This is the output pixel-level confidence map; S420: The training supervision signal for the confidence prediction network is constructed as follows: Simultaneously acquire the ground truth depth in the simulation environment. and simulated sensor depth The pixel-level confidence ground truth is constructed based on the relative error between the two at the same pixel location, specifically: in, For temperature coefficient, To prevent small constants with a denominator of zero, pixel position The simulated sensor depth at the location, pixel position The truth depth at that point pixel position The confidence level is the true value at the given position. This construction method makes the confidence level of pixels with larger errors lower. Using binary cross-entropy loss The confidence prediction network is trained as follows: S430: Aligns with the 2D visual token structure, and displays pixel-level confidence maps. The image is divided into blocks according to the block partitioning method of the 2D visual encoder. Average pooling is then performed on the pixel confidence scores within each block to obtain the block-level confidence vector corresponding to the 2D visual token. Specifically: in, Indicates pixel position The reliability of the depth measurement Indicates the first Image blocks corresponding to each two-dimensional visual token This represents the side length of an image block, and each image block contains... 1 pixel, This represents the total number of two-dimensional visual tokens. Indicates the index of the two-dimensional visual token. Indicates the first Block-level confidence vectors corresponding to each two-dimensional visual token; S440: Project the 3D point cloud token back onto the image plane using camera extrinsic parameters, sample the confidence scores at corresponding locations, and obtain the confidence vector of the 3D geometric token. .
6. The method according to claim 5, characterized in that, The S500 includes: S510: Using a 2D visual token sequence as the query and a 3D geometric token sequence as the key and value, a cross-modal attention mechanism is constructed, specifically as follows: in, , , It is a learnable linear projection matrix. For feature dimension, For querying the matrix, The key matrix, For value matrices, This is the original attention weight matrix. This is the scaling factor; S520: To introduce confidence gating, the confidence vector corresponding to the 3D geometric token sequence is... As a soft mask, attention weights are injected to suppress the attention weights of key-value pairs corresponding to depth-untrusted regions: in, For the suppressed attention weights, Added as a bias term In the original attention scores (logits) before normalization, the 3D geometric tokens with low confidence are... The weights are then close to zero, so in deep failure regions, the model automatically ignores unreliable 3D geometric information, ultimately obtaining fused multimodal features. for: in, This is element-wise multiplication weighted by token. The block-level weights control the relative contribution of the two-dimensional features and the fused three-dimensional features at each token location.
7. The method according to claim 6, characterized in that, The S600 includes: S610: Enables natural language operation commands The language token sequence is encoded by the pre-trained language encoder Llama. ,in, The number of language tokens; S620: Will fuse multimodal features With language token sequence Cross-modal alignment is performed using a multi-layer Transformer decoder to obtain the aligned joint feature representation; S630: Combine the joint feature representation with the current pose input action head of the end effector to regress and generate the target pose action sequence of the end effector. Specifically: in For the current moment, To predict the length of the time domain, For action generator; The motion generator uses a diffusion strategy or flow matching as its motion generator, outputting a target pose motion sequence. Including the target position of the end effector The posture quaternion and gripper / finger-pressing state, the action sequence represents the complete trajectory of the corresponding robot end effector from the current pose to pressing the target button and then retracting.
8. The method according to claim 7, characterized in that, Following S600, end-to-end training of the entire network is also included. The training loss of the entire network consists of three parts, specifically: in, For the total loss, The motion regression loss is used to monitor the prediction accuracy of the robotic arm's motion sequences. To predict loss based on confidence level, A contrast loss for visual-language alignment, used to encourage the fusion of visual features and linguistic instructions for alignment in the embedding space; and These are the weight hyperparameters for the confidence prediction loss and the contrastive loss for visual-language alignment, respectively. The network is trained end-to-end by collecting demonstration data of pressing operations in a simulation environment, then fine-tuned using real machine data, and finally deployed to the target robotic arm to complete the pressing operation of the substation secondary cabinet.
9. A substation cabinet robot pressing operating system integrating three-dimensional semantic information, characterized in that, include: The multimodal acquisition module is used to acquire multimodal input data of the robot operation scene. The multimodal input data includes RGB images, depth images, natural language operation commands and the current pose of the robot arm end effector. The RGB images and depth images are preprocessed to obtain preprocessed RGB images and depth images. The 2D visual token sequence extraction module is used to extract features from the preprocessed RGB image using a 2D visual encoder to obtain a 2D visual token sequence. The 3D geometric token sequence extraction module is used to construct a 3D geometric feature representation based on the preprocessed depth image through 3D spatial back projection, and obtain a 3D geometric token sequence. The confidence prediction module is used to evaluate the confidence of each pixel in the preprocessed depth image through a confidence prediction network, and obtain pixel-level confidence information that reflects the reliability of depth measurement. The adaptive fusion module is used to construct a cross-modal attention mechanism by using a two-dimensional visual token sequence as the guide and a three-dimensional geometric token sequence as the guided feature. It uses confidence information to suppress the attention response from untrusted regions in the cross-modal attention mechanism, and adaptively weights and fuses the suppressed attention output with the two-dimensional visual token sequence according to the confidence information to obtain fused multimodal features. The motion generation module is used to encode natural language operation instructions into language feature sequences, perform cross-modal alignment between the fused multimodal features and the language feature sequences, and generate robot motion sequences based on the aligned features and the current pose of the end effector.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.
