Method, device and equipment for predicting three-dimensional attitude of substation inspection robot and storage medium

Abstract

The invention discloses a transformer substation inspection robot three-dimensional attitude prediction method, device and equipment and a storage medium, and the method comprises the steps: obtaining a multi-view image, and inertial measurement unit data and historical attitude data of an inspection robot, and obtaining a three-dimensional attitude prediction result based on the multi-view image, the inertial measurement unit data and the historical attitude data; the method comprises the following steps: predicting a motion path of an inspection robot, generating fusion features containing multi-view image information and inspection robot attitude information, outputting a predicted three-dimensional attitude by a model, calculating a total loss value of the predicted three-dimensional attitude relative to a corresponding actual value so as to update model parameters, and finally inputting real-time three-dimensional attitude information and an environment image into the model. And outputting a three-dimensional attitude prediction result of the inspection robot. Therefore, the fusion features consider the multi-view image information and the attitude information of the inspection robot, so that attitude prediction and path planning are carried out cooperatively, the finally obtained three-dimensional attitude prediction result is more comprehensive, and the control stability and trajectory execution precision of the inspection robot are improved.

Description

Technical Field

[0001] This application relates to the field of substation inspection technology, and more specifically, to a method, device, equipment, and storage medium for predicting the three-dimensional posture of a substation inspection robot. Background Technology

[0002] With the advancement of new power systems and smart grid strategies, substations, as core power hubs, have become a key focus of intelligent operation and maintenance in terms of operational safety and equipment management. Traditional manual inspections suffer from problems such as complex environments, high labor intensity, low efficiency, and difficulty in structuring data. Intelligent inspection systems based on mobile robots have emerged to address these issues. In particular, wheeled inspection robots are widely used in mixed indoor and outdoor scenarios due to their flexibility and low cost.

[0003] With the improvement of the intelligence level of inspection robots, path planning becomes more flexible, requiring capabilities such as obstacle avoidance and autonomous target identification. This places higher demands on their environmental adaptability and attitude control. Outdoor substations have uneven surfaces such as ramps, while indoor equipment is dense and passageways are narrow. Robots need to precisely adjust their three-dimensional attitude (pitch, roll, and yaw angles) to maintain stable operation.

[0004] Current technologies are mostly based on two-dimensional modeling, and relying solely on real-time feedback can easily lead to control jitter and instability. Multi-sensor data is processed independently, and path planning and attitude prediction run independently. The attitude prediction results are inconsistent with the robot's actual motion target, resulting in low control stability and trajectory execution accuracy.

[0005] Therefore, how to integrate more comprehensive prediction of the three-dimensional posture of substation inspection robots in order to improve the control stability and trajectory execution accuracy of inspection robots is an issue that needs attention. Summary of the Invention

[0006] In view of the above problems, this application provides a method, device, equipment and storage medium for predicting the three-dimensional posture of a substation inspection robot, so as to more comprehensively predict the three-dimensional posture of the substation inspection robot and improve the control stability and trajectory execution accuracy of the inspection robot.

[0007] To achieve the above objectives, the following specific solutions are proposed:

[0008] A method for predicting the three-dimensional pose of a substation inspection robot, characterized by comprising:

[0009] Acquire real-time three-dimensional posture information of the inspection robot, as well as environmental images of the substation where the inspection robot is located;

[0010] The real-time 3D pose information and the environmental image are input into a pre-reinforced robot pose prediction model, and the 3D pose prediction result of the inspection robot is output.

[0011] The reinforcement training process of the robot posture prediction model includes:

[0012] The system acquires multi-view images of the substation environment where the inspection robot is located, inertial measurement unit data of the inspection robot, and historical attitude data of the inspection robot.

[0013] Based on the multi-view images, the inertial measurement unit data, and the historical posture data, the motion path of the inspection robot is predicted, and a fusion feature containing multi-view image information and inspection robot posture information is generated.

[0014] Based on the fused features and the motion path, the robot pose prediction model in the training phase outputs the predicted three-dimensional pose of the inspection robot.

[0015] Calculate the total loss value of the predicted 3D pose and the motion path relative to the corresponding actual values, and update the model parameters of the robot pose prediction model based on the total loss value under the backpropagation algorithm to complete the reinforcement training of the robot pose prediction model.

[0016] Optionally, based on the multi-view images, the inertial measurement unit data, and the historical attitude data, the motion path of the inspection robot is predicted, and a fusion feature containing multi-view image information and inspection robot attitude information is generated, including:

[0017] Feature extraction is performed on the multi-view images to obtain multi-view image coding features;

[0018] Based on the multi-view image encoding features, predict the motion path of the inspection robot;

[0019] Based on the inertial measurement unit data, the historical posture data is encoded in machine language to obtain the posture coding features of the inspection robot.

[0020] The multi-view image coding features are fused with the inspection robot posture coding features to obtain fused features.

[0021] Optionally, the multi-view image coding features are fused with the inspection robot posture coding features to obtain fused features, including:

[0022] Construct the first key matrix and the first value matrix of the multi-view image coding features;

[0023] Construct the second key matrix and the second value matrix of the posture encoding features of the inspection robot;

[0024] The first association weight between the first key matrix and the second key matrix, and the second association weight between the first value matrix and the second value matrix are calculated using an attention mechanism.

[0025] The first key matrix and the second key matrix are fused using the first association weight, and the first value matrix and the second value matrix are fused using the second association weight to obtain the fused feature.

[0026] Optionally, the multi-view image coding features are fused with the inspection robot posture coding features to obtain fused features, including:

[0027] Calculate the third association weight between the motion path and the historical posture data;

[0028] Calculate the fourth association weight between the motion path and the multi-view image coding features;

[0029] The third association weight is used as the fusion weight of the inspection robot posture coding feature, and the fourth association weight is used as the fusion weight of the multi-view image coding feature. The multi-view image coding feature is fused with the inspection robot posture coding feature to obtain the fused feature.

[0030] Optionally, calculating the total loss value of the predicted 3D pose and the motion path relative to the corresponding actual values ​​includes:

[0031] The total loss value of the predicted 3D pose and the motion path relative to the corresponding actual values ​​is calculated using the following formula:

[0032]

[0033] in, The total loss value is the sum of the predicted 3D pose and the motion path relative to the corresponding actual values. For the predicted number of future time steps, Let be the loss value of the motion path at time step t relative to the actual motion path at time step t. Let be the loss value of the predicted 3D pose at time step t relative to the actual 3D pose at time step t. These are the weighting coefficients for the motion path. These are the weighting coefficients for the three-dimensional attitude.

[0034] Optionally, after predicting the motion path of the inspection robot based on the multi-view images, the inertial measurement unit data, and the historical attitude data, and generating fused features containing multi-view image information and inspection robot attitude information, the method further includes:

[0035] The fused features are obtained by performing nonlinear transformation and dimension adjustment on the fused features through a feedforward neural network.

[0036] Optionally, the inspection robot has a built-in six-axis / nine-axis inertial measurement unit, and the substation environment where the inspection robot is located is equipped with multiple cameras;

[0037] Acquiring inertial measurement unit data of the inspection robot includes:

[0038] The inertial measurement unit data of the inspection robot is obtained through the inertial measurement unit;

[0039] Acquire multi-view images of the substation environment where the inspection robot is located, including:

[0040] The inspection robot acquires multi-view images of the substation environment using various cameras.

[0041] A device for predicting the three-dimensional pose of a substation inspection robot includes:

[0042] The real-time information acquisition unit is used to acquire the real-time three-dimensional posture information of the inspection robot, as well as the environmental image of the substation where the inspection robot is located.

[0043] The model prediction unit is used to input the real-time three-dimensional pose information and the environmental image into a pre-reinforced robot pose prediction model and output the three-dimensional pose prediction result of the inspection robot.

[0044] The data acquisition unit is used to acquire multi-view images of the substation environment where the inspection robot is located, inertial measurement unit data of the inspection robot, and historical attitude data of the inspection robot.

[0045] The fusion feature generation unit is used to predict the motion path of the inspection robot based on the multi-view images, the inertial measurement unit data and the historical posture data, and generate fusion features containing multi-view image information and inspection robot posture information.

[0046] The model output unit during training is used to output the predicted three-dimensional posture of the inspection robot based on the fused features and the motion path using the robot posture prediction model in the training phase.

[0047] The backpropagation training unit is used to calculate the total loss value of the predicted 3D pose and the motion path relative to the corresponding actual values, and to update the model parameters of the robot pose prediction model based on the total loss value under the backpropagation algorithm, thereby completing the reinforcement training of the robot pose prediction model.

[0048] Optionally, the fusion feature generation unit includes:

[0049] A multi-view feature extraction neural network is used to extract features from the multi-view images to obtain multi-view image encoded features;

[0050] The path planning and decision-making module is used to predict the motion path of the inspection robot based on the encoded features of the multi-view images.

[0051] A feature encoding generator is used to encode the historical posture data into machine language based on the inertial measurement unit data to obtain the posture encoding features of the inspection robot.

[0052] The feature fusion module is used to fuse the multi-view image coding features with the inspection robot posture coding features to obtain fused features.

[0053] Optionally, the feature fusion module includes:

[0054] The first feature fusion submodule is used to construct the first key matrix and the first value matrix of the multi-view image coding features;

[0055] The second feature fusion submodule is used to construct the second key matrix and the second value matrix of the posture encoding features of the inspection robot.

[0056] The third feature fusion submodule is used to calculate the first association weight between the first key matrix and the second key matrix, and the second association weight between the first value matrix and the second value matrix through an attention mechanism;

[0057] The fourth feature fusion submodule is used to fuse the first key matrix and the second key matrix through the first association weight, and to fuse the first value matrix and the second value matrix through the second association weight to obtain fused features.

[0058] Optionally, the feature fusion module includes:

[0059] The fifth feature fusion submodule is used to calculate the third association weight between the motion path and the historical posture data;

[0060] The sixth feature fusion submodule is used to calculate the fourth association weight between the motion path and the multi-view image coding features;

[0061] The seventh feature fusion submodule is used to use the third association weight as the fusion weight of the inspection robot posture coding feature and the fourth association weight as the fusion weight of the multi-view image coding feature to fuse the multi-view image coding feature with the inspection robot posture coding feature to obtain the fused feature.

[0062] Optionally, the back gradient propagation training unit includes:

[0063] The total loss calculation unit is used to calculate the total loss value of the predicted 3D pose and the motion path relative to the corresponding actual values ​​using the following formula:

[0064]

[0065] in, The total loss value is the sum of the predicted 3D pose and the motion path relative to the corresponding actual values. For the predicted number of future time steps, Let be the loss value of the motion path at time step t relative to the actual motion path at time step t. Let be the loss value of the predicted 3D pose at time step t relative to the actual 3D pose at time step t. These are the weighting coefficients for the motion path. These are the weighting coefficients for the three-dimensional pose.

[0066] The model parameter update unit is used to update the model parameters of the robot posture prediction model based on the total loss value under the backpropagation algorithm, thereby completing the reinforcement training of the robot posture prediction model.

[0067] Optionally, the device may also include:

[0068] The fusion feature adjustment unit is used to predict the motion path of the inspection robot based on the multi-view images, the inertial measurement unit data and the historical posture data, and generate fusion features containing multi-view image information and inspection robot posture information. Then, it performs nonlinear transformation and dimension adjustment on the fusion features through a feedforward neural network to obtain the fusion features after nonlinear transformation and dimension adjustment.

[0069] Optionally, the inspection robot has a built-in six-axis / nine-axis inertial measurement unit, and the substation environment where the inspection robot is located is equipped with multiple cameras;

[0070] The data acquisition unit includes:

[0071] An inertial measurement unit data acquisition unit for inspection robots is used to acquire inertial measurement unit data of the inspection robot through the inertial measurement unit.

[0072] The multi-view image acquisition unit is used to acquire multi-view images of the substation environment where the inspection robot is located through various cameras.

[0073] A device for predicting the three-dimensional pose of a substation inspection robot, including a memory and a processor;

[0074] The memory is used to store programs;

[0075] The processor is used to execute the program to implement the various steps of the method for predicting the three-dimensional posture of the substation inspection robot as described above.

[0076] A storage medium storing a computer program, which, when executed by a processor, implements the various steps of the method for predicting the three-dimensional posture of a substation inspection robot as described above.

[0077] By employing the aforementioned technical solution, this application acquires multi-view images of the substation environment where the inspection robot is located, inertial measurement unit (IMU) data of the inspection robot, and historical attitude data of the inspection robot. Based on these data, the motion path of the inspection robot is predicted, and a fusion feature containing multi-view image information and inspection robot attitude information is generated. Using the robot attitude prediction model in the training phase, based on the fusion feature and motion path, the predicted 3D attitude of the inspection robot is output. The total loss value of the predicted 3D attitude and motion path relative to the corresponding actual values ​​is calculated. Under the backpropagation algorithm, the model parameters of the robot attitude prediction model are updated based on the total loss value, completing the reinforcement training of the robot attitude prediction model. Finally, real-time 3D attitude information and environmental images are input into the robot attitude prediction model to output the 3D attitude prediction result of the inspection robot. Therefore, the fusion feature takes into account both multi-view image information and inspection robot attitude information, enabling attitude prediction and path planning to proceed collaboratively, resulting in a more comprehensive 3D attitude prediction result and improving the control stability and trajectory execution accuracy of the inspection robot. Attached Figure Description

[0078] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0079] Figure 1 A schematic diagram of a process for implementing three-dimensional posture prediction of a substation inspection robot provided in an embodiment of this application;

[0080] Figure 2 This is a schematic diagram of a process for implementing reinforcement training of a robot posture prediction model according to an embodiment of this application;

[0081] Figure 3 This is a schematic diagram of a process for fusing pose coding features and multi-view image coding features according to an embodiment of this application;

[0082] Figure 4A system logic diagram illustrating a method for predicting the three-dimensional posture of a substation inspection robot based on vision-inertial measurement, provided in an embodiment of this application.

[0083] Figure 5 A schematic diagram of a device for predicting the three-dimensional posture of a substation inspection robot, provided in an embodiment of this application;

[0084] Figure 6 This is a schematic diagram of the structure of a device for predicting the three-dimensional posture of a substation inspection robot, provided in an embodiment of this application. Detailed Implementation

[0085] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0086] The proposed solution can be implemented based on a terminal with data processing capabilities, such as a computer, cloud, or server.

[0087] Next, combined Figure 1 The three-dimensional posture prediction method for the substation inspection robot of this application may include the following steps:

[0088] Step S110: Obtain the real-time three-dimensional posture information of the inspection robot, as well as the environmental image of the substation where the inspection robot is located.

[0089] Specifically, real-time 3D attitude information can include the instantaneous values ​​of the inspection robot's pitch angle, roll angle, and yaw angle. Multiple wide-angle cameras deployed around the inspection robot are activated simultaneously to capture environmental images of the substation environment, providing a comprehensive 360° field of view. This clearly records indoor and outdoor power equipment, passageway width, ground conditions, and the distribution of various obstacles, providing real-time and complete environmental and self-status data support for subsequent processing.

[0090] Step S120: Input the real-time 3D pose information and environmental image into the pre-reinforced robot pose prediction model, and output the 3D pose prediction result of the inspection robot.

[0091] The reinforcement training process for the robot pose prediction model can be referenced from [reference needed]. Figure 2 Specifically, it can include:

[0092] Step S210: Obtain multi-view images of the substation environment where the inspection robot is located, inertial measurement unit data of the inspection robot, and historical attitude data of the inspection robot.

[0093] Specifically, the substation environment where the inspection robot operates is equipped with multiple cameras, allowing it to acquire multi-view images of the substation environment. These multi-view images are visual data used by the substation inspection robot to perceive its surroundings, generated synchronously by multiple wide-angle cameras installed around the robot. These cameras are distributed to cover the robot's 360° field of view, eliminating blind spots and fully capturing environmental details, including various electrical equipment within the substation, the width and flatness of indoor and outdoor passageways, ground elevation changes or obstacles, and the spatial structural features of the surrounding environment. This type of image data comes directly from the mature visual acquisition components pre-installed on the inspection robot, eliminating the need for additional equipment deployment and providing stable and continuous environmental visual information.

[0094] Inertial measurement data (IMU) represents data acquired by an inspection robot from its inertial measurement unit (IMU). It is core data reflecting the robot's motion state and is collected in real-time by the robot's built-in six-axis or nine-axis IMU. The inspection robot typically has a built-in six-axis or nine-axis IMU. A six-axis IMU may include a three-axis accelerometer and a three-axis gyroscope, while a nine-axis IMU may additionally include a three-axis magnetometer. The core data acquired includes the robot's acceleration in three-dimensional space (e.g., linear acceleration and deceleration during movement) and angular velocity (e.g., rotational speed during turning or tilting). This data accurately records every instantaneous change in the robot's motion and exhibits strong temporal continuity. This type of data originates from the standard inertial measurement unit of the inspection robot and is considered basic output data of the existing sensor system; it can be stably acquired without the need for additional acquisition equipment.

[0095] Step S220: Based on multi-view images, inertial measurement unit data and historical posture data, predict the motion path of the inspection robot and generate fused features containing multi-view image information and inspection robot posture information.

[0096] Understandably, multi-view images and inertial measurement unit data represent two different modalities. The fusion feature after fusing the two modalities takes into account both multi-view image information and the robot's posture information, enabling posture prediction and path planning to proceed in tandem, resulting in a more comprehensive 3D posture prediction result.

[0097] Specifically, the process of predicting the motion path of the inspection robot based on multi-view images, inertial measurement unit data, and historical attitude data, and generating fused features containing multi-view image information and inspection robot attitude information, can be found in [reference needed]. Figure 3 Specifically, it can include:

[0098] Step S221: Extract features from the multi-view images to obtain multi-view image coding features.

[0099] Specifically, the extraction of multi-view image coding features can be achieved by a multi-view feature extraction module. This module can directly reuse image features already extracted from existing visual perception systems, rather than re-extracting features from the original image.

[0100] Understandably, raw multi-view images may contain a large number of irrelevant pixels. Encoding can filter out key environmental features, simplify the amount of data, reduce the computational burden, and improve processing efficiency. Multi-view images come from different wide-angle cameras, resulting in viewpoint bias. Encoding can transform the scattered 360° visual information into standardized features, eliminating viewpoint fragmentation and forming a complete and consistent representation of environmental features. The encoded feature format is regular and highly compatible with non-visual data such as inertial measurement units, laying the foundation for subsequent cross-modal feature fusion and avoiding fusion barriers caused by data format differences. Encoding can be directly based on the intermediate results of the robot's existing visual perception system, eliminating the need to repeatedly process the original images, reducing redundant computation, and requiring no additional hardware, thus lowering application costs.

[0101] Step S222: Predict the motion path of the inspection robot based on the multi-view image encoding features.

[0102] The prediction of the motion path can be achieved by the path planning and decision module. The path planning and decision module can optimize the trajectory prediction results output by the existing substation inspection robot path planning system to avoid repeated path search and decision calculation.

[0103] Specifically, when predicting the movement path of the inspection robot based on multi-view image encoding features, the path planning and decision-making module first reads the core environmental information extracted after encoding. This core environmental information can include the location and outline of electrical equipment within the substation (such as circuit breakers, disconnectors, and busbar structures), the width and flatness of indoor and outdoor passageways, and the distribution of ground obstacles, among other key features. Combining the spatial structural characteristics of the substation scenario, algorithms can analyze passable areas, avoid densely populated equipment areas and obstacle points, and generate a continuous sequence of future movement trajectory points that meets the requirements of the inspection task. This ensures that the trajectory adapts to the robot's mobility and is compatible with mixed scenarios such as narrow indoor passageways and outdoor ramps.

[0104] Understandably, the encoded features remove redundant pixels from the original image, retaining only key environmental information. This allows path prediction to focus on accessibility assessment, reducing interference from invalid data, improving prediction accuracy, and preventing the path from deviating from the inspection target. The 360° panoramic encoded features can fully cover the robot's blind spots, avoiding missed obstacles due to missing viewpoints and ensuring path safety. The path planning and decision-making module can directly reuse the encoded results of the existing vision system without repeatedly processing the original image, significantly reducing computational overhead, improving the real-time performance of path prediction, and adapting to the rapid decision-making needs of the robot during dynamic inspections.

[0105] Step S223: Based on the inertial measurement unit data, perform machine language encoding on the historical posture data to obtain the posture encoding features of the inspection robot.

[0106] The inertial measurement unit data can be uniformly encoded by the feature encoding generator module.

[0107] Specifically, when encoding historical attitude data into machine language, the raw data can be preprocessed first to remove abnormal fluctuation values ​​and complete temporal alignment. Then, through encoding operations such as serialization conversion, numerical normalization, and temporal feature mapping, the discrete attitude parameters can be transformed into structured and standardized vector or matrix forms, while fully preserving the temporal continuity and motion change patterns of the data. Finally, attitude encoding features that can be directly read and calculated by the machine can be generated.

[0108] Understandably, historical posture data can provide historical motion context support, and the encoded features completely record the robot's past posture evolution trajectory, helping the model to grasp its motion laws and providing historical basis for predicting future postures; posture encoding features can adapt to the needs of multimodal fusion, and the standardized encoding format can be seamlessly compatible with multi-view image encoding features, eliminating fusion barriers caused by differences in data formats.

[0109] Step S224: Fuse the multi-view image coding features with the inspection robot posture coding features to obtain the fused features.

[0110] Specifically, the process of fusing multi-view image coding features with inspection robot posture coding features to obtain fused features can include two implementation methods. The first implementation method can construct a key-value matrix to calculate the correlation weights between matrices, thereby fusing the matrices to obtain fused features; the second implementation method can calculate the correlation weights between the motion path and the multi-view images and historical postures respectively, thereby weighted fusing the coding features to obtain fused features.

[0111] The first implementation involves constructing a first key matrix and a first value matrix for multi-view image encoding features, and a second key matrix and a second value matrix for the inspection robot's pose encoding features. An attention mechanism is used to calculate the first association weights between the first and second key matrices, and the second association weights between the first and second value matrices. The first and second key matrices are then fused using the first association weights, and the first and second value matrices are further fused using the second association weights to obtain the fused features.

[0112] The attention mechanism can be a cross-attention mechanism. A cross-attention mechanism can be designed with dynamic weight allocation strategies to automatically identify differences in feature distributions from different sensor data through learnable modal embedding vectors. For time-series inertial measurement unit posture data of an inspection robot, the attention layer enhances its temporal continuity; for spatially structured visual features, it focuses on preserving their geometric relationships.

[0113] Understandably, the cross-attention mechanism can adaptively capture the coupling relationship between environmental visual information and the motion state of the substation inspection robot, thereby generating a unified fusion feature representation with physical consistency and spatiotemporal alignment.

[0114] The second implementation involves calculating the third association weight between the motion path and historical posture data, and the fourth association weight between the motion path and multi-view image coding features. The first association weight is used as the fusion weight for the inspection robot's posture coding features, and the second association weight is used as the fusion weight for the multi-view image coding features. The multi-view image coding features are then fused with the inspection robot's posture coding features to obtain the fused features.

[0115] Furthermore, after predicting the motion path of the inspection robot based on multi-view images, inertial measurement unit data, and historical posture data, and generating fused features containing multi-view image information and inspection robot posture information, the fused features can be nonlinearly transformed and dimensionally adjusted by a feedforward neural network to obtain the fused features after nonlinear transformation and dimension adjustment.

[0116] Understandably, the robot pose prediction model consists of multiple cascaded deep learning neural networks based on attention mechanisms for decoders. Each decoding layer can perform the same feature processing flow. First, using the planned trajectory features as a query, the association weights between these features and the historical pose and visual fusion features are dynamically calculated to achieve cross-modal feature fusion. Second, a feedforward neural network is used to perform nonlinear transformations and dimensional adjustments on the fused features. After each stage of processing, residual connections and layer normalization operations are performed to ensure stable gradient propagation. Finally, the output layer of the robot pose prediction model uses a pose prediction network (such as a fully connected network) to map the multidimensional features output by the decoder into a 4-dimensional quaternion vector, which is then normalized to obtain a unit quaternion pose prediction.

[0117] Step S230: Using the robot pose prediction model in the training phase, based on fused features and motion path, output the predicted 3D pose of the inspection robot.

[0118] It is understandable that the predicted 3D pose output by the robot pose prediction model during the training phase can reflect the current prediction effect of the robot pose prediction model. Therefore, based on this current prediction effect, the degree to which the robot pose prediction model needs improvement can be evaluated by comparing it with the actual data.

[0119] Step S240: Calculate the total loss value of the predicted three-dimensional pose and motion path relative to the corresponding actual value, and update the model parameters of the robot pose prediction model based on the total loss value under the backpropagation algorithm to complete the reinforcement training of the robot pose prediction model.

[0120] Understandably, updating model parameters through backpropagation during training allows for continuous optimization of these parameters, thereby gradually improving prediction accuracy.

[0121] The total loss value of the predicted 3D pose and motion path relative to the corresponding actual values ​​can be calculated using the following formula:

[0122]

[0123] in, To predict the total loss value of 3D pose and motion path relative to the corresponding actual values, For the predicted number of future time steps, Let be the loss value of the motion path at time step t relative to the actual motion path at time step t. Let be the loss value of the predicted 3D pose at time step t relative to the actual 3D pose at time step t. These are the weighting coefficients for the motion path. These are the weighting coefficients for the three-dimensional attitude.

[0124] Understandably, the loss value calculated using this loss function can update the model parameters of the robot posture prediction model used to predict 3D posture, and it can also update the parameters of the path planning decision module used to predict motion paths.

[0125] Furthermore, once the performance of the entire system reaches the expected standard, the optimized model parameters can be saved for subsequent applications.

[0126] Combination Figure 4 The three-dimensional posture prediction scheme of the substation inspection robot provided in this embodiment is described. Figure 4 The leftmost layer is the sensor data input layer, containing multi-view cameras and inertial measurement unit (IMU) sensors. A multi-view feature extraction neural network is used to extract multi-view encoded features from the robot's surround-view images. A feature encoding generator encodes the historical posture of the inspection robot based on IMU data, obtaining the robot's posture encoded features. The middle layer is the path planning decision module and the modal perception cross-attention mechanism encoding feature fusion. The path planning decision module integrates visual feature data for path planning prediction and braking control; the modal perception cross-attention mechanism encoding feature fusion combines the inspection robot's posture encoded features and multi-view encoded features through a modal perception cross-attention mechanism. Finally, the robot posture prediction model uses the fused features to predict the target position based on the path prediction result, generating the final future robot posture prediction result.

[0127] The method for predicting the 3D posture of a substation inspection robot provided in this embodiment acquires multi-view images of the substation environment where the inspection robot is located, inertial measurement unit (IMU) data of the inspection robot, and historical posture data of the inspection robot. Based on the multi-view images, IMU data, and historical posture data, the method predicts the motion path of the inspection robot and generates fused features that include multi-view image information and inspection robot posture information. Using the robot posture prediction model in the training phase, based on the fused features and motion path, the method outputs the predicted 3D posture of the inspection robot. The method calculates the total loss value of the predicted 3D posture and motion path relative to the corresponding actual values, and updates the model parameters of the robot posture prediction model based on the total loss value using the backpropagation algorithm, thus completing the reinforcement training of the robot posture prediction model. Finally, the real-time 3D posture information and environmental images are input into the robot posture prediction model to output the 3D posture prediction result of the inspection robot. Therefore, the fused features take into account both multi-view image information and inspection robot posture information, enabling posture prediction and path planning to proceed in tandem, resulting in a more comprehensive 3D posture prediction result and improving the control stability and trajectory execution accuracy of the inspection robot.

[0128] The following describes the device for predicting the three-dimensional posture of a substation inspection robot provided in the embodiments of this application. The device for predicting the three-dimensional posture of a substation inspection robot described below can be referred to in correspondence with the method for predicting the three-dimensional posture of a substation inspection robot described above.

[0129] See Figure 5 , Figure 5 This is a schematic diagram of a device for predicting the three-dimensional posture of a substation inspection robot, as disclosed in an embodiment of this application.

[0130] like Figure 5 As shown, the device may include:

[0131] The real-time information acquisition unit 11 is used to acquire the real-time three-dimensional posture information of the inspection robot and the environmental image of the substation where the inspection robot is located.

[0132] The model prediction unit 12 is used to input the real-time three-dimensional posture information and the environmental image into a pre-reinforced robot posture prediction model and output the three-dimensional posture prediction result of the inspection robot.

[0133] The data acquisition unit 13 is used to acquire multi-view images of the substation environment where the inspection robot is located, inertial measurement unit data of the inspection robot, and historical attitude data of the inspection robot.

[0134] The fusion feature generation unit 14 is used to predict the motion path of the inspection robot based on the multi-view image, the inertial measurement unit data and the historical posture data, and generate fusion features containing multi-view image information and inspection robot posture information.

[0135] The training model output unit 15 is used to output the predicted three-dimensional posture of the inspection robot based on the fused features and the motion path using the robot posture prediction model in the training phase.

[0136] The backpropagation training unit 16 is used to calculate the total loss value of the predicted three-dimensional pose and the motion path relative to the corresponding actual values, and to update the model parameters of the robot pose prediction model based on the total loss value under the backpropagation algorithm, thereby completing the reinforcement training of the robot pose prediction model.

[0137] Optionally, the fusion feature generation unit includes:

[0138] A multi-view feature extraction neural network is used to extract features from the multi-view images to obtain multi-view image encoded features;

[0139] The path planning and decision-making module is used to predict the motion path of the inspection robot based on the encoded features of the multi-view images.

[0140] A feature encoding generator is used to encode the historical posture data into machine language based on the inertial measurement unit data to obtain the posture encoding features of the inspection robot.

[0141] The feature fusion module is used to fuse the multi-view image coding features with the inspection robot posture coding features to obtain fused features.

[0142] Optionally, the feature fusion module includes:

[0143] The first feature fusion submodule is used to construct the first key matrix and the first value matrix of the multi-view image coding features;

[0144] The second feature fusion submodule is used to construct the second key matrix and the second value matrix of the posture encoding features of the inspection robot.

[0145] The third feature fusion submodule is used to calculate the first association weight between the first key matrix and the second key matrix, and the second association weight between the first value matrix and the second value matrix through an attention mechanism;

[0146] The fourth feature fusion submodule is used to fuse the first key matrix and the second key matrix through the first association weight, and to fuse the first value matrix and the second value matrix through the second association weight to obtain fused features.

[0147] Optionally, the feature fusion module includes:

[0148] The fifth feature fusion submodule is used to calculate the third association weight between the motion path and the historical posture data;

[0149] The sixth feature fusion submodule is used to calculate the fourth association weight between the motion path and the multi-view image coding features;

[0150] The seventh feature fusion submodule is used to use the third association weight as the fusion weight of the inspection robot posture coding feature and the fourth association weight as the fusion weight of the multi-view image coding feature to fuse the multi-view image coding feature with the inspection robot posture coding feature to obtain the fused feature.

[0151] Optionally, the back gradient propagation training unit includes:

[0152] The total loss calculation unit is used to calculate the total loss value of the predicted 3D pose and the motion path relative to the corresponding actual values ​​using the following formula:

[0153]

[0154] in, The total loss value is the sum of the predicted 3D pose and the motion path relative to the corresponding actual values. For the predicted number of future time steps, Let be the loss value of the motion path at time step t relative to the actual motion path at time step t. Let be the loss value of the predicted 3D pose at time step t relative to the actual 3D pose at time step t. These are the weighting coefficients for the motion path. These are the weighting coefficients for the three-dimensional pose.

[0155] The model parameter update unit is used to update the model parameters of the robot posture prediction model based on the total loss value under the backpropagation algorithm, thereby completing the reinforcement training of the robot posture prediction model.

[0156] Optionally, the device may also include:

[0157] The fusion feature adjustment unit is used to predict the motion path of the inspection robot based on the multi-view images, the inertial measurement unit data and the historical posture data, and generate fusion features containing multi-view image information and inspection robot posture information. Then, it performs nonlinear transformation and dimension adjustment on the fusion features through a feedforward neural network to obtain the fusion features after nonlinear transformation and dimension adjustment.

[0158] Optionally, the inspection robot has a built-in six-axis / nine-axis inertial measurement unit, and the substation environment where the inspection robot is located is equipped with multiple cameras;

[0159] The data acquisition unit includes:

[0160] An inertial measurement unit data acquisition unit for inspection robots is used to acquire inertial measurement unit data of the inspection robot through the inertial measurement unit.

[0161] The multi-view image acquisition unit is used to acquire multi-view images of the substation environment where the inspection robot is located through various cameras.

[0162] The three-dimensional pose prediction device for substation inspection robots provided in this application embodiment can be applied to devices for predicting the three-dimensional pose of substation inspection robots, such as terminals like mobile phones and computers. Optionally, Figure 6 The hardware structure block diagram of the device for predicting the three-dimensional pose of a substation inspection robot is shown. (Refer to...) Figure 6The hardware structure of the substation inspection robot three-dimensional pose prediction device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4.

[0163] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0164] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0165] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0166] The memory stores a program, which the processor can call. The program is used for:

[0167] Acquire real-time three-dimensional posture information of the inspection robot, as well as environmental images of the substation where the inspection robot is located;

[0168] The real-time 3D pose information and the environmental image are input into a pre-reinforced robot pose prediction model, and the 3D pose prediction result of the inspection robot is output.

[0169] The reinforcement training process of the robot posture prediction model includes:

[0170] The system acquires multi-view images of the substation environment where the inspection robot is located, inertial measurement unit data of the inspection robot, and historical attitude data of the inspection robot.

[0171] Based on the multi-view images, the inertial measurement unit data, and the historical posture data, the motion path of the inspection robot is predicted, and a fusion feature containing multi-view image information and inspection robot posture information is generated.

[0172] Based on the fused features and the motion path, the robot pose prediction model in the training phase outputs the predicted three-dimensional pose of the inspection robot.

[0173] Calculate the total loss value of the predicted 3D pose and the motion path relative to the corresponding actual values, and update the model parameters of the robot pose prediction model based on the total loss value under the backpropagation algorithm to complete the reinforcement training of the robot pose prediction model.

[0174] Optionally, the refined and extended functions of the program can be found in the description above.

[0175] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0176] Acquire real-time three-dimensional posture information of the inspection robot, as well as environmental images of the substation where the inspection robot is located;

[0177] The real-time 3D pose information and the environmental image are input into a pre-reinforced robot pose prediction model, and the 3D pose prediction result of the inspection robot is output.

[0178] The reinforcement training process of the robot posture prediction model includes:

[0179] The system acquires multi-view images of the substation environment where the inspection robot is located, inertial measurement unit data of the inspection robot, and historical attitude data of the inspection robot.

[0180] Based on the multi-view images, the inertial measurement unit data, and the historical posture data, the motion path of the inspection robot is predicted, and a fusion feature containing multi-view image information and inspection robot posture information is generated.

[0181] Based on the fused features and the motion path, the robot pose prediction model in the training phase outputs the predicted three-dimensional pose of the inspection robot.

[0182] Calculate the total loss value of the predicted 3D pose and the motion path relative to the corresponding actual values, and update the model parameters of the robot pose prediction model based on the total loss value under the backpropagation algorithm to complete the reinforcement training of the robot pose prediction model.

[0183] Optionally, the refined and extended functions of the program can be found in the description above.

[0184] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0185] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0186] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for predicting a three-dimensional pose of a substation inspection robot, characterized in that, The method comprises the following steps: obtaining real-time three-dimensional attitude information of a patrol robot and environment images of a substation environment where the patrol robot is located; inputting the real-time three-dimensional attitude information and the environment images into a robot attitude prediction model that is pre-reinforced trained, and outputting a three-dimensional attitude prediction result of the patrol robot; the reinforcement training process of the robot attitude prediction model comprises the following steps: obtaining multi-view images of the substation environment where the patrol robot is located, inertial measurement unit data of the patrol robot, and historical attitude data of the patrol robot; based on the multi-view images, the inertial measurement unit data and the historical attitude data, predicting a motion path of the patrol robot, and generating fusion features containing multi-view image information and patrol robot attitude information; using the robot attitude prediction model in the training stage to output a predicted three-dimensional attitude of the patrol robot based on the fusion features and the motion path; calculating the total loss value of the predicted three-dimensional attitude and the motion path relative to the corresponding actual value, and updating the model parameters of the robot attitude prediction model based on the total loss value under the back propagation algorithm, to complete the reinforcement training of the robot attitude prediction model.

2. The method of claim 1, wherein, based on the multi-view images, the inertial measurement unit data and the historical attitude data, predicting a motion path of the patrol robot, and generating fusion features containing multi-view image information and patrol robot attitude information, comprising: extracting features from the multi-view images to obtain multi-view image encoding features; predicting the motion path of the patrol robot according to the multi-view image encoding features; based on the inertial measurement unit data, machine language encoding is performed on the historical attitude data to obtain patrol robot attitude encoding features; fuse the multi-view image encoding features and the patrol robot attitude encoding features to obtain fusion features.

3. The method of claim 2, wherein, fuse the multi-view image encoding features and the patrol robot attitude encoding features to obtain fusion features, comprising: constructing a first key matrix and a first value matrix of the multi-view image encoding features; constructing a second key matrix and a second value matrix of the patrol robot attitude encoding features; calculating a first correlation weight of the first key matrix and the second key matrix, and a second correlation weight of the first value matrix and the second value matrix through an attention mechanism; fuse the first key matrix and the second key matrix through the first correlation weight, and fuse the first value matrix and the second value matrix through the second correlation weight to obtain fusion features.

4. The method of claim 2, wherein, fuse the multi-view image encoding features and the patrol robot attitude encoding features to obtain fusion features, comprising: calculating a third correlation weight of the motion path and the historical attitude data; calculating a fourth correlation weight of the motion path and the multi-view image encoding features; Take the third correlation weight as a fusion weight of the inspection robot pose encoding feature, take the fourth correlation weight as a fusion weight of the multi-view image encoding feature, fuse the multi-view image encoding feature and the inspection robot pose encoding feature, and obtain fused features.

5. The method of claim 1, wherein, The total loss value of the predicted three-dimensional pose and the motion path relative to corresponding actual values is calculated, including: The total loss value of the predicted three-dimensional pose and the motion path relative to corresponding actual values is calculated by using the following formula: wherein, is a total loss value of the predicted three-dimensional pose and the motion path with respect to corresponding actual values, is a predicted future time step, is a loss value of the motion path of the t time step with respect to an actual motion path of the t time step, is a loss value of the predicted three-dimensional pose of the t time step with respect to an actual three-dimensional pose of the t time step, is a motion path weighting coefficient, is a three-dimensional pose weighting coefficient.

6. The method of claim 1, wherein, After predicting the motion path of the inspection robot based on the multi-view image, the inertial measurement unit data and the historical pose data, and generating fused features containing multi-view image information and inspection robot pose information, the method further includes: The fused features are nonlinearly transformed and dimensionally adjusted by a feedforward neural network to obtain nonlinearly transformed and dimensionally adjusted fused features.

7. The method according to any one of claims 1 to 6, characterized in that, The inspection robot is built-in with a six-axis / nine-axis inertial measurement unit, and a substation environment where the inspection robot is located is arranged with multiple cameras; The inertial measurement unit data of the inspection robot is obtained, including: The inertial measurement unit data of the inspection robot is obtained by the inertial measurement unit; The multi-view image of the substation environment where the inspection robot is located is obtained, including: The multi-view image of the substation environment where the inspection robot is located is obtained by each camera.

8. A device for predicting a three-dimensional pose of a substation inspection robot, characterized in that, Including: A real-time information acquisition unit is configured to acquire real-time three-dimensional pose information of the inspection robot and environment images of a substation environment where the inspection robot is located; A model prediction unit is configured to input the real-time three-dimensional pose information and the environment images into a robot pose prediction model that is pre-strengthened trained, and output a three-dimensional pose prediction result of the inspection robot; A data acquisition unit is configured to acquire multi-view images of a substation environment where the inspection robot is located, inertial measurement unit data of the inspection robot, and historical pose data of the inspection robot; A fused feature generation unit is configured to predict a motion path of the inspection robot based on the multi-view images, the inertial measurement unit data and the historical pose data, and generate fused features containing multi-view image information and inspection robot pose information; A training model output unit is configured to output a predicted three-dimensional pose of the inspection robot based on the fused features and the motion path by using a robot pose prediction model in a training stage. A reverse gradient propagation training unit is configured to calculate a total loss value of the predicted three-dimensional pose and the motion path relative to corresponding actual values, and update model parameters of the robot pose prediction model based on the total loss value under a back propagation algorithm, to complete the strengthening training of the robot pose prediction model.

9. A substation inspection robot three-dimensional pose prediction device, characterized by, Including a memory and a processor; The memory is configured to store a program; The processor is configured to execute the program to implement various steps of the method for predicting a three-dimensional pose of a substation inspection robot according to any one of claims 1-7.

10. A storage medium having stored thereon a computer program, characterized in that The computer program, when executed by a processor, implements each step of the method for predicting the three-dimensional pose of a substation inspection robot according to any one of claims 1-7.

Images