A Vision Positioning and Guidance Method for Substation Bionic Operation Robots

Abstract

This application relates to the field of robot control technology, and specifically provides a vision positioning and guiding method for a bionic operation robot in a substation. The method includes: obtaining substation image data and shooting position data; constructing a three-dimensional model of the substation based on the substation image data and shooting position data, and determining the position data of a preset component; the preset component is a button or knob that needs to be operated; obtaining the current coordinates and current pose of the robotic arm in the same coordinate system of the substation three-dimensional model; controlling the robotic arm to perform a preset operation according to the current coordinates and current pose and the position data of the preset component. In this application, by obtaining substation image data, the accurate position of the button or knob can be accurately identified, and by planning the end path and actions of the robotic arm, the automatic movement and actions of the robotic arm can be realized without background operation, and the automatic operation of the bionic operation robot can be achieved.

Description

Technical Field

[0001] This application relates to the technical field of robot control, and more specifically, to a visual positioning and guiding method for a bionic operation robot based on a substation. Background Art

[0002] The application of bionic operation robots in substations is one of the important development directions of modern power system automation. By combining bionic technology, robot technology, and automatic control, bionic operation robots can perform various complex and dangerous operation tasks in the substation environment, improving the efficiency, safety, and accuracy of operations.

[0003] However, the current control of bionic operation robots is all through manual background remote control, which still requires a large amount of human resources. Summary of the Invention

[0004] The problem solved by this application is that current bionic operation robots still require manual background remote control.

[0005] To solve the above problems, the first aspect of this application provides a visual positioning and guiding method for a bionic operation robot based on a substation, including:

[0006] Obtain substation image data and shooting position data;

[0007] Based on the substation image data and shooting position data, construct a three-dimensional model of the substation and determine the position data of preset components; the preset components are buttons or knobs that need to be operated;

[0008] Obtain the current coordinates and current pose of the robotic arm in the same coordinate system of the substation three-dimensional model;

[0009] According to the current coordinates, current pose, and the position data of the preset components, control the robotic arm to perform preset operations.

[0010] The second aspect of this application provides a visual positioning and guiding device for a bionic operation robot based on a substation, which includes:

[0011] A data acquisition module for obtaining substation image data and shooting position data;

[0012] A model construction module for constructing a three-dimensional model of the substation based on the substation image data and shooting position data and determining the position data of preset components; the preset components are buttons or knobs that need to be operated;

[0013] A pose acquisition module for obtaining the current coordinates and current pose of the robotic arm in the same coordinate system of the substation three-dimensional model;

[0014] The robotic arm control module is used to control the robotic arm to perform a preset operation according to the current coordinates, the current pose, and the position data of the preset part.

[0015] The third aspect of the present application provides an electronic device, which includes: a memory and a processor; the memory is used to store a program; the processor is coupled to the memory and is used to execute the program to:

[0016] Obtain substation image data and shooting position data;

[0017] Based on the substation image data and the shooting position data, construct a three-dimensional model of the substation and determine the position data of the preset part; the preset part is a button or a knob that needs to be operated;

[0018] Obtain the current coordinates and the current pose of the robotic arm in the same coordinate system of the substation three-dimensional model;

[0019] According to the current coordinates, the current pose, and the position data of the preset part, control the robotic arm to perform a preset operation.

[0020] The fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the above-mentioned vision positioning and guiding method based on a substation bionic operation robot.

[0021] In the present application, by obtaining substation image data, the accurate position of the button or the knob can be accurately identified, and by planning the end path and actions of the robotic arm, the automatic movement and actions of the robotic arm can be realized without background operation, and the automatic operation of the bionic operation robot can be realized. Description of the Drawings

[0022] Figure 1 It is a flowchart of the vision positioning and guiding method according to an embodiment of the present application;

[0023] Figure 2 It is an architecture diagram of the recognition model of the vision positioning and guiding method according to an embodiment of the present application;

[0024] Figure 3 It is an architecture diagram of the attention module of the vision positioning and guiding method according to an embodiment of the present application;

[0025] Figure 4 It is a flowchart of the robotic arm control similar to the vision positioning and guiding method according to an embodiment of the present application;

[0026] Figure 5 It is a structural block diagram of the vision positioning and guiding device according to an embodiment of the present application;

[0027] Figure 6 It is a structural block diagram of the electronic device according to an embodiment of the present application. Detailed implementation manners

[0028] To make the above objects, features, and advantages of the present application more obvious and understandable, the following will describe in detail the specific embodiments of the present application with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0029] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should have the ordinary meanings understood by those skilled in the art to which the present application belongs.

[0030] The embodiment of the present application provides a visual positioning and guiding method for a substation bionic operation robot. The specific scheme of this method is Figures 1 - 4 shown as follows. This method can be executed by a visual positioning and guiding device for a substation bionic operation robot, and this visual positioning and guiding device for a substation bionic operation robot can be integrated in electronic devices such as a computer, a server, a computer, a server cluster, a data center, etc. Combining Figure 1 、 Figure 2 shown as follows, which is a flowchart of a visual positioning and guiding method for a substation bionic operation robot according to an embodiment of the present application; wherein, the visual positioning and guiding method for a substation bionic operation robot includes:

[0031] S101, obtaining substation image data and shooting position data;

[0032] In the present application, the substation image data is image data obtained by a camera (such as an RGB camera, a depth camera, etc.). This image data can be single-image data obtained by direct shooting or image frames selected from a video stream.

[0033] Preferably, if the substation image data is image frame data, image frames corresponding to the same time step are selected from different cameras as the substation image data output by the camera.

[0034] In the present application, the shooting position data is the coordinate data and orientation data of the camera for shooting.

[0035] Preferably, the coordinate data of cameras at different positions are coordinate data in the same coordinate system.

[0036] S102, based on the substation image data and the shooting position data, constructing a three-dimensional model of the substation and determining the position data of a preset component; the preset component is a button or a knob that needs to be operated;

[0037] S103. Obtain the current coordinates and current pose of the robotic arm in the same coordinate system of the substation three-dimensional model;

[0038] In this application, the current coordinates of the robotic arm in the same coordinate system of the substation three-dimensional model refer to the current coordinates of the robotic arm when the robot has reached the operable area.

[0039] It should be noted that when the robot performs specific operations, it can be divided into three stages: the robot travels from a distance to the operable area of the preset part; when the robot is in the operable area, the end of the robotic arm moves from the initial pose along the planned path to the preparation pose for fitting the preset part (this stage corresponds to the planned path); the end of the robotic arm starts from the preparation pose and performs operations such as screwing and pressing on the preset part (this stage corresponds to the planned action).

[0040] In this application, the operable area of the preset part refers to the area where the robot can operate on the preset part through the robotic arm when it is in this area.

[0041] In this application, for the path setting and control in the stage where the robot travels from a distance to the operable area of the preset part, in the case of generating the substation three-dimensional model, through path planning algorithms such as RRT and PRM, the optimal path can be found in the complex substation environment, avoiding obstacles and reaching the target position.

[0042] S104. Control the robotic arm to perform preset operations according to the current coordinates, current pose and the position data of the preset part.

[0043] In this application, by obtaining the substation image data to accurately identify the accurate position of the button or knob, and by planning the end path and actions of the robotic arm, the automatic movement and actions of the robotic arm can be realized without background operation, and the automatic operation of the bionic operating robot can be achieved.

[0044] In one implementation, the S102, based on the substation image data and the shooting position data, constructs a substation three-dimensional model and determines the position data of the preset part, including:

[0045] Input the substation image data into a pre-trained recognition model for recognition to determine the position information of the preset part;

[0046] Generate a substation three-dimensional model based on the substation image data and the shooting position data, and determine the position data of the preset part in the substation three-dimensional model.

[0047] In this application, the substation image data can be a depth image or a non-depth two-dimensional image (without depth data on pixel points). It should be noted that the training samples of the recognition model are in the same format as the substation image data.

[0048] In this application, when generating a three-dimensional model of a substation based on the substation image data and the shooting position data, the substation image data is a depth image, and the viewing angles of multiple depth cameras can cover the planned area of the substation.

[0049] In this application, after obtaining the depth images of multiple depth cameras, the internal parameters and relative position relationships of the cameras can be determined by calibrating the internal parameters and external parameters of each depth camera; the coordinates of each pixel in the three-dimensional space can be calculated from each depth image through the internal parameters (focal length, optical center) and depth values of the camera; the depth images obtained from multiple different viewing angles, or the point clouds corresponding to the depth images, are aligned to the same coordinate system; the point clouds in the same coordinate are fused to eliminate redundant data and generate a complete and unified three-dimensional model, which is the three-dimensional model of the substation.

[0050] Preferably, the three-dimensional model of the substation can be a local model of the substation, and the robot operates in this local model, thus greatly reducing the depth camera data and the amount of calculation required for the three-dimensional model of the substation.

[0051] In this application, given the position information of a preset part in the depth image and the three-dimensional model constructed from this depth image, the position information of the preset part in the three-dimensional model can be determined.

[0052] In one implementation, as shown in Figure 2 When inputting the substation image data into a pre-trained recognition model for recognition to determine the position information of the preset part, it includes:

[0053] Performing multiple downsampling processes on the substation image data to obtain downsampled feature maps with gradually decreasing sizes;

[0054] Performing multiple upsampling processes on the downsampled feature map with the smallest size to obtain upsampled feature maps with gradually increasing sizes;

[0055] Performing secondary downsampling and splicing processes on the upsampled feature map with the largest size to obtain a fused feature map with a gradually decreasing size; the sizes of the fused feature map, the downsampled feature map, and the upsampled feature map correspond one by one;

[0056] Inputting the fused feature maps of multiple sizes into the detection module respectively and summarizing them to obtain the types and position information of the preset parts on the substation image.

[0057] As Figure 2As shown, in the recognition model, the input substation image is downsampled to obtain downsampled feature maps of three sizes (the first size, the second size, and the third size; the sizes decrease in sequence); by upsampling the downsampled feature map of the third size, upsampled feature maps of three sizes are obtained (the third size, the second size, and the first size; the sizes increase gradually); then, by downsampling the upsampled feature map of the first size again, fused feature maps of three sizes are obtained (the first size, the second size, and the third size; the sizes decrease in sequence).

[0058] It should be noted that the feature maps at the joints of the downsampled feature maps, upsampled feature maps, and fused feature maps are the same feature maps: the downsampled feature map of the third size and the upsampled feature map of the third size are the same feature map; the upsampled feature map of the first size and the fused feature map of the first size are the same feature map.

[0059] In this application, in the form of downsampling - upsampling - downsampling, compared with the traditional downsampling - upsampling method, information can be further compressed, redundant features can be removed, and sufficient key information can be retained at the same time; this operation can extract and retain multi - level features in images or signals at different scales; in addition, by detecting and summarizing feature maps of different sizes, the preset parts can be detected at different levels, greatly increasing the accuracy of detection.

[0060] In this application, the detection results of feature maps of different sizes include position information, and there is a linear correspondence between this position information and the size of the feature map; after the size of the feature map is enlarged in equal proportion, the corresponding detected position information is also enlarged in the same proportion, so as to convert the detection results of feature maps of different sizes into the same size, and then correspond to the input substation image to obtain the detection results marked on the substation image.

[0061] In one implementation, combined with Figure 2 As shown, the multiple downsampling processes performed on the substation image data to obtain downsampled feature maps with gradually decreasing sizes include:

[0062] Performing double - layer convolution processing, class attention processing, downsampling processing, and class attention processing on the substation image data in sequence to obtain the first downsampled feature map;

[0063] Performing downsampling processing and class attention processing on the first downsampled feature map to obtain the second downsampled feature map;

[0064] Performing downsampling processing, class attention processing, and fusion processing on the second downsampled feature map to obtain the third downsampled feature map; the sizes of the first downsampled feature map, the second downsampled feature map, and the third downsampled feature map gradually decrease.

[0065] Among them, the first downsampled feature map, the second downsampled feature map, and the third downsampled feature map are the downsampled feature maps of the first size, the second size, and the third size, respectively.

[0066] In this application, the double-layer convolution is two consecutive convolution operations; through the consecutive convolution, the receptive field can be significantly increased without significantly increasing the size of the convolution kernel.

[0067] In this application, the fusion processing can be simple fusion processing or global extraction processing.

[0068] Preferably, the fusion processing is global processing. In this way, better global feature enhancement can be achieved through the global processing.

[0069] Among them, the process of the fusion processing / global extraction processing is as follows:

[0070] Input the input feature map into the 1×1 convolutional layer set in parallel to obtain the first extraction map, the second extraction map, and the third extraction map respectively; among them, the first extraction map and the third extraction map are in the HW×N format, and the second extraction map is in the N×HW format;

[0071] Multiply the first extraction map and the second extraction map to obtain a multiplied feature map;

[0072] Perform softmax processing on the multiplied feature map to obtain a multiplication coefficient;

[0073] Multiply the multiplication coefficient by the third extraction map to obtain a coefficient-multiplied map;

[0074] Add the coefficient-multiplied map and the input feature map to obtain the output feature map of the global extraction / fusion processing.

[0075] In this application, the input feature map is divided into three branches for 1×1 convolution processing. Through this convolution, the dimensions of the first branch and the third branch are adjusted to the HW×N format, and the dimension of the second branch is adjusted to the N×HW format; then multiply the first branch and the third branch to obtain a feature map with the dimension of HW×HW, and perform softmax processing to obtain a coefficient; multiply the coefficient by the third branch, and add the multiplied feature map and the input feature map to obtain the output feature map.

[0076] In this application, through 1×1 convolution, the dimensions of the first branch and the second branch are reversed, so as to realize the multiplication between feature maps; through this multiplication, the long-range dependence relationship can be captured; by setting the coefficient, the dependence relationship can be embedded in the feature map; and on this basis, the original input is added, and the residual connection is used to embed the global extraction into the model without destroying the parameters.

[0077] In this application, long-range dependencies between captured features are globally extracted, enabling the model to extract global features and making up for the defect that current models mainly extract local features.

[0078] In this application, the response at a position is calculated as the weighted sum of features at all positions, so that when processing the information at each position, it is processed based on considering all position information that can be considered.

[0079] In one embodiment, in combination with Figure 2 as shown, the smallest downsampled feature map is upsample processed multiple times to obtain upsample feature maps with gradually increasing sizes, including:

[0080] The third downsampled feature map (third upsample feature map) is upsample processed, and after upsample processing, it is concatenated with the second downsampled feature map and attention processing is performed to obtain the second upsample feature map;

[0081] The second upsample feature map is upsample processed, and after upsample processing, it is concatenated with the first downsampled feature map and attention processing is performed to obtain the first upsample feature map.

[0082] In one embodiment, in combination with Figure 2 as shown, the largest upsample feature map is downsampled and concatenated twice to obtain fusion feature maps with gradually decreasing sizes, including:

[0083] The first upsample feature map (first fusion feature map) is downsampled, and after downsampling, it is concatenated with the second upsample feature map and attention processing is performed to obtain the second fusion feature map;

[0084] The second fusion feature map is downsampled, and after downsampling, it is concatenated with the third upsample feature map / third downsampled feature map and attention processing is performed to obtain the third fusion feature map.

[0085] In combination with Figure 2 as shown, it can be seen that in the upsample feature map, the upsample feature map after upsample is concatenated with the downsampled feature map of the corresponding size respectively, so as to retain the detailed information in the downsampling process; and in the process of the second downsampling, the downsampled feature map is concatenated with the upsample feature map of the corresponding size respectively, so as to retain the detailed information in the upsample process; through this layer-by-layer long connection processing, the loss of important detailed information is avoided and the problem of gradient disappearance is alleviated.

[0086] In one embodiment, in combination with Figure 3 as shown, the process of attention extraction is:

[0087] The input feature map is subjected to convolution processing, and the convolved feature map is divided into three branches: In the first branch, the feature map is successively subjected to convolution, fully connected, ReLU processing, and fully connected processing to obtain the first branch feature map; In the second branch, the feature map is successively subjected to vertical conversion (converting the feature map of H×W×C into a feature map of H×1×C), and expansion processing (expanding the feature map of H×1×C into a feature map of H×W×C) to obtain the second branch feature map; In the third branch, the feature map is successively subjected to convolution, ReLU, convolution, and ReLU processing (the subsequent arrows in this part do not represent the specific processing process, but only directly display the size information of the H×W×C after ReLU processing) to obtain the third branch feature map; After multiplying the first branch feature map and the second branch feature map (similarly, the arrows in this part do not represent the specific processing process, but only directly display the size information of the processed H×W×C), they are fused with the third branch feature map (this part of the fusion can be an ADD operation or other fusion operations, which is determined according to the actual situation) to obtain an output feature map with the size of H×W×C.

[0088] In this application, the sizes of the input feature map and the output feature map are both in the size of H×W×C.

[0089] In this application, by dividing the feature map into three branches for processing and performing vertical direction feature processing through vertical conversion, features can be extracted from different directions, local weights can be increased, and the attention enhancement characteristics of the features can be improved.

[0090] In this application, the feature map of H×W×C can be converted into a feature map of H×1×C by taking the mean value of the numerical values in the width direction; This part is used for the processing of vertical direction features and is merged into other branches by multiplication.

[0091] In this application, the feature map of H×1×C can be expanded into a feature map of H×W×C by means of multi-layer replication, and the numerical values in the width direction on the expanded feature map of H×W×C are the same.

[0092] In this application, the setting of the detection module can be a simple CNN structure or other improved structures, so as to synchronously output the detection position / detection box and the detection type.

[0093] In one implementation, in combination with Figure 4 as shown, in S104, according to the current coordinates, current pose, and the position data of the preset part, the robotic arm is controlled to perform a preset operation, including:

[0094] S401, according to the current coordinates, current pose of the robotic arm, and the position data of the preset part, determine the planned path and planned action of the robotic arm;

[0095] S402, control the robotic arm to execute the planned path and planned actions;

[0096] S403, obtain the real-time pose data of the end of the robotic arm;

[0097] In this application,

[0098] S404, according to the deviation between the real-time pose data and the planned path and planned actions, correct the robotic arm until the execution of the planned path and planned actions is completed.

[0099] In this application, by correcting the movement of the robotic arm in real time, the deviation of the movement of the robotic arm from the planned path can be avoided.

[0100] In this application, it should be noted that during the control process of the robotic arm, if the planning in the planned path or the planning in the planned actions fails to consider various actual control factors (the design of the planned path and planned actions is not smooth enough), control errors of the robotic arm may occur, resulting in the deviation of the movement of the robotic arm from the planned path.

[0101] In one implementation, according to the current coordinates, current pose of the robotic arm and the position data of the preset part, determine the planned path of the robotic arm, including:

[0102] Obtain expert path data;

[0103] Based on the inverse reinforcement learning algorithm, infer the corresponding reward function from the expert path data;

[0104] Based on the reinforcement learning algorithm and the inferred reward function, plan the path starting from the current coordinates and ending with the position data of the preset part to obtain the planned path of the robotic arm.

[0105] In this application, the expert path data can be drawn by a robot expert or the path data with the smallest deviation of the robotic arm selected in actual operation.

[0106] In this application, summarize the expert path data through inverse reinforcement learning, summarize the corresponding reward function, and use this reward function to train new path data, so as to obtain a planned path that summarizes expert experience, thereby maximizing the possibility of ensuring that the robotic arm can move along the planned path and greatly improving the accuracy during the path movement of the robotic arm.

[0107] In one implementation, the robotic arm is a multi-axis robotic arm, and the expert data is the angle change data of multiple motors corresponding to the multi-axis robotic arm over time.

[0108] In one embodiment, inferring the corresponding reward function from the expert path data based on the inverse reinforcement learning algorithm includes:

[0109] Obtain the operating environment data and construct a simulation space;

[0110] Based on the operating environment data and the expert path data, construct a state space;

[0111] Set an action space, where the action space includes multiple joint axes of the robotic arm, and each joint axis has a forward rotation action and a reverse rotation action;

[0112] Based on the state space and the action space, construct a complete operating trajectory;

[0113] Construct a reward function model;

[0114] Model the maximum entropy of the trajectory probability to obtain a maximum entropy model;

[0115] By maximizing the log-likelihood of the expert trajectory, iteratively optimize the parameters of the reward function model to obtain the optimal reward function parameters.

[0116] In this application, the motion environment data can be substation three-dimensional model data. The simulation space can be a virtual three-dimensional space.

[0117] In this application, in the state space, it can include the operating environment state determined by the operating environment data and the angle states of each joint of the robotic arm determined by the expert path data. Specifically, it can be represented as a corresponding vector.

[0118] In this application, in the action space, the robotic arm action represents the change in the joint angles of the robotic arm at each time step (specifically, the change in the joint angles of the robotic arm at each split time point), and can also be used as a vector. The operating trajectory consists of a series of state and action sequences.

[0119] In this application, the calculation of the maximum entropy involves summing over all possible trajectories. Therefore, maximizing the log-likelihood of the expert trajectory is used for optimization.

[0120] Among them, determine the log-likelihood of the expert trajectory, and through an optimization method such as gradient descent or gradient ascent, optimize it in the direction of the maximum value until the optimization is complete.

[0121] In this application, the specific optimization process of gradient descent or ascent will not be elaborated.

[0122] In this application, the expert trajectory is the complete operating trajectory constructed.

[0123] In this application, the reward function model and the maximum entropy model can be determined according to the actual situation and will not be elaborated herein.

[0124] In this application, based on the reinforcement learning algorithm and the speculated reward function, a path starting from the current coordinate and ending with the position data of the preset part is planned to obtain the planned path of the robotic arm. Using the reward function obtained through the above operations as the reward function, a new planned path policy is trained through reinforcement learning. Then, based on this planned path policy, a path starting from the current coordinate and ending with the position data of the preset part is planned.

[0125] In this application, training can be performed through the Deep Q-Network learning method using the known reward function. The specific training process can be as follows:

[0126] Construct a simulation environment and a state space;

[0127] Set the action space;

[0128] Obtain the known reward function, construct a Deep Q-Network model and an experience replay pool. The experience replay pool includes the actions, states, rewards, and next states of the agent;

[0129] Sample from the experience replay pool and calculate the Q value based on the Deep Q-Network model;

[0130] Update the weights of the Deep Q-Network model and the weights of the target network by minimizing the loss function until the training is completed.

[0131] The Deep Q-Network model and the target network are kept consistent.

[0132] Among them, the detailed content of Deep Q-Network learning will not be elaborated herein.

[0133] It should be noted that during the training process of Deep Q-Network learning, its simulation space, state space, and action space are consistent with those in the inverse reinforcement learning process.

[0134] In one implementation, before inferring the corresponding reward function from the expert path data based on the inverse reinforcement learning algorithm, the following steps are also included:

[0135] Preprocess the expert data and convert it into a preset format. Specifically:

[0136] Segment the expert data based on time steps, and based on the segmented expert data, determine the corresponding angle values of the motors of each joint of the robotic arm at each time step; perform a secondary division on the time steps, divide each time step into several time points (the number of which is the same as the number of axes of the robotic arm), and respectively execute the motor actions of the first joint to the end joint;

[0137] In this way, the execution of the actions of the robotic arm at a certain time step is converted into the sequential execution of the actions of the first joint axis, the second joint axis, …, and the end joint axis at multiple time points, thereby greatly reducing the number of actions of the entire robotic arm at the same time step. That is to say, at the first time point, the motor action of the first joint is executed (this motor action is the action corresponding to the corresponding time step), but the motors of other joints remain unchanged. In this way, the combination of the action space is greatly reduced.

[0138] Preferably, after determining the reward function based on the inverse reinforcement learning method of this preprocessing and training a new action sequence through a new reinforcement learning method based on this reward function, it is also necessary to synthesize the actions corresponding to six consecutive time points into the actions corresponding to one time step, so as to make the actions of the robotic arm coherent and obtain the corresponding planned path.

[0139] Similarly, in this application, the process of determining the planned actions can be similar to that of the planned path, or the planned actions of different preset parts can be directly designed by experts, so as to achieve the accurate operation of the preset parts.

[0140] The embodiment of this application provides a visual positioning and guiding device for a substation bionic operation robot, which is used to execute the substation bionic operation robot visual positioning and guiding method described above in this application. The following describes the visual positioning and guiding device for the substation bionic operation robot in detail.

[0141] As Figure 5 shown, the visual positioning and guiding device for the substation bionic operation robot includes:

[0142] A data acquisition module 101, which is used to acquire substation image data and shooting position data;

[0143] A model construction module 102, which is used to construct a three-dimensional model of the substation based on the substation image data and shooting position data, and determine the position data of the preset parts; the preset parts are buttons or knobs that need to be operated;

[0144] A pose acquisition module 103, which is used to acquire the current coordinates and current pose of the robotic arm in the same coordinate system of the three-dimensional model of the substation;

[0145] A robotic arm control module 104, which is used to control the robotic arm to execute preset operations according to the current coordinates, current pose and position data of the preset parts.

[0146] In one implementation, the model construction module 102 is further used for:

[0147] Input the substation image data into a pre-trained recognition model for recognition to determine the position information of the preset components; generate a 3D model of the substation based on the substation image data and the shooting position data, and determine the position data of the preset components in the 3D model of the substation.

[0148] In one implementation, the model construction module 102 is further configured to:

[0149] Perform multiple downsampling processes on the substation image data to obtain downsampled feature maps with gradually decreasing sizes; perform multiple upsampling processes on the downsampled feature map with the smallest size to obtain upsampled feature maps with gradually increasing sizes; perform secondary downsampling and splicing processes on the upsampled feature map with the largest size to obtain fused feature maps with gradually decreasing sizes; the sizes of the fused feature maps, the downsampled feature maps, and the upsampled feature maps correspond one by one; input the fused feature maps of multiple sizes into the detection module and summarize them to obtain the types and position information of the preset components on the substation image.

[0150] In one implementation, the model construction module 102 is further configured to:

[0151] Perform double-layer convolution processing, class attention processing, downsampling processing, and class attention processing on the substation image data in sequence to obtain a first downsampled feature map; perform downsampling processing and class attention processing on the first downsampled feature map to obtain a second downsampled feature map; perform downsampling processing, class attention processing, and fusion processing on the second downsampled feature map to obtain a third downsampled feature map; the sizes of the first downsampled feature map, the second downsampled feature map, and the third downsampled feature map gradually decrease.

[0152] In one implementation, the robotic arm control module 104 is further configured to:

[0153] Determine the planned path and planned actions of the robotic arm according to the current coordinates, current pose of the robotic arm, and the position data of the preset components; control the robotic arm to execute the planned path and planned actions; obtain the real-time pose data of the end of the robotic arm; correct the robotic arm according to the deviation between the real-time pose data and the planned path and the planned actions until the planned path and planned actions are completed.

[0154] In one implementation, the robotic arm control module 104 is further configured to:

[0155] Obtain expert path data; infer the corresponding reward function from the expert path data based on the inverse reinforcement learning algorithm; plan a path with the current coordinates as the starting point and the position data of the preset components as the ending point based on the reinforcement learning algorithm and the inferred reward function to obtain the planned path of the robotic arm.

[0156] In one embodiment, the robotic arm control module 104 is further configured to:

[0157] Obtain the operating environment data and construct a simulation space; construct a state space based on the operating environment data and the expert path data; set an action space, where the action space includes multiple joint axes of the robotic arm, and each joint axis has a forward rotation action and a reverse rotation action; construct a complete operating trajectory based on the state space and the action space; construct a reward function model; model the maximum entropy of the trajectory probability to obtain a maximum entropy model; and iteratively optimize the parameters of the reward function model by maximizing the log-likelihood of the expert trajectory to obtain the optimal reward function parameters.

[0158] The above-described embodiment of the present application provides a visual positioning and guiding device for a substation bionic operation robot, which has a corresponding relationship with the visual positioning and guiding method provided by the embodiment of the present application. Therefore, the specific content in this device has a corresponding relationship with the visual positioning and guiding method. The specific content can refer to the records in the visual positioning and guiding method, and will not be elaborated herein.

[0159] The above-described embodiment of the present application provides a visual positioning and guiding device for a substation bionic operation robot and the visual positioning and guiding method provided by the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0160] The internal functions and structures of the visual positioning and guiding device for a substation bionic operation robot are described above. As Figure 6 shown, in practice, the visual positioning and guiding device for a substation bionic operation robot can be implemented as an electronic device, including:

[0161] A memory 301 and a processor 303. The memory 301 can be configured to store programs. Additionally, the memory 301 can also be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application program or method for operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc. The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc. The processor 303 is coupled to the memory 301 and is configured to execute the programs in the memory 301 for:

[0162] Obtain substation image data and shooting position data; based on the substation image data and shooting position data, construct a 3D model of the substation and determine the position data of a preset component; the preset component is a button or knob that needs to be operated; obtain the current coordinates and current pose of the robotic arm in the same coordinate system of the substation 3D model; according to the current coordinates, current pose and the position data of the preset component, control the robotic arm to perform a preset operation.

[0163] In one implementation, the processor 303 is further configured to:

[0164] Input the substation image data into a pre-trained recognition model for recognition to determine the position information of the preset component; based on the substation image data and shooting position data, generate a 3D model of the substation and determine the position data of the preset component in the substation 3D model.

[0165] In one implementation, the processor 303 is further configured to:

[0166] Perform multiple downsampling processes on the substation image data to obtain downsampled feature maps with gradually decreasing sizes; perform multiple upsampling processes on the downsampled feature map with the smallest size to obtain upsampled feature maps with gradually increasing sizes; perform secondary downsampling and splicing processes on the upsampled feature map with the largest size to obtain a fused feature map with a gradually decreasing size; the sizes of the fused feature map, the downsampled feature maps, and the upsampled feature maps correspond one by one; input the fused feature maps of multiple sizes into the detection module and summarize them to obtain the types and position information of the preset components on the substation image.

[0167] In one implementation, the processor 303 is further configured to:

[0168] Perform double-layer convolution processing, class attention processing, downsampling processing, and class attention processing on the substation image data in sequence to obtain a first downsampled feature map; perform downsampling processing and class attention processing on the first downsampled feature map to obtain a second downsampled feature map; perform downsampling processing, class attention processing, and fusion processing on the second downsampled feature map to obtain a third downsampled feature map; the sizes of the first downsampled feature map, the second downsampled feature map, and the third downsampled feature map gradually decrease.

[0169] In one implementation, the processor 303 is further configured to:

[0170] According to the current coordinates, current pose of the robotic arm and the position data of the preset component, determine the planned path and planned actions of the robotic arm; control the robotic arm to execute the planned path and planned actions; obtain the real-time pose data of the end of the robotic arm; according to the deviation between the real-time pose data and the planned path and the planned actions, correct the robotic arm until the planned path and planned actions are completed.

[0171] In one embodiment, the processor 303 is further configured to:

[0172] Obtain expert path data; infer a corresponding reward function from the expert path data based on the inverse reinforcement learning algorithm; plan a path with the current coordinate as the starting point and the position data of a preset part as the ending point based on the reinforcement learning algorithm and the inferred reward function, to obtain the planned path of the robotic arm.

[0173] In one embodiment, the processor 303 is further configured to:

[0174] Obtain operating environment data and construct a simulation space; construct a state space based on the operating environment data and the expert path data; set an action space, where the action space includes multiple joint axes of the robotic arm, and each joint axis has a forward rotation action and a reverse rotation action; construct a complete operating trajectory based on the state space and the action space; construct a reward function model; model the maximum entropy of the trajectory probability to obtain a maximum entropy model; iteratively optimize the parameters of the reward function model by maximizing the log-likelihood of the expert trajectory to obtain optimal reward function parameters.

[0175] In this application, the processor is further specifically configured to execute all processes and steps of the above-mentioned visual positioning and guiding method for a substation bionic operation robot. For specific content, reference can be made to the records in the visual positioning and guiding method, which will not be elaborated herein.

[0176] In this application, Figure 6 only some components are schematically shown, and it does not mean that the electronic device only includes Figure 6 the components shown. The electronic device provided in this embodiment and the visual positioning and guiding method for a substation bionic operation robot provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run, or implemented by the application programs stored therein.

[0177] Those skilled in the art should understand that the embodiments of this application can be provided as a method, a system, or a computer program product. Therefore, this application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0180] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory. The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory (Flash RAM). The memory is an example of computer-readable media.

[0181] This application also provides a computer-readable storage medium corresponding to the method for visual positioning and guidance of a bionic operation robot for a substation provided in the foregoing embodiments. A computer program (i.e., a program product) is stored thereon. When the computer program is run by a processor, it will execute the method for visual positioning and guidance of a bionic operation robot for a substation provided in any of the foregoing embodiments.

[0182] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0183] The computer-readable storage medium provided by the above embodiments of the present application and the vision positioning and guiding method for a substation bionic operation robot provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run, or implemented by the application programs stored therein.

[0184] It should be noted that in the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this specification.

[0185] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, commodity or device comprising the said element.

[0186] The above are only the embodiments of the present application and are not used to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A visual positioning and guidance method based on a substation bionic operating robot, characterized in that: include: Obtain substation image data and shooting location data; Based on the substation image data and the shooting location data, construct a three-dimensional model of the substation and determine the location data of the preset parts; The preset part is a button or knob that needs to be operated; Obtain the current coordinates and current posture of the robotic arm in the same coordinate system of the substation 3D model; According to the current coordinates, current posture and position data of the preset parts, the robot arm is controlled to perform the preset operation; The controlling the robot arm to perform a preset operation according to the current coordinates, the current posture and the position data of the preset part includes: Determine the planned path and planned action of the robot arm according to the current coordinates, current posture and position data of the preset parts of the robot arm; Controlling the robotic arm to execute the planned path and planned action; Obtain real-time posture data of the end of the robotic arm; Correcting the robotic arm according to the real-time posture data and the deviation of the planned path and the planned action until the planned path and the planned action are completed; According to the current coordinates, current posture and position data of the robot arm, the planned path of the robot arm is determined, including: Get expert path data; Based on the inverse reinforcement learning algorithm, the corresponding reward function is inferred from the expert path data; Based on the inverse reinforcement learning algorithm and the inferred reward function, a path starting from the current coordinates and ending at the position data of the preset part is planned to obtain the planned path of the robot arm; The inverse reinforcement learning algorithm is used to infer the corresponding reward function from the expert path data, including: Obtain operating environment data and build a simulation space; Construct state space based on operating environment data and expert path data; Setting an action space, wherein the action space includes a plurality of joint axes of the robot arm, each joint axis having a forward rotation action and a reverse rotation action; Based on the state space and action space, a complete running trajectory is constructed; Construct a reward function model; Model the maximum entropy of trajectory probability to obtain the maximum entropy model; By maximizing the log-likelihood of the expert trajectory, the parameters of the reward function model are iteratively optimized to obtain the optimal reward function parameters; The step of constructing a three-dimensional model of a substation based on the substation image data and the shooting location data, and determining the location data of the preset parts, comprises: Inputting the substation image data into a pre-trained recognition model for recognition to determine the location information of the preset parts; Based on the substation image data and the shooting location data, a three-dimensional model of the substation is generated, and location data of preset parts in the three-dimensional model of the substation is determined; The step of inputting the substation image data into a pre-trained recognition model for recognition to determine the location information of the preset component includes: Performing multiple downsampling processes on the substation image data to obtain downsampled feature graphs with gradually smaller sizes; Perform multiple upsampling processes on the downsampled feature map with the smallest size to obtain upsampled feature maps with gradually increasing sizes; The up-sampled feature map with the largest size is subjected to secondary down-sampling and splicing processing to obtain a fused feature map with a gradually smaller size; the sizes of the fused feature map, the down-sampled feature map, and the up-sampled feature map correspond to each other; The fused feature maps of multiple sizes are input into the detection module respectively and summarized to obtain the type and location information of the preset parts on the substation image.

2. The visual positioning and guiding method based on the substation bionic operating robot according to claim 1 is characterized in that: The step of performing multiple downsampling processes on the substation image data to obtain downsampled feature graphs with gradually smaller sizes includes: The substation image data is sequentially subjected to double-layer convolution processing, class attention processing, downsampling processing and class attention processing to obtain a first downsampling feature map; Performing downsampling processing and attention-like processing on the first downsampled feature map to obtain a second downsampled feature map; The second down-sampled feature map is down-sampled, attention-like and fused to obtain a third down-sampled feature map; the sizes of the first down-sampled feature map, the second down-sampled feature map and the third down-sampled feature map gradually decrease.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement the visual positioning and guidance method based on the substation bionic operation robot as described in any one of claims 1-2.

Images