Visual positioning guiding method based on substation bionic operation robot

Abstract

The invention relates to the technical field of robot control, and particularly provides a substation-based bionic operation robot visual positioning guiding method, which comprises the following steps: acquiring substation image data and shooting position data; based on the transformer substation image data and the shooting position data, constructing a transformer substation three-dimensional model, and determining position data of a preset part; the preset piece is a button or a knob needing to be operated; the current coordinate and the current pose of the mechanical arm under the same coordinate system of the transformer substation three-dimensional model are obtained; and according to the current coordinate, the current pose and the position data of the preset part, the mechanical arm is controlled to execute preset operation. According to the method, the accurate position of the button or the knob is accurately recognized by obtaining the image data of the transformer substation, and the tail end path and action of the mechanical arm are planned, so that background operation is not needed, automatic movement and action of the mechanical arm are achieved, and automatic operation of the bionic operation robot is achieved.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a visual positioning and guidance method for a substation bionic operation robot. Background Art

[0002] The application of bionic operating robots in substations is one of the important development directions of modern power system automation. By combining bionic technology, robotics and automated control, bionic operating robots can perform various complex and dangerous operating tasks in substation environments, improving the efficiency, safety and accuracy of operations.

[0003] However, currently the control of bionic operating robots is all done through manual background remote control, which still requires a large amount of human resources. Summary of the invention

[0004] The problem that this application solves is that current bionic operating robots still require manual background remote control.

[0005] In order to solve the above problems, the first aspect of the present application provides a visual positioning and guidance method based on a substation bionic operation robot, comprising: Obtain substation image data and shooting location data; Based on the substation image data and the shooting location data, a three-dimensional model of the substation is constructed to determine the location data of a preset component; the preset component is a button or knob that needs to be operated; Obtain the current coordinates and current posture of the robotic arm in the same coordinate system of the substation 3D model; According to the current coordinates, current posture and position data of the preset parts, the robot arm is controlled to perform the preset operation.

[0006] The second aspect of the present application provides a visual positioning and guiding device based on a substation bionic operating robot, which includes: A data acquisition module, which is used to acquire substation image data and shooting location data; A model building module, which is used to build a three-dimensional model of the substation based on the substation image data and the shooting location data, and determine the location data of the preset parts; the preset parts are buttons or knobs that need to be operated; A posture acquisition module is used to obtain the current coordinates and current posture of the robot arm in the same coordinate system of the substation three-dimensional model; The robot arm control module is used to control the robot arm to perform preset operations according to the current coordinates, current posture and position data of the preset parts.

[0007] The third aspect of the present application provides an electronic device, comprising: a memory and a processor; the memory is used to store a program; the processor is coupled to the memory and is used to execute the program, so as to: Obtain substation image data and shooting location data; Based on the substation image data and the shooting location data, a three-dimensional model of the substation is constructed to determine the location data of a preset component; the preset component is a button or knob that needs to be operated; Obtain the current coordinates and current posture of the robotic arm in the same coordinate system of the substation 3D model; According to the current coordinates, current posture and position data of the preset parts, the robot arm is controlled to perform the preset operation.

[0008] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned visual positioning and guidance method based on a substation bionic operating robot.

[0009] In this application, the substation image data is obtained to accurately identify the exact position of the button or knob, and the end path and action of the robotic arm are planned, so that no background operation is required, the automatic movement and action of the robotic arm is achieved, and the automatic operation of the bionic operating robot is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flow chart of a visual positioning guidance method according to an embodiment of the present application; Figure 2 It is a structural diagram of a recognition model of a visual positioning guidance method according to an embodiment of the present application; Figure 3 It is an architecture diagram of the attention module of the visual positioning guidance method according to an embodiment of the present application; Figure 4 This is a flow chart of a robot arm control according to a visual positioning guidance method according to an embodiment of the present application; Figure 5 is a structural block diagram of a visual positioning guidance device according to an embodiment of the present application; Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0012] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0013] The embodiment of the present application provides a visual positioning and guidance method based on a substation bionic operation robot. The specific scheme of the method is as follows: Figure 1-Figure 4 As shown, the method can be performed by a visual positioning and guiding device based on a substation bionic operating robot, and the visual positioning and guiding device based on a substation bionic operating robot can be integrated into electronic devices such as computers, servers, computers, server clusters, and data centers. Figure 1 , Figure 2 As shown, it is a flow chart of a visual positioning and guiding method based on a substation bionic operating robot according to an embodiment of the present application; wherein the visual positioning and guiding method based on a substation bionic operating robot comprises: S101, obtaining substation image data and shooting location data; In the present application, the substation image data is image data acquired by a camera (such as an RGB camera, a depth camera, etc.), and the image data may be single image data directly captured or an image frame selected from a video stream.

[0014] Preferably, the substation image data is image frame data, and image frames corresponding to the same time step are selected from different cameras as the substation image data output by the camera.

[0015] In the present application, the shooting position data refers to the coordinate data and orientation data of the camera used for shooting.

[0016] Preferably, the coordinate data of cameras at different positions are coordinate data in the same coordinate system.

[0017] S102, constructing a three-dimensional model of the substation based on the substation image data and the shooting location data, and determining the location data of a preset component; the preset component is a button or knob that needs to be operated; S103, obtaining the current coordinates and current posture of the robot arm in the same coordinate system of the three-dimensional model of the substation; In the present application, the current coordinates of the robotic arm in the same coordinate system of the substation three-dimensional model refer to the current coordinates of the robotic arm when the robot has reached the operable area.

[0018] It should be noted that when the robot performs specific operations, it can be divided into three stages: the robot moves from a distance to the operable area of ​​the preset part; when the robot is in the operable area, the end of the robotic arm moves from the initial posture along the planned path to the preparation posture that fits the preset part (this stage corresponds to the planned path); the end of the robotic arm starts from the preparation posture and performs operations such as screwing and pressing on the preset part (this stage corresponds to the planned action).

[0019] In the present application, the operable area of ​​a preset component refers to the area in which the robot can operate the preset component through the robotic arm when the robot is in the area.

[0020] In the present application, the path setting and control of the robot in the stage of traveling from a distance to the operable area of ​​the preset part can be achieved by generating a three-dimensional model of the substation through path planning algorithms such as RRT and PRM, so as to find the optimal path in a complex substation environment, avoid obstacles and reach the target position.

[0021] S104, controlling the robot arm to perform a preset operation according to the current coordinates, the current posture and the position data of the preset part.

[0022] In this application, the substation image data is obtained to accurately identify the exact position of the button or knob, and the end path and action of the robotic arm are planned, so that no background operation is required, the automatic movement and action of the robotic arm is achieved, and the automatic operation of the bionic operating robot is realized.

[0023] In one embodiment, the step S102, constructing a three-dimensional model of the substation based on the substation image data and the shooting location data, and determining the location data of the preset components, includes: Inputting the substation image data into a pre-trained recognition model for recognition to determine the location information of the preset parts; Based on the substation image data and the shooting position data, a three-dimensional model of the substation is generated, and the position data of the preset parts in the three-dimensional model of the substation is determined.

[0024] In the present application, the substation image data may be a depth image or a non-depth two-dimensional image (no depth data is contained on the pixel points). It should be noted that the training samples of the recognition model and the substation image data have the same format.

[0025] In the present application, a three-dimensional model of a substation is generated based on the substation image data and the shooting location data. The substation image data is a depth image, and the viewing angles of multiple depth cameras can cover the planned area of ​​the substation.

[0026] In the present application, after acquiring depth images of multiple depth cameras, intrinsic parameters and relative positional relationships of the cameras can be determined by performing internal and external calibration on each depth camera; the coordinates of each pixel in three-dimensional space can be calculated for each depth image through the intrinsic parameters (focal length, optical center) and depth value of the camera; the depth images acquired from multiple different perspectives, or the point clouds corresponding to the depth images, are aligned to the same coordinate system; the point clouds under the same coordinate system are fused to eliminate redundant data and generate a complete and unified three-dimensional model, which is the three-dimensional model of the substation.

[0027] Preferably, the substation three-dimensional model may be a local model of the substation, and the robot runs in the local model, thereby greatly reducing the depth camera data and calculation amount required for the substation three-dimensional model.

[0028] In the present application, the position information of the preset component in the three-dimensional model can be determined by knowing the position information of the preset component in the depth image and the three-dimensional model constructed by the depth image.

[0029] In one embodiment, in combination Figure 2 As shown, the substation image data is input into a pre-trained recognition model for recognition to determine the location information of the preset component, including: Performing multiple downsampling processes on the substation image data to obtain downsampled feature graphs with gradually smaller sizes; Perform multiple upsampling processes on the downsampled feature map with the smallest size to obtain upsampled feature maps with gradually increasing sizes; The up-sampled feature map with the largest size is subjected to secondary down-sampling and splicing processing to obtain a fused feature map with a gradually smaller size; the sizes of the fused feature map, the down-sampled feature map, and the up-sampled feature map correspond to each other; The fused feature maps of multiple sizes are input into the detection module respectively and summarized to obtain the type and location information of the preset parts on the substation image.

[0030] like Figure 2 As shown, in the recognition model, the input substation image is downsampled to obtain downsampled feature maps of three sizes (first size, second size, third size; the sizes decrease successively); by upsampling the downsampled feature map of the third size, three upsampled feature maps of sizes are obtained (third size, second size, first size; the sizes increase gradually); and then by downsampling the upsampled feature map of the first size again, three fused feature maps of sizes are obtained (first size, second size, third size; the sizes decrease successively).

[0031] It should be noted that the feature map at the junction of the downsampled feature map, the upsampled feature map, and the fused feature map is the same feature map: the downsampled feature map of the third size and the upsampled feature map of the third size are the same feature map; the upsampled feature map of the first size and the fused feature map of the first size are the same feature map.

[0032] In the present application, compared with the traditional downsampling-upsampling method, the downsampling-upsampling method can further compress information and remove redundant features while retaining sufficient key information; this operation can extract and retain multi-level features in images or signals at different scales; in addition, by detecting and summarizing feature maps of different sizes, the preset parts can be detected at different levels, greatly increasing the accuracy of detection.

[0033] In the present application, the detection results of feature maps of different sizes include position information, and there is a linear correspondence between the position information and the size of the feature map; after the size of the feature map is proportionally enlarged, the corresponding detection position information is also proportionally enlarged, thereby converting the detection results of feature maps of different sizes into the same size, and then corresponding to the input substation image to obtain the detection results marked on the substation image.

[0034] In one embodiment, in combination Figure 2 As shown, the substation image data is subjected to multiple downsampling processes to obtain downsampled feature graphs with gradually smaller sizes, including: The substation image data is sequentially subjected to double-layer convolution processing, class attention processing, downsampling processing and class attention processing to obtain a first downsampling feature map; Performing downsampling processing and attention-like processing on the first downsampled feature map to obtain a second downsampled feature map; The second down-sampled feature map is down-sampled, attention-like and fused to obtain a third down-sampled feature map; the sizes of the first down-sampled feature map, the second down-sampled feature map and the third down-sampled feature map gradually decrease.

[0035] Among them, the first down-sampling feature map, the second down-sampling feature map, and the third down-sampling feature map are down-sampling feature maps of the first size, the second size, and the third size.

[0036] In this application, double-layer convolution refers to two consecutive convolution operations; through continuous convolution, the perception can be significantly increased without significantly increasing the size of the convolution kernel.

[0037] In the present application, the fusion processing may be a simple fusion processing or a global extraction processing.

[0038] Preferably, the fusion processing is global processing, so that better global feature enhancement can be achieved through global processing.

[0039] The process of fusion processing / global extraction processing is as follows: Input the input feature map into the 1×1 convolutional layers set in parallel to obtain the first extraction map, the second extraction map and the third extraction map respectively; wherein the first extraction map and the third extraction map are in HW×N format, and the second extraction map is in N×HW format; Multiply the first extracted image and the second extracted image to obtain a multiplied feature image; Perform softmax processing on the multiplied feature map to obtain the multiplication coefficient; Multiplying the multiplication coefficient with the third extraction map to obtain a coefficient multiplication map; Add the coefficient multiplication map and the input feature map to obtain the output feature map of the global extraction / fusion processing.

[0040] In the present application, the input feature map is divided into three branches for 1×1 convolution processing. The dimensions of the first branch and the third branch are adjusted to HW×N format, and the dimension of the second branch is N×HW format; then the first branch is multiplied by the third branch to obtain a feature map with a dimension of HW×HW, and a coefficient is obtained through softmax processing; the coefficient is multiplied with the third branch, and the multiplied feature map is added to the input feature map to obtain an output feature map.

[0041] In this application, the dimensions of the first branch and the second branch are reversed through 1×1 convolution, thereby realizing multiplication between feature maps; through this multiplication, long-range dependencies are captured; by setting coefficients, the dependencies are embedded in the feature maps; and on this basis, the original input is added and a residual connection is used to embed the global extraction into the model without destroying the parameters.

[0042] In this application, long-range dependencies between features are captured through global extraction, so that the model can extract global features, making up for the defect that the current model mainly extracts local features.

[0043] In the present application, the response at a position is calculated as the weighted sum of the features at all positions, so that when processing the information at each position, the processing is performed on the basis of considering all the position information that can be considered.

[0044] In one embodiment, in combination Figure 2 As shown, the downsampled feature map with the smallest size is upsampled multiple times to obtain upsampled feature maps with gradually increasing sizes, including: The third down-sampled feature map (the third up-sampled feature map) is up-sampled, and after up-sampling, it is concatenated with the second down-sampled feature map, and attention processing is performed to obtain a second up-sampled feature map; The second upsampled feature map is upsampled, and after upsampling, it is concatenated with the first downsampled feature map, and attention processing is performed to obtain the first upsampled feature map.

[0045] In one embodiment, in combination Figure 2 As shown in the figure, the upsampled feature map with the largest size is downsampled and concatenated twice to obtain a fused feature map with a gradually smaller size, including: The first upsampled feature map (first fused feature map) is downsampled, and after downsampling, it is concatenated with the second upsampled feature map, and attention processing is performed to obtain a second fused feature map; The second fused feature map is downsampled, and after downsampling, it is concatenated with the third upsampled feature map / the third downsampled feature map, and attention processing is performed to obtain a third fused feature map.

[0046] Combination Figure 2 As shown, it can be seen that in the upsampling feature map, the upsampled feature map is spliced ​​with the downsampled feature map of the corresponding size, so as to retain the detail information in the downsampling process; and in the secondary downsampling process, the downsampled feature map is spliced ​​with the upsampled feature map of the corresponding size, so as to retain the detail information in the upsampling process; through this layer-by-layer long connection processing, the loss of important detail information is avoided and the problem of gradient disappearance is alleviated.

[0047] In one embodiment, in combination Figure 3 As shown, the process of attention extraction is: The input feature map is convolved, and the convolved feature map is divided into three branches: in the first branch, the feature map is sequentially convolved, fully connected, ReLU processed, and fully connected to obtain the first branch feature map; in the second branch, the feature map is sequentially vertically converted (converting the H×W×C feature map to the H×1×C feature map) and expanded (expanding the H×1×C feature map to the H×W×C feature map) to obtain the second branch feature map; in the third branch, the feature map is sequentially convolved, ReLU, convolution, ReLU processed (the subsequent arrows in this part do not indicate the specific processing process, but directly display the size information of H×W×C after ReLU processing) to obtain the third branch feature map; after multiplying the first branch feature map and the second branch feature map (similarly, the arrow after this part does not indicate the specific processing process, but directly displays the size information of H×W×C after processing), it is fused with the third branch feature map (this fusion can be ADD operation or other fusion operation, which is determined according to the actual situation) to obtain an output feature map of size H×W×C.

[0048] In this application, the sizes of the input feature map and the output feature map are both H×W×C.

[0049] In the present application, the feature map is divided into three branches for processing, and vertical feature processing is performed through vertical transformation, so that features can be extracted from different directions, and local weights are increased to improve the attention enhancement characteristics of the features.

[0050] In this application, the H×W×C feature map can be converted into a H×1×C feature map by averaging the values ​​in the width direction; this part is used as the processing of the vertical direction features and is merged into other branches by multiplication.

[0051] In the present application, the H×1×C feature map can be expanded to a H×W×C feature map by multi-layer replication, and the values ​​in the width direction of the expanded H×W×C feature map are the same.

[0052] In the present application, the detection module may be configured as a simple CNN structure or other improved structures, so that the detection position / detection frame and the detection type may be outputted synchronously.

[0053] In one embodiment, in combination Figure 4 As shown, the S104, according to the current coordinates and the current posture and the position data of the preset part, controls the robot arm to perform the preset operation, including: S401, determining a planned path and a planned action of the robot arm according to the current coordinates, current posture and position data of the preset part of the robot arm; S402, controlling the robot arm to execute the planned path and planned action; S403, obtaining real-time position and posture data of the end of the robotic arm; In this application, S404, correcting the robotic arm according to the real-time posture data and the deviation of the planned path and the planned action until the planned path and the planned action are completed.

[0054] In the present application, the movement of the robot arm is corrected in real time, thereby preventing the movement of the robot arm from deviating from the planned path.

[0055] In the present application, it should be noted that, during the control process of the robotic arm, if the planning in the planned path or the planned action fails to take into account a variety of actual control factors (the planned path and the planned action are not designed to be smooth enough), control errors of the robotic arm may occur, causing the movement of the robotic arm to deviate from the planned path.

[0056] In one embodiment, determining the planned path of the robotic arm according to the current coordinates, current posture and position data of the preset part of the robotic arm includes: Get expert path data; Based on the inverse reinforcement learning algorithm, the corresponding reward function is inferred from the expert path data; Based on the reinforcement learning algorithm and the inferred reward function, a path with the current coordinates as the starting point and the position data of the preset part as the end point is planned to obtain the planned path of the robotic arm.

[0057] In the present application, the expert path data may be drawn by a robot expert, or may be path data with the smallest deviation of the robot arm selected from actual operation.

[0058] In this application, the expert path data is summarized through inverse reinforcement learning, the corresponding reward function is summarized, and the new path data is trained using the reward function to obtain a planned path that summarizes the expert experience, thereby ensuring that the robotic arm can move along the planned path to the greatest extent possible, greatly improving the accuracy of the robotic arm's path movement process.

[0059] In one implementation, the robotic arm is a multi-axis robotic arm, and the expert data is angle variation data of multiple motors corresponding to the multi-axis robotic arm over time.

[0060] In one embodiment, the inferring a corresponding reward function from expert path data based on an inverse reinforcement learning algorithm includes: Obtain operating environment data and build a simulation space; Construct state space based on operating environment data and expert path data; Setting an action space, wherein the action space includes a plurality of joint axes of the robot arm, each joint axis having a forward rotation action and a reverse rotation action; Based on the state space and action space, a complete running trajectory is constructed; Construct a reward function model; Model the maximum entropy of trajectory probability to obtain the maximum entropy model; By maximizing the log-likelihood of the expert trajectory, the parameters of the reward function model are iteratively optimized to obtain the optimal reward function parameters.

[0061] In the present application, the motion environment data may be three-dimensional model data of a substation. The simulation space may be a virtual three-dimensional space.

[0062] In the present application, the state space may include the operating environment state determined by the operating environment data and the angle state of each joint of the robot arm determined by the expert path data, which may be specifically represented as a corresponding vector.

[0063] In this application, in the action space, the robot action represents the change of the joint angle of the robot at each time step (specifically, the change of the joint angle of the robot at each time point after the split), which can also be used as a vector. The running trajectory is composed of a series of states and action sequences.

[0064] In this application, the calculation of maximum entropy involves the sum of all possible trajectories. Therefore, the optimization is performed by maximizing the log-likelihood of the expert trajectory.

[0065] Among them, the log-likelihood of the expert trajectory is determined, and it is optimized towards the maximum value by gradient descent or gradient ascent optimization until the optimization is completed.

[0066] In this application, the specific optimization process of gradient descent or ascent will not be described in detail.

[0067] In this application, the expert trajectory is the constructed complete operation trajectory.

[0068] In this application, the reward function model and the maximum entropy model can be determined according to actual conditions and will not be described in detail in this application.

[0069] In the present application, based on the reinforcement learning algorithm and the inferred reward function, a path with the current coordinates as the starting point and the position data of the preset part as the end point is planned, and the reward function obtained by the above operation is used as the reward function in the planned path of the robotic arm. Through reinforcement learning, a new planning path strategy is trained, and then based on the planning path strategy, a path with the current coordinates as the starting point and the position data of the preset part as the end point is planned.

[0070] In this application, training can be performed by deep Q network learning using a known reward function. The specific training process can be: Build simulation environments and state spaces; Set up the action space; Obtain a known reward function, build a deep Q network model and an experience replay pool, which includes the agent's actions, states, rewards, and next states; Sampling from the experience replay pool and calculating the Q value based on the deep Q network model; By minimizing the loss function, the weights of the deep Q network model and the weights of the target network are updated until the training is completed. The deep Q network model is consistent with the target network.

[0071] The details of deep Q network learning will not be repeated in this application.

[0072] It should be noted that in the training process of deep Q network learning, its simulation space, state space, and action space are consistent with those in the inverse reinforcement learning process.

[0073] In one embodiment, before inferring the corresponding reward function from the expert path data based on the inverse reinforcement learning algorithm, the method further includes: Preprocess the expert data and convert it into a preset format, specifically: The expert data is segmented based on the time step, and based on the segmented expert data, the corresponding angle value of the motor of each joint of the robot arm at each time step is determined; the time step is divided twice, and each time step is divided into a number of time points (the number is consistent with the number of axes of the robot arm), and the motor actions from the first joint to the end joint are executed respectively; In this way, the action execution of the robot arm at a certain time step is converted into the action execution of the first joint axis, the second joint axis, ..., and the end joint axis in sequence at multiple time points, thereby greatly reducing the number of actions of the entire robot arm at the same time step. In other words, at the first time point, the motor action of the first joint is executed (the motor action is the action of the corresponding time step), but the motors of other joints remain unchanged. In this way, the combination of action space is greatly reduced.

[0074] Preferably, a reward function is determined based on the preprocessed inverse reinforcement learning method. After a new action sequence is obtained by training through a new reinforcement learning method based on the reward function, it is also necessary to synthesize the actions corresponding to six consecutive time points into the action corresponding to one time step, so as to connect the actions of the robotic arm and obtain the corresponding planned path.

[0075] In the present application, similarly, the process of determining the planned action may be similar to the planned path, or experts may directly design the planned actions of different preset parts to achieve accurate operation of the preset parts.

[0076] An embodiment of the present application provides a visual positioning and guiding device based on a substation bionic operating robot, which is used to execute the visual positioning and guiding method based on a substation bionic operating robot described in the above content of the present application. The visual positioning and guiding device based on a substation bionic operating robot is described in detail below.

[0077] like Figure 5 As shown, the visual positioning and guiding device based on the substation bionic operation robot includes: A data acquisition module 101 is used to acquire substation image data and shooting location data; A model building module 102, which is used to build a three-dimensional model of the substation based on the substation image data and the shooting location data, and determine the location data of the preset parts; the preset parts are buttons or knobs that need to be operated; A posture acquisition module 103, which is used to obtain the current coordinates and current posture of the robot arm in the same coordinate system of the substation three-dimensional model; The robot arm control module 104 is used to control the robot arm to perform a preset operation according to the current coordinates and the current posture as well as the position data of the preset part.

[0078] In one embodiment, the model building module 102 is further configured to: The substation image data is input into a pre-trained recognition model for recognition to determine the location information of the preset parts; based on the substation image data and the shooting location data, a substation three-dimensional model is generated, and the location data of the preset parts in the substation three-dimensional model is determined.

[0079] In one embodiment, the model building module 102 is further configured to: The substation image data is downsampled multiple times to obtain downsampled feature maps with gradually decreasing sizes; the downsampled feature map with the smallest size is upsampled multiple times to obtain upsampled feature maps with gradually increasing sizes; the upsampled feature map with the largest size is downsampled twice and spliced ​​to obtain a fused feature map with gradually decreasing sizes; the sizes of the fused feature map, the downsampled feature map, and the upsampled feature map correspond to each other one by one; the fused feature maps of multiple sizes are respectively input into the detection module and summarized to obtain the type and location information of the preset parts on the substation image.

[0080] In one embodiment, the model building module 102 is further configured to: The substation image data is sequentially subjected to double-layer convolution processing, class attention processing, downsampling processing and class attention processing to obtain a first downsampling feature map; the first downsampling feature map is subjected to downsampling processing and class attention processing to obtain a second downsampling feature map; the second downsampling feature map is subjected to downsampling processing, class attention processing and fusion processing to obtain a third downsampling feature map; the sizes of the first downsampling feature map, the second downsampling feature map and the third downsampling feature map gradually decrease.

[0081] In one embodiment, the robot control module 104 is further configured to: Determine the planned path and planned action of the robotic arm according to the current coordinates, current posture and position data of the preset parts of the robotic arm; control the robotic arm to execute the planned path and planned action; obtain the real-time posture data of the end of the robotic arm; correct the robotic arm according to the deviation between the real-time posture data and the planned path and the planned action until the planned path and planned action are completed.

[0082] In one embodiment, the robot control module 104 is further configured to: Obtain expert path data; based on the inverse reinforcement learning algorithm, infer the corresponding reward function from the expert path data; based on the reinforcement learning algorithm and the inferred reward function, plan a path with the current coordinates as the starting point and the position data of the preset part as the end point to obtain the planned path of the robotic arm.

[0083] In one embodiment, the robot control module 104 is further configured to: Acquire operating environment data and construct a simulation space; construct a state space based on the operating environment data and expert path data; set an action space, wherein the action space includes multiple joint axes of the robot arm, each joint axis has a forward rotation action and a reverse rotation action; construct a complete operating trajectory based on the state space and the action space; construct a reward function model; model the maximum entropy of the trajectory probability to obtain a maximum entropy model; iteratively optimize the parameters of the reward function model by maximizing the log-likelihood of the expert trajectory to obtain the optimal reward function parameters.

[0084] The visual positioning and guidance device based on the substation bionic operating robot provided in the above-mentioned embodiment of the present application has a corresponding relationship with the visual positioning and guidance method based on the substation bionic operating robot provided in the embodiment of the present application. Therefore, the specific content in the device has a corresponding relationship with the visual positioning and guidance method. The specific content can be referred to the record in the visual positioning and guidance method, which will not be repeated in this application.

[0085] The visual positioning and guidance device based on the substation bionic operating robot provided in the above-mentioned embodiment of the present application and the visual positioning and guidance method based on the substation bionic operating robot provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0086] The above describes the internal functions and structure of the visual positioning and guidance device based on the substation bionic operation robot. Figure 6 As shown, in practice, the visual positioning and guiding device based on the substation bionic operation robot can be realized as an electronic device, including: Memory 301 and processor 303. Memory 301 can be configured to store programs. In addition, memory 301 can also be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc. Memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. Processor 303, coupled to memory 301, is used to execute the program in memory 301 for: Obtain substation image data and shooting position data; construct a three-dimensional model of the substation based on the substation image data and shooting position data, and determine the position data of preset parts; the preset parts are buttons or knobs that need to be operated; obtain the current coordinates and current posture of the robotic arm in the same coordinate system of the substation three-dimensional model; control the robotic arm to perform preset operations according to the current coordinates and current posture and the position data of the preset parts.

[0087] In one implementation, the processor 303 is further configured to: The substation image data is input into a pre-trained recognition model for recognition to determine the location information of the preset parts; based on the substation image data and the shooting location data, a substation three-dimensional model is generated, and the location data of the preset parts in the substation three-dimensional model is determined.

[0088] In one implementation, the processor 303 is further configured to: The substation image data is downsampled multiple times to obtain downsampled feature maps with gradually decreasing sizes; the downsampled feature map with the smallest size is upsampled multiple times to obtain upsampled feature maps with gradually increasing sizes; the upsampled feature map with the largest size is downsampled twice and spliced ​​to obtain a fused feature map with gradually decreasing sizes; the sizes of the fused feature map, the downsampled feature map, and the upsampled feature map correspond to each other one by one; the fused feature maps of multiple sizes are respectively input into the detection module and summarized to obtain the type and location information of the preset parts on the substation image.

[0089] In one implementation, the processor 303 is further configured to: The substation image data is sequentially subjected to double-layer convolution processing, class attention processing, downsampling processing and class attention processing to obtain a first downsampling feature map; the first downsampling feature map is subjected to downsampling processing and class attention processing to obtain a second downsampling feature map; the second downsampling feature map is subjected to downsampling processing, class attention processing and fusion processing to obtain a third downsampling feature map; the sizes of the first downsampling feature map, the second downsampling feature map and the third downsampling feature map gradually decrease.

[0090] In one implementation, the processor 303 is further configured to: Determine the planned path and planned action of the robotic arm according to the current coordinates, current posture and position data of the preset parts of the robotic arm; control the robotic arm to execute the planned path and planned action; obtain the real-time posture data of the end of the robotic arm; correct the robotic arm according to the deviation between the real-time posture data and the planned path and the planned action until the planned path and planned action are completed.

[0091] In one implementation, the processor 303 is further configured to: Obtain expert path data; based on the inverse reinforcement learning algorithm, infer the corresponding reward function from the expert path data; based on the reinforcement learning algorithm and the inferred reward function, plan a path with the current coordinates as the starting point and the position data of the preset part as the end point to obtain the planned path of the robotic arm.

[0092] In one implementation, the processor 303 is further configured to: Acquire operating environment data and construct a simulation space; construct a state space based on the operating environment data and expert path data; set an action space, wherein the action space includes multiple joint axes of the robot arm, each joint axis has a forward rotation action and a reverse rotation action; construct a complete operating trajectory based on the state space and the action space; construct a reward function model; model the maximum entropy of the trajectory probability to obtain a maximum entropy model; iteratively optimize the parameters of the reward function model by maximizing the log-likelihood of the expert trajectory to obtain the optimal reward function parameters.

[0093] In the present application, the processor is also specifically used to execute all processes and steps of the above-mentioned visual positioning and guidance method based on the substation bionic operation robot. The specific content can refer to the records in the visual positioning and guidance method, which is not described in this application.

[0094] In this application, Figure 6 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 6 The electronic device provided in this embodiment is based on the same inventive concept as the visual positioning and guiding method based on the substation bionic operation robot provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0095] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0098] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). Memory is an example of a computer-readable medium.

[0099] The present application also provides a computer-readable storage medium corresponding to the visual positioning and guidance method based on the substation bionic operating robot provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the visual positioning and guidance method based on the substation bionic operating robot provided in any of the aforementioned embodiments.

[0100] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0101] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the visual positioning and guidance method based on the substation bionic operation robot provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0102] It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0103] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0104] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A visual positioning and guidance method based on a substation bionic operating robot, characterized in that: include: Obtain substation image data and shooting location data; Based on the substation image data and the shooting location data, construct a three-dimensional model of the substation and determine the location data of the preset parts; The preset part is a button or knob that needs to be operated; Obtain the current coordinates and current posture of the robotic arm in the same coordinate system of the substation 3D model; According to the current coordinates, current posture and position data of the preset parts, the robot arm is controlled to perform the preset operation; The controlling the robot arm to perform a preset operation according to the current coordinates, the current posture and the position data of the preset part includes: Determine the planned path and planned action of the robot arm according to the current coordinates, current posture and position data of the preset parts of the robot arm; Controlling the robotic arm to execute the planned path and planned action; Obtain real-time posture data of the end of the robotic arm; Correcting the robotic arm according to the real-time posture data and the deviation of the planned path and the planned action until the planned path and the planned action are completed; According to the current coordinates, current posture and position data of the robot arm, the planned path of the robot arm is determined, including: Get expert path data; Based on the inverse reinforcement learning algorithm, the corresponding reward function is inferred from the expert path data; Based on the reinforcement learning algorithm and the inferred reward function, a path starting from the current coordinates and ending at the position data of the preset part is planned to obtain a planned path of the robot arm; the inferred reward function from the expert path data based on the inverse reinforcement learning algorithm includes: Obtain operating environment data and build a simulation space; Construct state space based on operating environment data and expert path data; Setting an action space, wherein the action space includes a plurality of joint axes of the robot arm, each joint axis having a forward rotation action and a reverse rotation action; Based on the state space and action space, a complete running trajectory is constructed; Construct a reward function model; Model the maximum entropy of trajectory probability to obtain the maximum entropy model; By maximizing the log-likelihood of the expert trajectory, the parameters of the reward function model are iteratively optimized to obtain the optimal reward function parameters.

2. The visual positioning and guiding method based on the substation bionic operating robot according to claim 1 is characterized in that: The step of constructing a three-dimensional model of a substation based on the substation image data and the shooting location data, and determining the location data of the preset parts, comprises: Inputting the substation image data into a pre-trained recognition model for recognition to determine the location information of the preset parts; Based on the substation image data and the shooting position data, a three-dimensional model of the substation is generated, and the position data of the preset parts in the three-dimensional model of the substation is determined.

3. The visual positioning and guiding method based on the substation bionic operating robot according to claim 2 is characterized in that: The step of inputting the substation image data into a pre-trained recognition model for recognition to determine the location information of the preset component includes: Performing multiple downsampling processes on the substation image data to obtain downsampled feature graphs with gradually smaller sizes; Perform multiple upsampling processes on the downsampled feature map with the smallest size to obtain upsampled feature maps with gradually increasing sizes; The up-sampled feature map with the largest size is subjected to secondary down-sampling and splicing processing to obtain a fused feature map with a gradually smaller size; the sizes of the fused feature map, the down-sampled feature map, and the up-sampled feature map correspond to each other; The fused feature maps of multiple sizes are input into the detection module respectively and summarized to obtain the type and location information of the preset parts on the substation image.

4. The visual positioning and guiding method based on the substation bionic operating robot according to claim 3 is characterized in that: The step of performing multiple downsampling processes on the substation image data to obtain downsampled feature graphs with gradually smaller sizes includes: The substation image data is sequentially subjected to double-layer convolution processing, class attention processing, downsampling processing and class attention processing to obtain a first downsampling feature map; Performing downsampling processing and attention-like processing on the first downsampled feature map to obtain a second downsampled feature map; The second down-sampled feature map is down-sampled, attention-like and fused to obtain a third down-sampled feature map; the sizes of the first down-sampled feature map, the second down-sampled feature map and the third down-sampled feature map gradually decrease.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement the visual positioning and guidance method based on the substation bionic operation robot as described in any one of claims 1-4.

Images