Visual identification-based electrified maintenance mechanical device pose identification method

Abstract

The invention provides a visual identification-based live-line maintenance mechanical device pose identification method, belongs to the technical field of power equipment maintenance, and aims to solve the problems that live-line maintenance faces a complex environment, electrical equipment layout is complex, multiple obstacles exist, a maintenance target and a device pose are difficult to accurately position in a traditional mode, real-time adjustment cannot be performed during movement, and the maintenance time is short. And the precision is insufficient. According to the method, equipment such as an industrial camera is installed on a live maintenance mechanical device, environment and maintenance target visual feature data are collected for model training, various data are collected and preprocessed during maintenance, a maintenance target is recognized and positioned through model and sensor data, the pose is accurately calculated, and multi-sensor data optimization is fused. A path is planned and motion is guided according to the pose of a maintenance target and the live maintenance mechanical device, continuous monitoring is performed during motion, adjustment or re-planning is performed according to pose errors, the pose is accurately identified, dangers are effectively avoided, motion is corrected in real time, and the safety, accuracy and efficiency of maintenance are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power equipment maintenance. Specifically, it is a method for identifying the pose of a live maintenance mechanical device based on visual recognition. Background Art

[0002] In the field of power system maintenance, live maintenance operations are crucial for ensuring the continuity of power supply. With the continuous expansion of the power network scale and increasing complexity, electrical equipment requires more frequent maintenance and repair. The live maintenance technology has become a key requirement for the industry's development. The live maintenance mechanical device emerged under such circumstances. It can detect, repair and other operations on electrical equipment without power outage, avoiding the great inconvenience caused by power outage to production and life.

[0003] For example, the patent with the publication number CN108714897A discloses an insulating arm pose control system and method for a live maintenance operation robot in a substation, which collects and controls the relative positions, angles of the robot body mobile chassis and the insulating telescopic arm, as well as the distance between the operation tool and the equipment to be operated. Combining the operation state data, it adjusts the poses of the robot body mobile chassis and the insulating telescopic arm. It can quickly record the robot state by fusing the data of each sensor, effectively ensuring the safety of the robot operation. By dividing the priority levels of the sensors, it can effectively prevent the problem of system misjudgment due to the same priority level. However, there are many defects in the existing technologies for determining the pose of live maintenance mechanical devices. On the one hand, in a complex live environment, the layout around electrical equipment is messy, with a large number of potential obstacles, making it difficult for traditional positioning methods to accurately obtain the position and pose information of the maintenance target, resulting in difficult accurate implementation of maintenance operations. On the other hand, there is a lack of effective real-time monitoring and adjustment means. The device cannot adapt to environmental changes in a timely manner during movement, easily deviates, which not only affects the maintenance efficiency, but also increases the operation risk, and may even cause safety accidents, endangering the safety of personnel and the stable operation of the power system.

[0004] Therefore, a method for identifying the pose of a live maintenance mechanical device based on visual recognition is introduced. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for identifying the pose of a live maintenance mechanical device based on visual recognition, aiming to solve the problems in the above background art that live maintenance faces a complex environment, the layout of electrical equipment is complex and there are many obstacles, it is difficult to accurately locate the maintenance target and the device pose by traditional methods, and it cannot be adjusted in real time during movement and has insufficient accuracy.

[0006] To achieve the above purpose, the present invention provides the following technical solution: A method for identifying the pose of a live maintenance mechanical device based on visual recognition, including the following specific implementation steps:

[0007] S1: Equipment preparation and environmental information acquisition. Install a certain number of industrial cameras and auxiliary positioning sensors on the live maintenance mechanical device, and collect and preprocess the maintenance environment data;

[0008] S2: Target feature extraction and model training. Collect image samples of the electrical equipment to be maintained, extract key features, train the maintenance target recognition model and establish a feature database;

[0009] S3: Real-time visual data collection and preprocessing. Collect and preprocess visual, distance, and pose data during maintenance;

[0010] S4: Target recognition and preliminary positioning. Identify the target and perform preliminary positioning through the model and sensor data, and divide the region of interest to screen the target;

[0011] S5: Precise pose determination. Precisely determine the target and optimize by fusing multi-sensor data;

[0012] S6: Path planning and motion guidance. Plan the path according to the target and the pose of the live maintenance mechanical device and guide the live maintenance mechanical device to move to the maintenance position, and continuously monitor during the movement;

[0013] S7: Pose feedback and correction. During the movement of the live maintenance mechanical device, update the maintenance target and the pose of the live maintenance mechanical device in real time, calculate the error, adjust the movement according to the error or re-plan the path, and perform emergency braking for re-positioning until the live maintenance mechanical device completes the maintenance of the electrical equipment.

[0014] Further, in S1, for equipment preparation and environmental information acquisition, specifically:

[0015] Install and configure the vision system. Install a number of industrial cameras on the live maintenance mechanical device. These industrial cameras cover the area where the maintenance target appears from different angles. At the same time, install auxiliary positioning sensors on the live maintenance mechanical device;

[0016] Collect environmental data. Before the maintenance operation, use mobile scanning equipment including drones to scan the entire maintenance environment, obtain the three-dimensional point cloud data and clear images of the environment, and transmit information including the layout of electrical equipment near the maintenance target, the position and shape of potential obstacles to the control system of the live maintenance mechanical device for storage and preprocessing.

[0017] Further, the industrial camera is responsible for collecting image data of the maintenance target and its surrounding environment, and the auxiliary positioning sensor specifically includes:

[0018] Camera parameter adjustment device, used to adjust parameters including focal length, aperture, and time to promote the industrial camera to collect visual image data for different maintenance environments and target machines;

[0019] A laser rangefinder for real-time measuring the distance information between a live maintenance mechanical device and a maintenance target and the surrounding objects, providing depth data for the positioning of the maintenance target;

[0020] An inertial measurement unit for recording the attitude changes of the live maintenance mechanical device and assisting in determining the pose state of the live maintenance mechanical device itself.

[0021] Further, in the step S2, for the target feature extraction and model training, specifically:

[0022] Target feature analysis: For the electrical equipment or components to be maintained, a sufficient amount of image samples are collected, and these images are analyzed using computer vision algorithms to extract the key feature information of the maintenance target, including shape features, texture features, color features, and local features;

[0023] Model training: Using a deep learning framework, a maintenance target recognition model based on a convolutional neural network is constructed. The key features of the maintenance target extracted are used as training data to input into the model to train the maintenance target recognition model. At the same time, a maintenance target feature database is established to store various feature data of the maintenance target and its corresponding three-dimensional spatial position information.

[0024] Further, in the step S4, for the target recognition and preliminary positioning, specifically:

[0025] Target recognition: The preprocessed image data collected for the maintenance target and its surrounding environment is input into the trained maintenance target recognition model. The maintenance target recognition model outputs whether there is a maintenance target in the image and the position information of the maintenance target in the image. Combining the data of the laser rangefinder and the inertial measurement unit, a preliminary estimate of the position of the maintenance target in the three-dimensional space is made;

[0026] Region division and screening: According to the preliminarily estimated position of the maintenance target, a target region is divided in the three-dimensional space. The three-dimensional point cloud data and image data within the target region are further analyzed, and using the shape and color features of the maintenance target, the part that is the maintenance target is screened out from the environmental data within the target region.

[0027] Further, in the step S5, for the accurate determination of the pose, specifically:

[0028] Feature matching and pose calculation: In the screened target region, more detailed feature points of the maintenance target are extracted, and these feature points are matched with the features in the feature database of the maintenance target. Through the matched feature point pairs, the accurate pose of the maintenance target relative to the live maintenance mechanical device is calculated using the pose analysis algorithm;

[0029] Multi-sensor fusion optimization: The distance information measured by the laser rangefinder and the attitude information measured by the inertial measurement unit are fused with the pose result obtained by visual calculation, and a fusion algorithm is used to optimize the pose information, and the pose data of the live maintenance mechanical device after precision is output.

[0030] Further, in S6, for path planning and motion guidance, specifically:

[0031] Path planning: According to the determined maintenance target pose after precision and the current pose of the live maintenance mechanical device, use the RRT path planning algorithm to plan a guiding motion path from the current position to the maintenance target.

[0032] Motion guidance: Convert the planned guiding motion path into motion commands for each joint and actuator of the live maintenance mechanical device, and precisely control the motion of the live maintenance mechanical device through the controller of the control system. During the motion process, continuously monitor using the vision system and auxiliary positioning sensors, and adjust the motion trajectory of the live maintenance mechanical device in real time.

[0033] Further, in S7, for pose feedback and correction, specifically:

[0034] Real-time monitoring and error calculation: During the process of the live maintenance mechanical device moving towards the maintenance target, continuously repeat the above steps S3 - S5, update the pose information of the maintenance target and the self-pose information of the live maintenance mechanical device in real time, and calculate the error between the current pose of the live maintenance mechanical device and the maintenance target pose, including position error and pose error.

[0035] Correction adjustment: According to the calculated error, adjust the motion direction and speed of the live maintenance mechanical device. If the error exceeds the set threshold, re-plan the guiding motion path or perform an emergency brake and re-positioning on the motion of the live maintenance mechanical device.

[0036] Further, in S7, the pose update and error calculation formulas for the maintenance target pose and the live maintenance mechanical device pose are as follows:

[0037] Pose update of the live maintenance mechanical device:

[0038] Let the pose vector of the live maintenance mechanical device at time t be where (x, y, z) are three-dimensional position coordinates, and (α, β, γ) are the attitude angles around the x, y, z axes. Update the pose of the live maintenance mechanical device through the data of the vision system, laser rangefinder and inertial measurement unit:

[0039]

[0040] where, The change amount of the pose of the live maintenance mechanical device detected by the vision system within a unit time is a vector containing position and attitude changes, that is

[0041]

[0042] is the vision pose change amount of the live maintenance mechanical device in the x direction, and fx is the focal length of the industrial camera in the x direction; is the x coordinate of the corresponding feature point in the adjacent frame images in the image coordinate system; Z vis is the distance from the live maintenance mechanical device to a certain reference plane obtained by the laser rangefinder; Similarly, the vision pose change amounts of the live maintenance mechanical device in the y and z directions can be obtained and as well as the vision change amounts of the pose angles of the live maintenance mechanical device around the x, y, and z axes and

[0043] is the change amount of the pose of the live maintenance mechanical device deduced from the laser rangefinder data within the time interval Δt. The position change is determined by measuring the change in the distance between the live maintenance mechanical device and the surrounding fixed reference points. Its position change calculation in the x direction is: is the pose change amount of the live maintenance mechanical device deduced from the laser rangefinder data in the x direction. Among them, d x,t+1 and d x,t are the distances from the live maintenance mechanical device to a certain fixed reference point in the x direction at times t + 1 and t respectively. cos(θx) is the angle between the laser beam and the positive x-axis direction. Similarly, the pose change amounts of the live maintenance mechanical device deduced from the laser rangefinder data in the y and z directions can be obtained and

[0044] is the change amount of the pose of the live maintenance mechanical device detected by the inertial measurement unit within the time interval Δt. The inertial measurement unit outputs the angular velocity and acceleration information of the live maintenance mechanical device, and the changes in the attitude angle and position are obtained through operations such as integration. Its change in the attitude angle around the x axis is:

[0045] where ω x,imu (s) is the function of the angular velocity of the live maintenance mechanical device around the x axis detected by the inertial measurement unit with respect to time within the time interval [t, t + 1]. For the position change, it is obtained by double integration based on the acceleration information. In the x direction: where αx,imu (u) is the function of the acceleration of the live maintenance mechanical device detected by the inertial measurement unit in the x-direction with respect to time within the time interval [t, t + 1]. Similarly, the change amounts in other directions can be obtained;

[0046] Update of the maintenance target pose:

[0047] Let the pose vector of the maintenance target at time t be Then

[0048] where is the change amount of the maintenance target pose observed by the vision system per unit time, which is a vector containing position and pose changes. The calculation method is the same as the vision part in the device pose update, that is is the distance from the maintenance target to a certain reference plane obtained by the laser rangefinder, are the x-coordinates of the corresponding feature points of the maintenance target in the image coordinate system in the subsequent frame and the previous frame image respectively;

[0049] is the change amount of the maintenance target pose predicted based on the maintenance target recognition model per unit time, which is a vector containing position and pose changes. The calculation method is where f model is the function corresponding to the maintenance target recognition model, and It is the image data at the current time t;

[0050] Error calculation:

[0051] Position error Position error vector, each component of which is the difference in position coordinates between the maintenance target and the live maintenance mechanical device in the corresponding coordinate axis direction;

[0052] Attitude error Attitude error vector, each component of which is the difference in attitude angles between the maintenance target and the live maintenance mechanical device in the corresponding coordinate axis direction.

[0053] Furthermore, in the above S7, the calculation formulas for the motion adjustment and the re-planning of the motion trajectory of the live maintenance mechanical device are as follows:

[0054] Let the motion speed vector of the live maintenance mechanical device be

[0055] V t =(v x,t , v y,t , v z,t , ω x,t, , ω y,t, , ω z,t, ), then where

[0056] is the velocity adjustment vector calculated according to the error, and each component corresponds to the adjustment amount of the velocity in each direction;

[0057] is the linear velocity adjustment amount of the live maintenance mechanical device in the x direction, where kv is a proportionality constant related to velocity adjustment, and Δt is the time interval. The adjustment amount of the linear velocity in the x direction is obtained by multiplying the change amount of the position error between the maintenance target and the live maintenance mechanical device in the x direction per unit time by this proportionality constant;

[0058] is the angular velocity adjustment amount of the live maintenance mechanical device around the x axis, where k ω is a proportionality constant related to angular velocity adjustment. The adjustment amount of the angular velocity in the direction around the x axis is obtained by multiplying the change amount of the attitude angle error between the maintenance target and the live maintenance mechanical device around the x axis per unit time by this proportionality constant. Similarly, the linear velocity adjustment amounts of the live maintenance mechanical device in the y and z directions and the angular velocity adjustment amounts around the y and z axes can be obtained.

[0059] Re-plan the guiding motion path:

[0060] Let the pose of the re-planned guiding motion path point be P i new (i = 1, 2, …, n), then the cost function is The cost function is used to measure the quality of the newly planned path;

[0061] The velocity of the new path point is That is, the motion velocity vector of the i-th point on the new path, where Δt new is the time interval between two adjacent points on the new path;

[0062] Emergency braking:

[0063] When the error reaches the condition V t+1 =(0, 0, 0, 0, 0, 0), re-initialize and plan the path.

[0064] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0065] The pose identification method of the live maintenance mechanical device based on visual recognition proposed by the present invention equips the live maintenance mechanical device with industrial cameras, laser rangefinders, inertial measurement units, etc., first collects environmental and maintenance target sample data and trains the model; during operation, real-time data is collected, and after preprocessing, target recognition and positioning, precise pose calculation and multi-sensor fusion optimization, the precise pose is obtained, and the path is planned accordingly to guide the movement of the live maintenance mechanical device. During the movement, the pose is continuously monitored, adjusted or re-planned according to the error. The multi-sensor fusion and visual recognition ensure the accuracy of the pose, lay the foundation for path planning, can avoid dangerous areas, and the real-time feedback correction mechanism enables the live maintenance mechanical device to adapt to environmental changes, improves the success rate, quality and efficiency of maintenance, reduces costs, and effectively guarantees the safe and efficient progress of live maintenance operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is the overall flowchart of the pose identification method of the live maintenance mechanical device based on visual recognition of the present invention;

[0067] Figure 2 is the module diagram of the pose identification control system of the live maintenance mechanical device based on visual recognition of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0068] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0069] To solve the problems that live maintenance faces a complex environment, the electrical equipment layout is complex and there are many obstacles, it is difficult to accurately locate the maintenance target and the device pose by traditional methods, and it is impossible to adjust in real time during movement and the accuracy is insufficient. Please refer to Figure 1 - Figure 2 , the following preferred technical solutions are provided:

[0070] The pose identification method of the live maintenance mechanical device based on visual recognition includes the following specific implementation steps:

[0071] The first step, equipment preparation and environmental information acquisition: Install a number of industrial cameras and auxiliary positioning sensors on the live maintenance mechanical device, and collect and preprocess the maintenance environment data; that is

[0072] Install and configure the vision system

[0073] Install multiple industrial cameras on the live maintenance mechanical device to ensure that the perspectives of these cameras can comprehensively cover the areas where the maintenance target may appear, and obtain image information from different angles.

[0074] Install auxiliary positioning sensors simultaneously, including a camera parameter adjustment device, a laser rangefinder, and an inertial measurement unit (IMU). The camera parameter adjustment device is used to flexibly adjust parameters such as the focal length, aperture, and exposure time of the industrial camera to adapt to different maintenance environments and target characteristics, ensuring clear and accurate visual image data is collected.

[0075] Collect environmental data

[0076] Before carrying out maintenance operations, use a movable scanning device (such as a drone) to comprehensively scan the entire maintenance environment.

[0077] Collect three-dimensional point cloud data and high-resolution images of the environment, focusing on key information such as the layout of electrical equipment around the maintenance target, the position and shape of potential obstacles, etc.

[0078] Transmit the collected information to the control system of the live maintenance mechanical device, store it in the system, and perform necessary data processing in advance to provide basic data support for subsequent operations.

[0079] The second step: Target feature extraction and model training: Collect image samples of the electrical equipment to be maintained, extract key features, train the maintenance target recognition model, and establish a feature database; that is

[0080] Target feature analysis

[0081] For the electrical equipment or components to be maintained, collect a large number of image samples under different angles and lighting conditions.

[0082] Use computer vision algorithms to deeply analyze these images and extract various key features of the maintenance target, covering shape features (such as contours, geometric shapes, etc.), texture features (surface patterns, designs, etc.), color features (specific color distribution patterns), and unique local features (such as specific markings, interface shapes, etc.).

[0083] Model training

[0084] With the help of a deep learning framework, build a maintenance target recognition model based on a convolutional neural network (CNN).

[0085] Input the key features of the maintenance target extracted as training data into the model, and fully train the model so that it can accurately identify the maintenance target in different states.

[0086] At the same time, establish a maintenance target feature database to store various feature data of the maintenance target and the corresponding three-dimensional spatial position information, providing a basis for subsequent pose calculation and matching.

[0087] Step 3. Real-time Visual Data Acquisition and Preprocessing: During maintenance, acquire and preprocess visual, distance, and attitude data; that is

[0088] Data Acquisition

[0089] After the maintenance mechanical device starts working, each industrial camera starts synchronously and continuously acquires image data of the maintenance target and its surrounding environment.

[0090] The laser rangefinder measures the distance information between the live maintenance mechanical device and the maintenance target and surrounding objects in real time, providing depth data for target positioning.

[0091] The inertial measurement unit (IMU) continuously records the attitude changes of the live maintenance mechanical device, assisting in determining the pose state of the device itself. All these data are acquired at a high frequency to ensure that the dynamic changes of the target and the environment can be captured in time.

[0092] Data Preprocessing

[0093] Perform a series of preprocessing operations on the acquired image data. First, perform denoising processing. Median filtering, Gaussian filtering, etc. can be used to remove noise interference in the image and improve image quality. Then, perform contrast enhancement processing, such as histogram equalization, etc., to enhance the contrast between the target and the background in the image, facilitating subsequent feature extraction. Finally, perform normalization processing on the image to unify the image data into a specific standard range.

[0094] For the distance data measured by the laser rangefinder and the attitude data recorded by the inertial measurement unit (IMU), perform data calibration and filtering processing to remove outliers and noise interference, improving the accuracy and reliability of the data.

[0095] Step 4. Target Recognition and Preliminary Positioning: Identify the target and perform preliminary positioning through the model and sensor data, and divide the region of interest to screen the target; that is

[0096] Target Recognition

[0097] Input the preprocessed image data into a pre-trained maintenance target recognition model. The model will analyze the input image and output whether there is a maintenance target in the image and the approximate position information of the maintenance target in the image.

[0098] Combined with the distance data measured by the laser rangefinder and the attitude data recorded by the inertial measurement unit (IMU), comprehensively estimate the position of the maintenance target in three-dimensional space.

[0099] Region Division and Screening

[0100] According to the preliminarily estimated position of the maintenance target, delimit a target area (region of interest, ROI) in three-dimensional space.

[0101] Further in-depth analysis is carried out on the 3D point cloud data and image data within the ROI. By making full use of the shape and color characteristics of the maintenance target, the part that is most likely to be the maintenance target is accurately selected from the complex environmental data, further narrowing down the target range.

[0102] Step Five: Precise Pose Determination: Precisely determine the target and optimize by fusing multi-sensor data; that is

[0103] Feature Matching and Pose Calculation

[0104] In the selected target area, more detailed feature points of the maintenance target are extracted. Feature extraction algorithms such as SIFT and SURF can be used.

[0105] These feature points are matched with the features in the feature database of the maintenance target. Through the successfully matched feature point pairs, algorithms such as perspective transformation and triangulation are used to accurately calculate the pose of the maintenance target relative to the live maintenance mechanical device, including position coordinates and attitude angles.

[0106] Multi-Sensor Fusion Optimization

[0107] The distance information measured by the laser rangefinder and the attitude information measured by the inertial measurement unit (IMU) are fused with the pose results obtained through visual calculation. Data fusion algorithms such as Kalman filtering or particle filtering are used to optimize the initially calculated pose information, improve the accuracy and stability of pose determination, and finally output the precise pose data of the live maintenance mechanical device.

[0108] Step Six: Path Planning and Motion Guidance: Plan a path based on the target and the pose of the live maintenance mechanical device and guide the live maintenance mechanical device to move to the maintenance position, and continuously monitor during the movement; that is

[0109] Path Planning

[0110] According to the accurately determined pose of the maintenance target and the current pose of the live maintenance mechanical device, the RRT path planning algorithm is used to carefully plan a guiding motion path from the current position to the maintenance target position. When planning the path, the particularity of the live environment is fully considered to ensure that the path avoids electrical dangerous areas and other various obstacles, guaranteeing the safety of the maintenance process.

[0111] Motion Guidance

[0112] Convert the planned guiding motion path into motion instructions for each joint and actuator of the live maintenance mechanical device. Precisely control the live maintenance mechanical device to move along the planned path through the controller in the control system. During the movement, continuously use the vision system and auxiliary positioning sensors for real-time monitoring, timely obtain the actual position and attitude information of the device, and adjust the motion trajectory of the live maintenance mechanical device in real time according to the monitoring results to ensure that the device can accurately and stably reach the maintenance position.

[0113] Step 7. Pose feedback and correction: During the movement of the live maintenance mechanical device, update the maintenance target and the pose of the live maintenance mechanical device in real time, calculate the error, and adjust the motion or re-plan the path and perform emergency braking for re-positioning according to the error; that is

[0114] Real-time monitoring and error calculation

[0115] During the movement of the live maintenance mechanical device towards the maintenance target, continuously repeat the above steps 1 to 5 to continuously update the pose information of the maintenance target and the self-pose information of the live maintenance mechanical device.

[0116] Calculate the error between the current pose of the live maintenance mechanical device and the pose of the maintenance target, including the position error (obtained by calculating the difference in position coordinates in each axis direction of the three-dimensional space between the two) and the attitude error (calculating the difference in attitude angles around the x, y, and z axes between the two).

[0117] Correction and adjustment

[0118] According to the calculated error, adjust the motion direction and speed of the live maintenance mechanical device. If the error does not exceed the set threshold, adjust the motion speed and direction according to the error in the conventional manner to make the device gradually approach the target; if the error exceeds the set threshold, it is necessary to re-plan the guiding motion path, or in an emergency, perform emergency braking on the motion of the live maintenance mechanical device, and re-perform pose positioning and path planning to ensure the safe and accurate execution of the maintenance operation.

[0119] Formulas related to pose update and motion adjustment

[0120] (1) Pose update formula

[0121] Pose update of the live maintenance mechanical device

[0122] Let the pose vector of the live maintenance mechanical device at time t be where (x, y, z) represents the three-dimensional position coordinates, and (α, β, γ) represents the attitude angles around the x, y, and z axes.

[0123] The device pose update formula is:

[0124] where The change amount of the device pose detected by the vision system within a unit time, that is

[0125] For Its calculation method is (fx is the focal length of the industrial camera in the x direction; is the x coordinate of the corresponding feature point in the adjacent frame images in the image coordinate system; Z vis is the distance from the live maintenance mechanical device obtained by the laser rangefinder to a certain reference plane), and similarly, and and the visual change amounts of the attitude angles around the x, y, and z axes and

[0126] is the change amount of the pose of the live maintenance mechanical device deduced from the laser rangefinder data within the time interval Δt. The position change is determined by measuring the change in the distance between the live maintenance mechanical device and the surrounding fixed reference points. Its position change calculation in the x direction is: is the pose change amount of the live maintenance mechanical device deduced from the laser rangefinder data in the x direction, (d x,t+1 and d x,t are the distances from the live maintenance mechanical device to a certain fixed reference point in the x direction at times t + 1 and t respectively; cos(θx) is the angle between the laser beam and the positive x-axis direction), and similarly, the pose change amounts of the live maintenance mechanical device deduced from the laser rangefinder data in the y and z directions can be obtained and and the visual change amounts of the attitude angles around the x, y, and z axes

[0127] is the change amount of the pose of the live maintenance mechanical device detected by the inertial measurement unit within the time interval Δt. The inertial measurement unit outputs the angular velocity and acceleration information of the live maintenance mechanical device, and the change in the attitude angle and position is obtained through operations such as integration. Its change in the attitude angle around the x axis is:

[0128] (ω x,imu (s) is the function of the angular velocity of the live maintenance mechanical device around the x axis detected by the inertial measurement unit with respect to time within the time interval [t, t + 1]), and for the position change, it is obtained by double integration based on the acceleration information. In the x direction: (α x,imu(u) is the function of the acceleration of the live maintenance mechanical device in the x - direction detected by the inertial measurement unit within the time interval [t, t + 1], and the change amounts in other directions can be obtained similarly.

[0129] Overhaul target pose update

[0130] Let the pose vector of the overhaul target at time t be

[0131] The target pose update formula is:

[0132] Where is the change amount of the overhaul target pose observed by the vision system per unit time, which is a vector containing position and attitude changes. The calculation method is the same as the vision part in the device pose update, that is ( is the distance from the overhaul target to a certain reference plane obtained by the laser rangefinder; are the x - coordinates of the corresponding feature points of the overhaul target in the image coordinate system in the latter frame and the former frame of images respectively, and similarly and the visual change amounts of the attitude angles around the x, y, and z axes

[0133] is the change amount of the overhaul target pose predicted by the maintenance target recognition model per unit time (f model is the function corresponding to the maintenance target recognition model; It is the image data at the current time t), and similarly and the model - predicted change amounts of the attitude angles around the x, y, and z axes

[0134] (II) Error calculation formula

[0135] Position error

[0136] The position error vector, each component of which represents the difference in the position coordinates of the overhaul target and the live maintenance mechanical device in the corresponding coordinate axis direction.

[0137] Attitude error

[0138] The attitude error vector, each component of which represents the difference in the attitude angles of the overhaul target and the live maintenance mechanical device in the corresponding coordinate axis direction.

[0139] (III) Motion adjustment formula

[0140] Let the motion speed vector of the live maintenance mechanical device be V t =(v x,t ,vy,t , v z,t , ω x,t ,, ω y,t, , ω z,t, ).

[0141] The formula for adjusting the movement speed is as follows:

[0142] where is the vector of speed adjustment amount calculated according to the error,

[0143] (kv is a proportional constant related to speed adjustment; Δt is the time interval), by multiplying the change amount of the position error between the maintenance target and the live maintenance mechanical device in the x - direction within a unit time, the adjustment amount of the linear speed in the x - direction is obtained. Similarly,

[0144] (k ω is a proportional constant related to angular velocity adjustment), by multiplying the change amount of the attitude angle error between the maintenance target and the live maintenance mechanical device around the x - axis within a unit time, the adjustment amount of the angular velocity in the direction around the x - axis is obtained. Similarly,

[0145] (4) Formula for re - planning the path

[0146] Let the pose of the point guiding the re - planned movement path be P i new (i = 1, 2, …, n).

[0147] The cost function is used to measure the quality of the newly planned path. It is the sum of the distances from each point on the new path to the target pose and the current pose of the device. The optimal re - planned path is obtained by minimizing this cost function. Where P t tar is the pose of the maintenance target at time t, and P t dev is the pose of the live maintenance mechanical device at time t.

[0148] The speed of the new path point is (Δt new is the time interval between two adjacent points on the new path), that is, the movement speed vector of the i - th point on the new path.

[0149] (5) Emergency braking condition

[0150] When the error reaches the condition V t+1=(0, 0, 0, 0, 0, 0), that is, when the motion speed vector becomes 0, it is necessary to re-initialize the pose estimation and planned path to ensure the safe and accurate progress of the maintenance operation.

[0151] Specifically, install industrial cameras and auxiliary positioning sensors on the live maintenance mechanical device, use mobile scanning devices such as drones to collect and preprocess the maintenance environment data, then collect image samples of the electrical equipment to be maintained, extract features to train the maintenance target recognition model and establish a feature database. During maintenance, industrial cameras, etc. collect visual, distance, and attitude data and preprocess it. Input the image data into the model to identify the target and initially position it in combination with the sensor data. Divide the region of interest to screen the target, then extract feature points in the screened area and match them with the database to calculate the target pose. Optimize by fusing multi-sensor data, then plan the path according to the maintenance target and the pose of the live maintenance mechanical device and guide the movement of the live maintenance mechanical device. Continuously monitor during the movement, and continuously repeat the previous steps during the movement of the live maintenance mechanical device to update the pose, calculate the error, adjust the movement according to the error or re-plan the path, and perform emergency braking for re-positioning; through multi-sensor fusion and visual recognition technology, it is possible to accurately determine the pose of the maintenance target and the live maintenance mechanical device, effectively improve the accuracy and stability of pose identification. The accurate pose information provides a reliable basis for path planning, ensuring that the planned path is safe and collision-free, guaranteeing the safe and efficient progress of the live maintenance operation. The real-time pose feedback and correction mechanism enables the live maintenance mechanical device to adjust its movement in a timely manner according to the actual situation, adapt to the complex and changeable maintenance environment, greatly improve the success rate and quality of the maintenance, reduce operation errors and risks caused by inaccurate poses, and at the same time help improve the maintenance efficiency and reduce labor costs and time costs.

[0152] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device.

[0153] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for identifying the posture of a live maintenance mechanical device based on visual recognition, characterized in that: The specific implementation steps include the following: S1: Equipment preparation and environmental information acquisition: install a number of industrial cameras and auxiliary positioning sensors on the live maintenance mechanical device to collect and pre-process the maintenance environment data; S2: Target feature extraction and model training: collect image samples of electrical equipment to be repaired, extract key features, train the maintenance target recognition model and establish a feature database; S3: Real-time visual data acquisition and preprocessing, collecting and preprocessing visual, distance and posture data during maintenance; S4: Target recognition and preliminary positioning: identifying and preliminarily positioning targets through models and sensor data, dividing regions of interest and screening targets; S5: Accurately determine the position and pose, accurately determine the target and fuse multi-sensor data for optimization; S6: Path planning and motion guidance, planning the path according to the target and the posture of the live maintenance mechanical device and guiding the live maintenance mechanical device to move to the maintenance position, and continuously monitoring during the movement; S7: Posture feedback and correction: update the maintenance target and the posture of the live maintenance mechanical device in real time during its movement, calculate the error, adjust the movement or re-plan the path, perform emergency braking and repositioning according to the error, until the live maintenance mechanical device completes the maintenance of the electrical equipment.

2. The method for position and posture identification of a live maintenance mechanical device based on visual recognition according to claim 1, characterized in that: In S1, the equipment preparation and environment information acquisition are specifically as follows: Install and configure the visual system, install several industrial cameras on the live maintenance mechanical device, these industrial cameras cover the area where the maintenance target appears from different angles, and at the same time, install auxiliary positioning sensors on the live maintenance mechanical device; Collect environmental data. Before maintenance work, use mobile scanning equipment such as drones to scan the entire maintenance environment, obtain three-dimensional point cloud data and clear images of the environment, and transmit information including the layout of electrical equipment near the maintenance target and the location and shape of potential obstacles to the control system of the live maintenance mechanical device for storage and preprocessing.

3. The method for identifying the posture of a live maintenance mechanical device based on visual recognition according to claim 2, characterized in that: The industrial camera is responsible for collecting image data of the inspection target and its surrounding environment, and the auxiliary positioning sensor specifically includes: Camera parameter adjustment device, used to adjust parameters including focal length, aperture and time, so as to promote the industrial camera to collect visual image data for different maintenance environments and target machinery; Laser rangefinder, used to measure the distance information between the live maintenance mechanical device and the maintenance target and its surrounding objects in real time, providing depth data for the positioning of the maintenance target; The inertial measurement unit is used to record the posture changes of the live maintenance mechanical device and assist in determining the posture state of the live maintenance mechanical device itself.

4. The method for position and posture identification of a live maintenance mechanical device based on visual recognition according to claim 1, characterized in that: In S2, for target feature extraction and model training, specifically: Target feature analysis: Collect sufficient image samples for the electrical equipment or components to be repaired, analyze these images using computer vision algorithms, and extract key feature information of the repair target, including shape features, texture features, color features, and local features; Model training: Using a deep learning framework, we build a maintenance target recognition model based on a convolutional neural network. The key features of the extracted maintenance targets are input into the model as training data to train the maintenance target recognition model. At the same time, we establish a maintenance target feature database to store various feature data of the maintenance targets and their corresponding three-dimensional spatial position information.

5. The method for position and posture identification of a live maintenance mechanical device based on visual recognition as claimed in claim 3, characterized in that: In S4, the target identification and preliminary positioning are specifically as follows: Target recognition: The pre-processed image data collected for the maintenance target and its surrounding environment is input into the trained maintenance target recognition model. The maintenance target recognition model outputs information about whether there is a maintenance target in the image and the location of the maintenance target in the image. Combined with the data from the laser rangefinder and the inertial measurement unit, the location of the maintenance target in three-dimensional space is preliminarily estimated; Area division and screening: Based on the preliminary estimated location of the maintenance target, a target area is divided in three-dimensional space, and the three-dimensional point cloud data and image data in the target area are further analyzed. The shape and color characteristics of the maintenance target are used to screen out the maintenance target part from the environmental data of the target area.

6. The method for position and posture identification of a live maintenance mechanical device based on visual recognition as claimed in claim 5, characterized in that: In S5, the position and posture are accurately determined, specifically: Feature matching and posture calculation: In the selected target area, more detailed feature points of the maintenance target are extracted, and these feature points are matched with the features in the feature database of the maintenance target. Through the matched feature point pairs, the posture analysis algorithm is used to calculate the precise posture of the maintenance target relative to the live maintenance mechanical device; Multi-sensor fusion optimization: The distance information measured by the laser rangefinder and the posture information measured by the inertial measurement unit are integrated with the posture results obtained by visual calculation, and the fusion algorithm is used to optimize the posture information to output accurate posture data of the live maintenance mechanical device.

7. The method for position and posture identification of a live maintenance mechanical device based on visual recognition as claimed in claim 6, characterized in that: In S6, the path planning and motion guidance are specifically as follows: Path planning: Based on the precisely determined position of the maintenance target and the current position of the live maintenance mechanical device, the RRT path planning algorithm is used to plan a guided motion path from the current position to the maintenance target; Motion guidance: Convert the planned guided motion path into motion instructions for each joint and actuator of the live maintenance mechanical device, accurately control the motion of the live maintenance mechanical device through the controller of the control system, continuously use the visual system and auxiliary positioning sensors for monitoring during the motion process, and adjust the motion trajectory of the live maintenance mechanical device in real time.

8. The method for position and posture identification of a live maintenance mechanical device based on visual recognition as claimed in claim 7, characterized in that: In S7, the posture feedback and correction are specifically as follows: Real-time monitoring and error calculation: When the live maintenance mechanical device moves toward the maintenance target, the above steps S3-S5 are repeated continuously, the posture information of the maintenance target and the posture information of the live maintenance mechanical device are updated in real time, and the error between the current posture of the live maintenance mechanical device and the posture of the maintenance target is calculated, including position error and posture error; Correction adjustment: According to the calculated error, adjust the movement direction and speed of the live maintenance mechanical device. If the error exceeds the set threshold, re-plan and guide the movement path or perform emergency braking and repositioning on the movement of the live maintenance mechanical device.

9. The method for position and posture identification of a live maintenance mechanical device based on visual recognition as claimed in claim 8, characterized in that: In S7, the posture update and error calculation formula for the posture of the maintenance target and the posture of the live maintenance mechanical device are as follows: Update of the position of the live maintenance mechanical device: Let the posture vector of the live maintenance mechanical device at time t be Where (x, y, z) is the three-dimensional position coordinate, (α, β, γ) is the attitude angle around the x, y, z axis, and the posture of the live maintenance mechanical device is updated through the data of the visual system, laser rangefinder and inertial measurement unit: in, is the change in the position and posture of the live maintenance mechanical device detected by the visual system in unit time, which is a vector containing position and posture changes, that is, is the visual posture change of the live maintenance mechanical device in the x direction, and fx is the focal length of the industrial camera in the x direction; is the x coordinate of the corresponding feature point in the adjacent frame image in the image coordinate system; Z vis is the distance from the live maintenance mechanical device to a certain reference plane obtained by the laser rangefinder; similarly, the visual posture change of the live maintenance mechanical device in the y and z directions can be obtained and And the visual change of the posture angle of the live maintenance mechanical device around the x, y, and z axes and It is the change in the posture of the live maintenance mechanical device in the time interval Δt calculated by the laser rangefinder data. The position change is determined by measuring the change in the distance between the live maintenance mechanical device and the surrounding fixed reference points. The position change in the x direction is calculated as: is the position change of the live maintenance mechanical device in the x direction calculated from the laser rangefinder data, where d x,t+1 and d x,t are the distances from the live maintenance mechanical device to a fixed reference point in the x direction at time t+1 and t respectively. cos(θx) is the angle between the laser beam and the positive direction of the x axis. Similarly, the position changes of the live maintenance mechanical device in the y and z directions calculated from the laser rangefinder data can be obtained. and It is the change of the posture of the live maintenance mechanical device detected by the inertial measurement unit within the time interval Δt. The inertial measurement unit outputs the angular velocity and acceleration information of the live maintenance mechanical device. The change of the posture angle and position is obtained through integration and other operations. The change of the posture angle around the x-axis is: where ω x,imu (s) is the angular velocity of the live maintenance mechanical device around the x-axis detected by the inertial measurement unit in the time interval [t, t+1] as a function of time. The position change is obtained by quadratic integration based on the acceleration information. In the x-direction: where α x,imu (u) is the acceleration of the live maintenance mechanical device in the x direction detected by the inertial measurement unit in the time interval [t, t+1] as a function of time. Similarly, the changes in other directions can be obtained; Overhaul target pose update: Let the posture vector of the maintenance target at time t be but in is the change in the position and posture of the maintenance target observed by the visual system in unit time. It is a vector containing position and posture changes. The calculation method is the same as the visual part in the device posture update, that is, It is the distance from the inspection target to a certain reference plane obtained by the laser rangefinder. are the x-coordinates of the corresponding feature points of the inspection target in the next frame and the previous frame in the image coordinate system; It is the change of the maintenance target posture in unit time based on the maintenance target recognition model prediction. It is a vector containing position and posture changes. It is calculated as where f model is the function corresponding to the maintenance target recognition model, It is the image data at the current time t; Error calculation: Position error Position error vector, each component of which is the position coordinate difference between the maintenance target and the live maintenance mechanical device in the direction of the corresponding coordinate axis; Attitude Error The attitude error vector, each component of which is the attitude angle difference between the maintenance target and the live maintenance mechanical device in the direction of the corresponding coordinate axis.

10. The method for position and posture identification of a live maintenance mechanical device based on visual recognition according to claim 9, characterized in that: In S7, the calculation formula for the motion adjustment and re-planning of the motion trajectory of the live maintenance mechanical device is as follows: Let the motion velocity vector of the live maintenance mechanical device be V t =(v x,t ,v y,t ,v z,t ,ω x,t, ,ω y,t, ,ω z,t, ), then where is the speed adjustment vector calculated based on the error, and each component corresponds to the speed adjustment in each direction; is the linear speed adjustment of the live maintenance mechanical device in the x direction, where kv is a proportional constant related to the speed adjustment, Δt is the time interval, and the linear speed adjustment in the x direction is obtained by multiplying the change in the position error between the maintenance target and the live maintenance mechanical device in the x direction per unit time by the proportional constant; is the angular velocity adjustment of the live maintenance mechanical device around the x-axis, where k ω It is a proportional constant related to the angular velocity adjustment. The angular velocity adjustment around the x-axis is obtained by multiplying the change in the attitude angle error between the maintenance target and the live maintenance mechanical device around the x-axis by the proportional constant. Similarly, the linear velocity adjustment of the live maintenance mechanical device in the y and z directions and the angular velocity adjustment around the y and z axes can be obtained. Re-planning the guided motion path: Let the position of the re-planned guided motion path point be P i new (i=1,2,…,n), then the cost function is The cost function is used to measure the quality of the new planned path; The new waypoint speed is That is, the motion velocity vector of the i-th point on the new path, where Δtnew is the time interval between two adjacent points on the new path; Emergency Braking: When the error reaches the condition Vt +1 =(0,0,0,0,0,0), reinitialize and plan the path.

Images