Cable positioning method, device and equipment and storage medium
By acquiring video streams of target locations in substations using depth cameras and combining deep learning and chessboard calibration technologies, efficient and accurate cable positioning was achieved. This solved the safety hazards and installation quality problems of traditional robotic equipment operating in confined spaces with live wires, and improved the intelligence level and operation and maintenance management quality of substations.
Patent Information
- Application Number
- CN202411143244.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-22
AI Technical Summary
In substations, traditional robotic equipment cannot efficiently perform live-line work on 110kV high-voltage cables and transformer equipment in confined spaces, posing safety hazards and making it difficult to guarantee installation quality.
By acquiring color and depth video streams of the target location in the substation using a depth camera, the pixel coordinates and depth distance of the cable are determined. This is then combined with a robotic arm controller for cable positioning, and deep learning and chessboard calibration techniques are used to improve positioning accuracy.
It enables efficient and accurate cable positioning, reduces manual intervention, lowers the probability of errors, improves work efficiency and safety, and enhances the intelligence level and operation and maintenance management quality of substations.
Smart Images

Figure CN119006558A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of power distribution networks, and in particular to a cable positioning method, device, equipment and storage medium. Background Art
[0002] In the field of power, substation equipment is a vital component of the power system. With the continuous development of power technology and the increasing complexity of substations, the demand for efficient operation and maintenance of substation equipment is also growing.
[0003] In the existing technology, employees of major power grid companies often need to come into contact with 110kV high-voltage cables and various substation equipment during their daily work. The manual disassembly and installation of substation drainage wires need to be carried out in a live environment, and the existing safety hazards always threaten personal safety. In the scenario of dense substation equipment, traditional engineering equipment cannot operate in a small space. The output capacity / volume weight ratio of traditional robots is too small to guarantee the quality of drainage wire installation. Summary of the invention
[0004] The present invention provides a cable positioning method, device, equipment and storage medium to determine the cable space coordinates, which are transmitted to a robot arm controller for cable positioning, thereby ensuring the accuracy of the robot arm positioning.
[0005] According to one aspect of the present invention, a cable positioning method is provided, the method comprising:
[0006] Acquire a video stream of a target location of a substation through a depth camera, wherein the video stream includes a color video stream and a depth video stream;
[0007] Determine the pixel coordinates of the target cable according to the color video stream, and determine the depth distance of the target cable according to the pixel coordinates and the depth video stream;
[0008] The cable space coordinates are determined according to the pixel coordinates and the depth distance, and the robot arm controller is controlled based on the cable space coordinates to perform cable positioning.
[0009] Optionally, obtaining a video stream of a target location of a substation through a depth camera includes: taking the substation equipment location set by a user as the target location; shooting a color video stream of the target location through a color RGB camera of the depth camera; and shooting a depth video stream of the target location through a depth camera of the depth camera.
[0010] Optionally, the pixel coordinates of the target cable are determined based on the color video stream, including: inputting the color video stream frame by frame into a pre-built cable identification model to obtain a cable identification frame output by the model, wherein the cable identification frame includes the target cable; establishing a coordinate system based on the color video stream, and determining the horizontal and vertical coordinates of the center point of the cable identification frame in the coordinate system as the pixel coordinates of the target cable.
[0011] Optionally, determining the depth distance of the target cable according to the pixel coordinates and the depth video stream includes: aligning the depth video stream and the color video stream to generate an aligned depth video stream; and obtaining the depth distance of the pixel coordinates in the aligned depth video stream.
[0012] Optionally, the cable space coordinates are determined based on the pixel coordinates and the depth distance, including: combining the pixel coordinates and the depth distance to generate the pixel positioning coordinates of the target cable; performing chessboard calibration on the depth camera to obtain a camera intrinsic parameter matrix; and multiplying the pixel positioning coordinates and the camera intrinsic parameter matrix to generate the cable space coordinates.
[0013] Optionally, the cable positioning is performed by controlling a robotic arm controller based on the cable space coordinates, including: performing hand-eye calibration on the robotic arm and a calibration camera to determine a coordinate conversion formula; substituting the cable space coordinates into the coordinate conversion formula to determine the robotic arm control coordinates, and controlling the robotic arm controller according to the robotic arm control coordinates.
[0014] Optionally, hand-eye calibration is performed on the robotic arm and the calibration camera to determine the coordinate transformation formula, including: constructing each transformation coordinate system based on the robotic arm and the calibration camera, wherein the transformation coordinate system includes the robotic arm base coordinate system, the robotic arm end coordinate system, the camera coordinate system and the calibration plate coordinate system; generating each coordinate system transformation matrix according to each transformation coordinate system, wherein the coordinate system transformation matrix includes the calibration plate coordinate system to the robotic arm base coordinate system transformation matrix, the calibration plate coordinate system to the camera coordinate system transformation matrix, the camera coordinate system to the robotic arm end coordinate system transformation matrix and the robotic arm end coordinate system to the robotic arm base coordinate system transformation matrix; determining the coordinate transformation formula according to each coordinate system transformation matrix.
[0015] According to another aspect of the present invention, there is provided a cable positioning device, the device comprising:
[0016] A video stream acquisition module, used to acquire a video stream of a target location of a substation through a depth camera, wherein the video stream includes a color video stream and a depth video stream;
[0017] A pixel coordinate and depth distance determination module is used to determine the pixel coordinates of the target cable according to the color video stream, and to determine the depth distance of the target cable according to the pixel coordinates and the depth video stream;
[0018] The robot arm cable positioning module is used to determine the cable space coordinates according to the pixel coordinates and the depth distance, and control the robot arm controller to perform cable positioning based on the cable space coordinates.
[0019] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0020] at least one processor;
[0021] and a memory communicatively coupled to the at least one processor;
[0022] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a cable locating method described in any embodiment of the present invention.
[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a cable locating method according to any embodiment of the present invention when executed.
[0024] The technical solution of the embodiment of the present invention can obtain the video stream of the target position of the substation through the depth camera, so as to obtain the information of the target area in a comprehensive and real-time manner, and provide a rich data basis for subsequent analysis and processing. The pixel coordinates of the target cable are determined by the color video stream, so that the position of the target cable in the image can be accurately identified and located, and the precision and accuracy of positioning are improved. The depth distance of the target cable is determined by combining the pixel coordinates and the depth video stream, so as to obtain the three-dimensional spatial information of the target cable and understand the position status of the target cable more comprehensively and accurately. The cable space coordinates are determined according to the pixel coordinates and the depth distance, so that the description of the cable position is more accurate and specific, and a reliable basis is provided for the precise control of the robot controller. The cable positioning is controlled by the robot controller based on the cable space coordinates, so that efficient and accurate cable positioning operation can be realized, human intervention can be reduced, the error probability can be reduced, work efficiency and safety can be improved, and the normal operation and maintenance of the substation equipment can be guaranteed. At the same time, it is also conducive to improving the overall intelligence level and operation and maintenance management quality of the substation.
[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0027] Figure 1 is a flow chart of a cable positioning method provided according to Embodiment 1 of the present invention;
[0028] Figure 2 is a flow chart of another cable locating method provided according to Embodiment 2 of the present invention;
[0029] Figure 3 is a schematic structural diagram of a cable positioning device provided according to Embodiment 3 of the present invention;
[0030] Figure 4 It is a structural schematic diagram of an electronic device for implementing a cable locating method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0033] Embodiment 1
[0034] Figure 1 A flowchart of a cable positioning method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of controlling a mechanical arm to position a cable. The method can be performed by a cable positioning device. The cable positioning device can be implemented in the form of hardware and / or software. The cable positioning device can be configured in a computer controller. Figure 1 As shown, the method includes:
[0035] S110. Acquire a video stream of a target location of a substation through a depth camera, wherein the video stream includes a color video stream and a depth video stream.
[0036] Among them, the depth camera can be an Orbbec FemtoW depth camera, and the video stream refers to a data stream formed by transmitting continuous image frames in chronological order.
[0037] Specifically, the color video stream can present rich colors and clear image information of the target location, while the depth video stream can provide relevant data about the distance between each point of the target location and the camera. When acquiring the video stream of the target location of the substation, the color camera part of the depth camera is responsible for collecting the color video stream, focusing the light on the image sensor through the lens, and forming an image with rich colors and details. The depth acquisition module in the depth camera generates a depth video stream. For example, a specific coded pattern is projected, and when these patterns are deformed after being irradiated onto the surface of an object, the camera captures the deformed pattern and determines the distance between the object and the camera by calculating the degree of deformation of the pattern, thereby forming a depth video stream.
[0038] Optionally, obtaining a video stream of a target location of a substation through a depth camera includes: taking the substation equipment location set by a user as the target location; shooting a color video stream of the target location through a color RGB camera of the depth camera; and shooting a depth video stream of the target location through a depth camera of the depth camera.
[0039] Specifically, the user needs to set the location of the substation equipment as the target location according to actual needs. The color RGB camera in the depth camera adopts the RGB color imaging principle. That is, light enters the camera through the lens and shines on the image sensor. Each pixel senses the light intensity of the three colors of red (R), green (G), and blue (B), and then generates a color video stream through signal processing and encoding. Through the collaborative work of the color RGB camera and the depth camera of the depth camera, the color video stream and depth video stream of the target location can be obtained respectively. The color video stream provides a clear appearance and color features of the object at the target location, and the depth video stream supplements the spatial distance information of the object.
[0040] S120, determining the pixel coordinates of the target cable according to the color video stream, and determining the depth distance of the target cable according to the pixel coordinates and the depth video stream.
[0041] The pixel coordinate refers to the position of the target cable on the two-dimensional plane. The depth distance refers to the straight-line distance between the target cable and the depth camera.
[0042] Specifically, the controller can determine the pixel coordinates of the target cable based on the color video stream, that is, process and analyze the color image through image processing algorithms, such as edge detection and feature extraction, to determine the pixel position of the cable in the image. Each pixel in the depth video stream corresponds to a distance value. By extracting the depth value corresponding to the pixel coordinate, the depth distance between the target cable and the camera can be obtained.
[0043] Optionally, the pixel coordinates of the target cable are determined based on the color video stream, including: inputting the color video stream frame by frame into a pre-built cable identification model to obtain a cable identification frame output by the model, wherein the cable identification frame includes the target cable; establishing a coordinate system based on the color video stream, and determining the horizontal and vertical coordinates of the center point of the cable identification frame in the coordinate system as the pixel coordinates of the target cable.
[0044] Specifically, the controller inputs the color video stream frame by frame into the pre-built cable recognition model. The cable recognition model is built based on deep learning technology, such as convolutional neural network (CNN). Convolutional neural network can automatically learn the characteristic patterns of cables from a large amount of training data. During the training phase, the model is exposed to a large number of images containing cables in various scenarios, and adjusts internal parameters based on the correct information manually annotated (i.e. which areas are cables) to gradually improve the recognition ability of cables. When each frame of the color video stream is input, the model extracts, analyzes and classifies the image features, and finally outputs a cable recognition box to frame the area where the target cable is located.
[0045] Furthermore, the controller will establish a coordinate system based on the color video stream. The coordinate system is usually established with the upper left corner of the image as the origin, the horizontal right as the positive direction of the X axis, and the vertical downward as the positive direction of the Y axis. Then, in this coordinate system, the center point of the cable identification frame is determined. For example, by obtaining the coordinates of the upper left corner and the lower right corner of the identification frame, then adding the corresponding X coordinates and dividing by 2 to obtain the X coordinate of the center point, and adding the corresponding Y coordinates and dividing by 2 to obtain the Y coordinate of the center point.
[0046] It should be noted that the cable recognition model usually selects a convolutional neural network architecture suitable for image recognition tasks, such as ResNet, VGG, etc. The training process requires the collection of a large amount of image and video data containing cables. The collected data is then manually annotated to clearly indicate which areas are cables. The annotations can be rectangular boxes, polygons, or pixel-level markers so that the model can learn the accurate features and locations of the cables. The annotated data is then input into the model, which performs feature extraction and classification prediction through a series of convolution, pooling, and fully connected layers, and outputs the prediction results for the cables. Through continuous iteration and optimization, the cable recognition model can learn the characteristic patterns and laws of cables, so that it can accurately identify cables from new images or videos.
[0047] Optionally, determining the depth distance of the target cable according to the pixel coordinates and the depth video stream includes: aligning the depth video stream and the color video stream to generate an aligned depth video stream; and obtaining the depth distance of the pixel coordinates in the aligned depth video stream.
[0048] Specifically, the depth video stream and the color video stream need to be aligned to accurately match the pixel coordinates with the depth information. When performing the alignment operation, the camera's intrinsic and extrinsic parameters and information such as the timestamp of the image are usually used. The camera's intrinsic parameters include focal length, principal point coordinates, pixel size, etc. The extrinsic parameters include the position and posture of the camera in space. At the same time, considering that the video stream is a series of continuous image frames, it is also necessary to synchronize the color frames and depth frames according to the timestamp to ensure their temporal consistency.
[0049] Furthermore, when the pixel coordinates are known, the corresponding depth value can be obtained by searching for the position corresponding to the pixel coordinates in the aligned depth video stream. The depth value is the depth distance of the target cable at the pixel position relative to the camera.
[0050] S130, determining the cable space coordinates according to the pixel coordinates and the depth distance, and controlling the robot arm controller to perform cable positioning based on the cable space coordinates.
[0051] The cable space coordinates refer to the coordinates that describe the target cable position in three-dimensional space, usually determined by three values: X, Y, and Z. The robot controller refers to a device or system used to control the movement of the robot, which can receive instructions and convert them into actions of each joint of the robot.
[0052] Specifically, a coordinate system is established with the optical center of the camera as the origin. According to the pixel coordinates, camera internal parameters and depth distance, the pixel coordinates and depth distance can be converted into actual coordinates in three-dimensional space through trigonometric functions and matrix operations. Then, based on the obtained cable space coordinates, the control algorithm inside the robot controller will convert the space coordinates into motion parameters of each joint of the robot arm, and through the drive motor and transmission mechanism, the end effector of the robot arm will be accurately moved to the position of the cable, completing the cable positioning operation.
[0053] Optionally, the cable space coordinates are determined based on the pixel coordinates and the depth distance, including: combining the pixel coordinates and the depth distance to generate the pixel positioning coordinates of the target cable; performing chessboard calibration on the depth camera to obtain a camera intrinsic parameter matrix; and multiplying the pixel positioning coordinates and the camera intrinsic parameter matrix to generate the cable space coordinates.
[0054] Specifically, the pixel coordinates can be expressed as (x, y), which represent the horizontal and vertical positions on the image plane, respectively. The depth distance is represented by D. By combining these three values, the pixel location coordinates can be generated, such as (x, y, D).
[0055] It should be noted that in the calibration process of chessboard calibration, a chessboard with regular black and white squares is used as a calibration plate, which is placed at different positions and postures within the camera's field of view. The camera captures multiple sets of images containing the calibration plate, and then detects the corner positions of the chessboard in the image, uses the correspondence between the actual physical coordinates of these corners and the pixel coordinates in the image, combines the camera's imaging model, and solves the camera's intrinsic parameter matrix through complex mathematical calculations and optimization algorithms. The camera's intrinsic parameter matrix usually contains important parameters such as the camera's focal length (fx, fy), principal point coordinates (cx, cy), and describes the process of the camera projecting points in the three-dimensional world onto the two-dimensional image plane. Finally, the pixel positioning coordinates and the camera's intrinsic parameter matrix are multiplied to generate the cable space coordinates. For example, assuming that the pixel positioning coordinates are (x, y, D), and the camera intrinsic parameter matrix is [fx 0cx 0; 0fy cy 0; 0 0 1 0], through matrix multiplication operations: X = (x-cx)*D / fx; Y = (y-cy)*D / fy; Z = D, the coordinates of the cable in three-dimensional space (X, Y, Z), that is, the cable space coordinates, can be obtained.
[0056] The technical solution of the embodiment of the present invention can obtain the video stream of the target position of the substation through the depth camera, so as to obtain the information of the target area in a comprehensive and real-time manner, and provide a rich data basis for subsequent analysis and processing. The pixel coordinates of the target cable are determined by the color video stream, so that the position of the target cable in the image can be accurately identified and located, and the precision and accuracy of positioning are improved. The depth distance of the target cable is determined by combining the pixel coordinates and the depth video stream, so as to obtain the three-dimensional spatial information of the target cable and understand the position status of the target cable more comprehensively and accurately. The cable space coordinates are determined according to the pixel coordinates and the depth distance, so that the description of the cable position is more accurate and specific, and a reliable basis is provided for the precise control of the robot controller. The cable positioning is controlled by the robot controller based on the cable space coordinates, so that efficient and accurate cable positioning operation can be realized, human intervention can be reduced, the error probability can be reduced, work efficiency and safety can be improved, and the normal operation and maintenance of the substation equipment can be guaranteed. At the same time, it is also conducive to improving the overall intelligence level and operation and maintenance management quality of the substation.
[0057] Embodiment 2
[0058] Figure 2 This is a flow chart of a cable positioning method provided in the second embodiment of the present invention. This embodiment adds a specific process of controlling the mechanical arm controller based on the cable space coordinates to position the cable on the basis of the above-mentioned first embodiment. Among them, the specific contents of steps S210-S220 are roughly the same as steps S110-S120 in the first embodiment, so they will not be repeated in this embodiment. Figure 2 As shown, the method includes:
[0059] S210. Acquire a video stream of a target location of a substation through a depth camera, wherein the video stream includes a color video stream and a depth video stream.
[0060] Optionally, obtaining a video stream of a target location of a substation through a depth camera includes: taking the substation equipment location set by a user as the target location; shooting a color video stream of the target location through a color RGB camera of the depth camera; and shooting a depth video stream of the target location through a depth camera of the depth camera.
[0061] S220 , determining the pixel coordinates of the target cable according to the color video stream, and determining the depth distance of the target cable according to the pixel coordinates and the depth video stream.
[0062] Optionally, the pixel coordinates of the target cable are determined based on the color video stream, including: inputting the color video stream frame by frame into a pre-built cable identification model to obtain a cable identification frame output by the model, wherein the cable identification frame includes the target cable; establishing a coordinate system based on the color video stream, and determining the horizontal and vertical coordinates of the center point of the cable identification frame in the coordinate system as the pixel coordinates of the target cable.
[0063] Optionally, determining the depth distance of the target cable according to the pixel coordinates and the depth video stream includes: aligning the depth video stream and the color video stream to generate an aligned depth video stream; and obtaining the depth distance of the pixel coordinates in the aligned depth video stream.
[0064] S230. Determine the cable space coordinates according to the pixel coordinates and the depth distance.
[0065] Optionally, the cable space coordinates are determined based on the pixel coordinates and the depth distance, including: combining the pixel coordinates and the depth distance to generate the pixel positioning coordinates of the target cable; performing chessboard calibration on the depth camera to obtain a camera intrinsic parameter matrix; and multiplying the pixel positioning coordinates and the camera intrinsic parameter matrix to generate the cable space coordinates.
[0066] S240, performing hand-eye calibration on the robotic arm and the calibration camera to determine a coordinate transformation formula.
[0067] Specifically, the purpose of hand-eye calibration is to establish the relative position relationship between the end effector of the robot and the calibration camera, so as to determine the coordinate transformation formula. By acquiring images of the calibration plate or specific markers in different postures and combining them with the joint angle information of the robot, the rotation and translation matrices between the camera coordinate system and the robot base coordinate system can be calculated.
[0068] Optionally, hand-eye calibration is performed on the robotic arm and the calibration camera to determine the coordinate transformation formula, including: constructing each transformation coordinate system based on the robotic arm and the calibration camera, wherein the transformation coordinate system includes the robotic arm base coordinate system, the robotic arm end coordinate system, the camera coordinate system and the calibration plate coordinate system; generating each coordinate system transformation matrix according to each transformation coordinate system, wherein the coordinate system transformation matrix includes the calibration plate coordinate system to the robotic arm base coordinate system transformation matrix, the calibration plate coordinate system to the camera coordinate system transformation matrix, the camera coordinate system to the robotic arm end coordinate system transformation matrix and the robotic arm end coordinate system to the robotic arm base coordinate system transformation matrix; determining the coordinate transformation formula according to each coordinate system transformation matrix.
[0069] Specifically, the hand-eye calibration process is that when the human eye sees an object and the hand is used to grab it, the brain needs to know the coordinate relationship between the eye and the hand. What is obtained from the camera is the pixel coordinates, and the manipulator is a spatial coordinate system, so the hand-eye calibration is to obtain the coordinate transformation relationship between the pixel coordinate system and the spatial manipulator coordinate system. Hand-eye calibration is divided into: eye on hand-the camera is fixed at the end of the manipulator, eye outside the hand-the camera is fixed somewhere outside the manipulator. This patent will explain the detailed process method of eye outside hand calibration. Before performing hand-eye calibration, the four coordinate systems used must be clarified: ① The base coordinate system of the manipulator (expressed as base), ② The end coordinate system of the manipulator (expressed as end), ③ The camera coordinate system (expressed as cam), and ④ The calibration plate coordinate system (expressed as target).
[0070] Furthermore, based on these four coordinate systems, four coordinate system transformation matrices can be derived: the transformation matrix from the calibration plate coordinate system to the robot base coordinate system Transformation matrix from calibration plate coordinate system to camera coordinate system Transformation matrix from camera coordinate system to robot end coordinate system And the transformation matrix from the robot end coordinate system to the robot base coordinate system According to the properties of matrix multiplication in linear algebra, we can know
[0071] Specifically, the calibration process includes the following steps: Step 1: Fix the calibration plate at the end of the robotic arm. Step 2: Fix the camera somewhere outside the robotic arm. Step 3: Move the robotic arm multiple times so that the camera can successfully capture the calibration plate and record the robotic arm posture data X, Y, Z, Rx, Ry, Rz at each moment. After completing the above steps, the required matrix for converting the camera coordinate system to the robotic arm base coordinate system is solved by analyzing the relationship between the transformation matrices and simplifying the formulas using known quantities. The final solution to the equation system is
[0072] It should be noted that since the camera and the base of the robot arm are fixed, The transformation matrix is constant and is the unknown quantity to be found. The equation can be viewed as AX=XB, and the unknown quantity X can be found. is the transformation matrix from the calibration plate to the camera. Each time you use the camera to shoot the calibration plate, you can obtain this matrix through the camera calibration method mentioned above. It is the transformation matrix from the base of the robot to the end of the robot, which is derived from the previously recorded robot pose data.
[0073] Specifically, for each transformation matrix T, its form is a homogeneous matrix, that is The transformation matrix T consists of a 3*3 rotation matrix R and a 3*1 displacement vector T. Each photo taken is calibrated by the camera, that is, the solvePnP function in OpenCV2 is used to obtain the rotation vector r and translation vector T corresponding to the photo. The rotation vector r is converted into the rotation matrix R through the Rodrigues function. R, T and 0001 are also formed by column merging np.column_stack and row merging np.row_stack respectively. Some of the parameters required during this period will require the use of the np.linalg.inv function to perform matrix inversion operations. and And their inverse matrices, and finally solve That's it. Considering that it is difficult to solve the equation AX=XB, we can use the calibrateHandEye function in OpenCV2 to solve it. When using this function to calibrate the hand outside the eye, the input format is as follows: CalibrateHandEye This method does not need to solve the complete transformation matrix T, but directly inputs the rotation matrix R and displacement vector t in T. The function will complete all subsequent calculations by itself. Finally, the function will return the rotation matrix of the camera to the robot base coordinate system. and translation vector We manually merge the two columns to form The ordinary matrix form (not a homogeneous matrix) ends the whole process of hand-eye calibration.
[0074] Furthermore, in obtaining After that, the final transformation process can be carried out. In the process of using yolov5 combined with a depth camera to control the robotic arm, we obtain the coordinate information and depth distance of the object in the pixel coordinate system, which needs to be finally converted into the coordinates of the robotic arm base, so that the robotic arm can move to the object to perform operations. Combining the camera intrinsic parameters and transformation matrix obtained through camera calibration and hand-eye calibration above, the coordinate transformation formula can be obtained, as shown in the following formula (1):
[0075]
[0076] in, is the rotation matrix, is the translation vector, Z C is the Z-axis distance of the object on the image, which can be approximately considered as the depth distance D, X W , Y W and Z W is the input cable space coordinate, u and v are the robot control coordinates. This formula can realize the conversion of the object in the robot base coordinate system to the pixel coordinate system. We only need to use the formula in reverse to obtain the coordinates in the robot reference system through the pixel coordinates. After obtaining the coordinates, use the communication protocol to send the coordinates to the robot.
[0077] S250, substituting the cable space coordinates into the coordinate conversion formula to determine the robot arm control coordinates, and controlling the robot arm controller according to the robot arm control coordinates.
[0078] Specifically, through the calculation of the formula, the position information of the cable in space can be converted from its original coordinate system to the control coordinate system that the robot can understand and execute. The robot controller can calculate the angle and speed of each joint according to the robot control coordinates, drive the motor to make the end of the robot reach the specified position, thereby achieving precise positioning of the cable.
[0079] In practical applications, in order to improve the accuracy and reliability of positioning, factors such as camera resolution, measurement error, and robot arm motion accuracy need to be considered, and corresponding calibration and compensation measures should be taken. At the same time, a real-time feedback control mechanism is also essential so that adjustments and corrections can be made according to actual conditions during the movement of the robot arm to ensure the accuracy of positioning.
[0080] The technical solution of the embodiment of the present invention can obtain the video stream of the target position of the substation through the depth camera, so as to obtain the information of the target area in a comprehensive and real-time manner, and provide a rich data basis for subsequent analysis and processing. The pixel coordinates of the target cable are determined by the color video stream, so that the position of the target cable in the image can be accurately identified and located, and the precision and accuracy of positioning are improved. The depth distance of the target cable is determined by combining the pixel coordinates and the depth video stream, so as to obtain the three-dimensional spatial information of the target cable and understand the position status of the target cable more comprehensively and accurately. The cable space coordinates are determined according to the pixel coordinates and the depth distance, so that the description of the cable position is more accurate and specific, and a reliable basis is provided for the precise control of the robot controller. The cable positioning is controlled by the robot controller based on the cable space coordinates, so that efficient and accurate cable positioning operation can be realized, human intervention can be reduced, the error probability can be reduced, work efficiency and safety can be improved, and the normal operation and maintenance of the substation equipment can be guaranteed. At the same time, it is also conducive to improving the overall intelligence level and operation and maintenance management quality of the substation.
[0081] Embodiment 3
[0082] Figure 3 This is a schematic diagram of the structure of a cable positioning device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a video stream acquisition module 310, which is used to acquire a video stream of a target location of a substation through a depth camera, wherein the video stream includes a color video stream and a depth video stream;
[0083] A pixel coordinate and depth distance determination module 320, for determining the pixel coordinates of the target cable according to the color video stream, and determining the depth distance of the target cable according to the pixel coordinates and the depth video stream;
[0084] The robot arm cable positioning module 330 is used to determine the cable space coordinates according to the pixel coordinates and the depth distance, and control the robot arm controller to perform cable positioning based on the cable space coordinates.
[0085] Optionally, the video stream acquisition module 310 is specifically used to: take the substation equipment location set by the user as the target location; shoot the color video stream of the target location through the color RGB camera of the depth camera; shoot the depth video stream of the target location through the depth camera of the depth camera.
[0086] Optionally, the pixel coordinate and depth distance determination module 320 specifically includes: a pixel coordinate determination unit, used to: input the color video stream frame by frame into a pre-built cable identification model to obtain a cable identification frame output by the model, wherein the cable identification frame includes a target cable; establish a coordinate system based on the color video stream, and determine the horizontal and vertical coordinates of the center point of the cable identification frame in the coordinate system as the pixel coordinates of the target cable.
[0087] Optionally, the pixel coordinate and depth distance determination module 320 specifically includes: a depth distance determination unit, used to: align the depth video stream and the color video stream to generate an aligned depth video stream; and obtain the depth distance of the pixel coordinate in the aligned depth video stream.
[0088] Optionally, the robot cable positioning module 330 specifically includes: a cable space coordinate determination unit, used to: combine the pixel coordinates and the depth distance to generate the pixel positioning coordinates of the target cable; perform chessboard calibration on the depth camera to obtain the camera intrinsic parameter matrix; multiply the pixel positioning coordinates and the camera intrinsic parameter matrix to generate the cable space coordinates.
[0089] Optionally, the robot cable positioning module 330 specifically includes: a hand-eye calibration unit, used to: perform hand-eye calibration on the robot arm and the calibration camera to determine the coordinate conversion formula; a robot cable positioning unit, used to: substitute the cable space coordinates into the coordinate conversion formula to determine the robot arm control coordinates, and control the robot arm controller according to the robot arm control coordinates.
[0090] Optionally, a hand-eye calibration unit is specifically used to: construct each transformation coordinate system based on the robotic arm and the calibration camera, wherein the transformation coordinate system includes the robotic arm base coordinate system, the robotic arm end coordinate system, the camera coordinate system and the calibration plate coordinate system; generate each coordinate system transformation matrix according to each transformation coordinate system, wherein the coordinate system transformation matrix includes the calibration plate coordinate system to the robotic arm base coordinate system transformation matrix, the calibration plate coordinate system to the camera coordinate system transformation matrix, the camera coordinate system to the robotic arm end coordinate system transformation matrix and the robotic arm end coordinate system to the robotic arm base coordinate system transformation matrix; determine the coordinate conversion formula according to each coordinate system transformation matrix.
[0091] The technical solution of the embodiment of the present invention can obtain the video stream of the target position of the substation through the depth camera, so as to obtain the information of the target area in a comprehensive and real-time manner, and provide a rich data basis for subsequent analysis and processing. The pixel coordinates of the target cable are determined by the color video stream, so that the position of the target cable in the image can be accurately identified and located, and the precision and accuracy of positioning are improved. The depth distance of the target cable is determined by combining the pixel coordinates and the depth video stream, so as to obtain the three-dimensional spatial information of the target cable and understand the position status of the target cable more comprehensively and accurately. The cable space coordinates are determined according to the pixel coordinates and the depth distance, so that the description of the cable position is more accurate and specific, and a reliable basis is provided for the precise control of the robot controller. The cable positioning is controlled by the robot controller based on the cable space coordinates, so that efficient and accurate cable positioning operation can be realized, human intervention can be reduced, the error probability can be reduced, work efficiency and safety can be improved, and the normal operation and maintenance of the substation equipment can be guaranteed. At the same time, it is also conducive to improving the overall intelligence level and operation and maintenance management quality of the substation.
[0092] A cable locating device provided in an embodiment of the present invention can execute a cable locating method provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects of the execution method.
[0093] Embodiment 4
[0094] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0095] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0096] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0097] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a cable positioning method.
[0098] In some embodiments, a cable locating method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of a cable locating method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform a cable locating method in any other suitable manner (e.g., by means of firmware).
[0099] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0100] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0101] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0102] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0103] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0104] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0105] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0106] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A cable positioning method, characterized in that: include: Acquire a video stream of a target location of a substation through a depth camera, wherein the video stream includes a color video stream and a depth video stream; Determine the pixel coordinates of the target cable according to the color video stream, and determine the depth distance of the target cable according to the pixel coordinates and the depth video stream; The cable space coordinates are determined according to the pixel coordinates and the depth distance, and the robot arm controller is controlled based on the cable space coordinates to perform cable positioning.
2. The method according to claim 1, characterized in that The method of obtaining a video stream of a target location of a substation by using a depth camera includes: Taking the substation equipment location set by the user as the target location; Shooting a color video stream of the target position by using a color RGB camera of the depth camera; The depth video stream of the target position is captured by the depth camera of the depth camera.
3. The method according to claim 1, characterized in that Determining the pixel coordinates of the target cable according to the color video stream comprises: Inputting the color video stream frame by frame into a pre-built cable identification model to obtain a cable identification frame output by the model, wherein the cable identification frame includes a target cable; A coordinate system is established based on the color video stream, and the horizontal and vertical coordinates of the center point of the cable identification frame are determined in the coordinate system as the pixel coordinates of the target cable.
4. The method according to claim 3, characterized in that Determining the depth distance of the target cable according to the pixel coordinates and the depth video stream includes: Aligning the depth video stream and the color video stream to generate an aligned depth video stream; Obtain the depth distance of the pixel coordinate in the aligned depth video stream.
5. The method according to claim 1, characterized in that Determining the cable space coordinates according to the pixel coordinates and the depth distance comprises: combining the pixel coordinates and the depth distance to generate pixel location coordinates of the target cable; Performing chessboard calibration on the depth camera to obtain a camera intrinsic parameter matrix; The pixel location coordinates and the camera intrinsic parameter matrix are multiplied to generate the cable space coordinates.
6. The method according to claim 1, characterized in that The method of controlling the robot arm controller to perform cable positioning based on the cable space coordinates includes: Perform hand-eye calibration on the robotic arm and calibration camera to determine the coordinate transformation formula; Substitute the cable space coordinates into the coordinate conversion formula to determine the robot arm control coordinates, and control the robot arm controller according to the robot arm control coordinates.
7. The method according to claim 6, characterized in that The hand-eye calibration of the robotic arm and the calibration camera to determine the coordinate transformation formula includes: Constructing each transformation coordinate system based on the robotic arm and the calibration camera, wherein the transformation coordinate system includes a robotic arm base coordinate system, a robotic arm end coordinate system, a camera coordinate system and a calibration plate coordinate system; Generate each coordinate system transformation matrix according to each of the transformation coordinate systems, wherein the coordinate system transformation matrix includes a transformation matrix from the calibration plate coordinate system to the robot base coordinate system, a transformation matrix from the calibration plate coordinate system to the camera coordinate system, a transformation matrix from the camera coordinate system to the robot end coordinate system, and a transformation matrix from the robot end coordinate system to the robot base coordinate system; A coordinate conversion formula is determined according to each of the coordinate system transformation matrices.
8. A cable positioning device, characterized in that: include: A video stream acquisition module, used to acquire a video stream of a target location of a substation through a depth camera, wherein the video stream includes a color video stream and a depth video stream; A pixel coordinate and depth distance determination module, used to determine the pixel coordinates of the target cable according to the color video stream, and determine the depth distance of the target cable according to the pixel coordinates and the depth video stream; The robot arm cable positioning module is used to determine the cable space coordinates according to the pixel coordinates and the depth distance, and control the robot arm controller to perform cable positioning based on the cable space coordinates.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 7.
10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 7 when executed.
