Method, device, electronic device and storage medium for identifying hidden dangers of external damage to power transmission cables

Abstract

This invention relates to the field of image processing technology, providing a method, apparatus, electronic device, and storage medium for identifying external damage hazards in power transmission cables. The method includes: acquiring two target images of construction machinery and power transmission cables taken from different perspectives; extracting contours from each target image to obtain the position information and image features of each contour point of each target object in each target image, where the target object includes construction machinery or power transmission cables; performing stereo matching on each contour point of the same target object based on the position information and image features of each contour point in the two target images to determine the three-dimensional coordinates of each contour point of the same target object, obtaining the three-dimensional coordinates of all contour points of the construction machinery and all contour points of the power transmission cable, and identifying external damage hazards in the power transmission cable based on these coordinates. This invention improves the accuracy and efficiency of identifying external damage hazards.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for identifying hidden dangers of external damage to a power transmission cable. Background Art

[0002] The key to identifying hidden dangers of external damage to transmission cables lies in the three-dimensional spatial perception of the target object. It is necessary to first accurately locate the three-dimensional spatial coordinates of the target object such as construction machinery and transmission cables, and then estimate the distance between the two based on their three-dimensional spatial coordinates, and then identify the hidden dangers of external damage based on the distance between the two.

[0003] Depth estimation is a key link in calculating three-dimensional space coordinates. Its accuracy and processing efficiency directly affect the accuracy and processing efficiency of three-dimensional space coordinates, and ultimately affect the accuracy and efficiency of identifying external damage hazards.

[0004] Existing binocular depth estimation methods can obtain more accurate depth estimation results by using the disparity information of views captured from different perspectives. However, this method requires complex stereo matching, which results in large computational complexity and low efficiency. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device, electronic device and storage medium for identifying external damage hazards of transmission cables, which can reduce the amount of stereo matching calculations of binocular depth estimation in the external damage hazard identification process while ensuring the accuracy of matching, improve the efficiency and accuracy of binocular depth estimation, and ultimately ensure that the identification efficiency and accuracy of external damage hazards are improved.

[0006] The embodiments of the present invention can be implemented as follows:

[0007] In a first aspect, the present invention provides a method for identifying hidden dangers of external damage to a power transmission cable, the method comprising:

[0008] Acquire two target images of the construction machinery and the power transmission cable taken from different viewing angles;

[0009] Performing contour extraction on each of the target images respectively to obtain position information and image features of each contour point of each target object in each of the target images, wherein the target object includes the construction machinery or the power transmission cable;

[0010] According to the position information and image features of each contour point of the same target object in the two target images, stereo matching is performed on each contour point of the same target object to determine the three-dimensional coordinates of each contour point of the same target object, and the three-dimensional coordinates of all the contour points of the construction machinery and the three-dimensional coordinates of all the contour points of the transmission cable are obtained;

[0011] According to the three-dimensional coordinates of all the contour points of the construction machinery and the three-dimensional coordinates of all the contour points of the transmission cable, the external damage hidden dangers of the transmission cable are identified.

[0012] In an optional embodiment, the step of performing stereo matching on each contour point of the same target object according to the position information and image features of each contour point of the same target object in the two target images to determine the three-dimensional coordinates of each contour point of the same target object comprises:

[0013] For each of the target objects, use any one of the two target images as a main view, use the other one as a secondary view, and use contour points of the target object in the main view and the secondary view as first contour points and second contour points, respectively;

[0014] For any target first contour point among all the first contour points of the target object, determine each target second contour point on the same horizontal polar line as the target first contour point from all the second contour points;

[0015] Calculating each matching cost according to the position information and image features of the target first contour point and each target second contour point;

[0016] Determine the disparity of the target first contour point according to the target first contour point and the target second contour point corresponding to the minimum matching cost among all the matching costs;

[0017] Calculating the three-dimensional coordinates of the first contour point of the target according to the parallax of the first contour point of the target;

[0018] All the first contour points of the target object are traversed to finally obtain the three-dimensional coordinates of each contour point of the target object.

[0019] In an optional implementation, the step of calculating each matching cost according to the position information and image features of the target first contour point and each target second contour point includes:

[0020] Calculating each initial cost according to the position information of the target first contour point and each target second contour point;

[0021] Calculating a weight of each initial cost according to the image features of the target first contour point and each target second contour point;

[0022] According to each initial cost and its weight, each matching cost is calculated.

[0023] In an optional embodiment, there are multiple transmission cables, and the step of identifying external damage hazards of the transmission cables according to the three-dimensional coordinates of all the contour points of the construction machinery and the three-dimensional coordinates of all the contour points of the transmission cables includes:

[0024] Determine the target transmission cable closest to the construction machine according to the three-dimensional coordinates of all the contour points of the construction machine and the three-dimensional coordinates of all the contour points of each of the transmission cables;

[0025] Calculating the distance between the three-dimensional coordinates of each of the contour points of the construction machinery and the target transmission cable to obtain a plurality of point-line distances;

[0026] Using the minimum distance among all the point-line distances as the reference distance between the construction machine and the target transmission cable;

[0027] The target power transmission cable is identified for potential external damage according to the reference distance.

[0028] In an optional embodiment, the step of determining the target transmission cable closest to the construction machine according to the three-dimensional coordinates of all the contour points of the construction machine and the three-dimensional coordinates of all the contour points of each of the transmission cables comprises:

[0029] Calculating the centroid of the construction machinery according to the three-dimensional coordinates of all contour points of the construction machinery;

[0030] According to the three-dimensional coordinates of all the contour points of each of the transmission cables, a fitting line of each of the transmission cables projected on a two-dimensional plane is obtained;

[0031] Calculating the relative distance between the construction machine and each of the transmission cables according to the centroid of the construction machine and the fitting line of each of the transmission cables;

[0032] The power transmission cable corresponding to the smallest distance among the plurality of relative distances is taken as the target power transmission cable.

[0033] In an optional embodiment, the step of obtaining a fitting line projected on a two-dimensional plane by each transmission cable according to the three-dimensional coordinates of all the contour points of each transmission cable comprises:

[0034] Performing plane fitting according to the three-dimensional coordinates of all the contour points of each of the transmission cables to obtain a two-dimensional plane where each of the transmission cables is located;

[0035] Projecting each of the contour points of each of the transmission cables onto the two-dimensional plane to obtain each projection coordinate of each of the contour points of each of the transmission cables in the two-dimensional plane;

[0036] Calculating a correlation coefficient representing a linear correlation between the projected coordinates of all the contour points of each of the transmission cables;

[0037] Line fitting is performed according to the correlation coefficient and the projection coordinates of all the contour points of each of the transmission cables to obtain a fitting line of each of the transmission cables projected on a two-dimensional plane.

[0038] In an optional implementation manner, the step of performing line fitting according to the correlation coefficient and the projection coordinates of all the contour points of each of the transmission cables to obtain a fitting line projected on a two-dimensional plane by each of the transmission cables comprises:

[0039] If the correlation coefficient is greater than a preset threshold, a straight line fitting is performed on the projection coordinates to obtain a fitting straight line and the fitting straight line is used as the fitting line;

[0040] If the correlation coefficient is less than or equal to the preset threshold, a quadratic polynomial fitting is performed on the projection coordinates to obtain a fitting curve and the fitting curve is used as the fitting line.

[0041] In a second aspect, the present invention provides a device for identifying hidden dangers of external damage to a power transmission cable, the device comprising:

[0042] An acquisition module is used to acquire two target images obtained by photographing the construction machinery and the power transmission cable from different viewing angles;

[0043] An extraction module, used to extract contours of each target image respectively, to obtain position information and image features of each contour point of each target object in each target image, wherein the target object includes the construction machinery or the transmission cable;

[0044] A matching module, for performing stereo matching on each contour point of the same target object according to the position information and image features of each contour point of the same target object in the two target images, determining the three-dimensional coordinates of each contour point of the same target object, and obtaining the three-dimensional coordinates of all the contour points of the construction machinery and the three-dimensional coordinates of all the contour points of the transmission cable;

[0045] The identification module is used to identify external damage hazards of the transmission cable based on the three-dimensional coordinates of all the contour points of the construction machinery and the three-dimensional coordinates of all the contour points of the transmission cable.

[0046] In a third aspect, the present invention provides an electronic device, including a processor and a memory, wherein the memory is used to store a program, and the processor is used to implement a method for identifying hidden dangers of external damage to a transmission cable as described in any of the aforementioned embodiments when executing the program.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for identifying hidden dangers of external damage to a transmission cable as described in any one of the aforementioned embodiments.

[0048] Compared with the prior art, the present invention has the following beneficial effects: when matching the same target object in two target images taken from different perspectives, the present invention uses the contour points of the target object in the two target images for matching, thereby reducing the amount of matching data and improving the matching efficiency. At the same time, the position information and image information are used for matching to ensure the accuracy of the matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is an example flowchart of the method for identifying hidden dangers of external damage to power transmission cables provided in this embodiment.

[0051] Figure 2 This is an example diagram of the overall process of the method for identifying hidden dangers of external damage to transmission cables provided in this embodiment.

[0052] Figure 3 This is a block diagram of an example of a device for identifying hidden dangers of external damage to a transmission cable provided in this embodiment.

[0053] Figure 4 An example block diagram of an electronic device provided in this embodiment.

[0054] Icons: 10 - electronic device; 11 - processor; 12 - memory; 13 - bus; 100 - device for identifying hidden dangers of external damage to power transmission cables; 110 - acquisition module; 120 - extraction module; 130 - matching module; 140 - identification module. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0058] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, they are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the invention is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0059] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0060] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0061] Please refer to Figure 1 , Figure 1 The following is a flow chart of a method for identifying a potential risk of external damage to a power transmission cable provided in this embodiment. The method includes the following steps:

[0062] Step S101, acquiring two target images of a construction machine and a power transmission cable respectively taken from different viewing angles.

[0063] In this embodiment, the construction machinery includes but is not limited to excavators, cranes, bulldozers, loaders, etc. The power transmission cable can be a cable hung in the air within the working range of the construction site. The different viewing angles can be left and right viewing angles, or upper and lower viewing angles, etc., as long as the construction machinery and power transmission cables can be clearly and completely photographed from different viewing angles.

[0064] Step S102 , respectively extracting contours of each target image to obtain position information and image features of each contour point of each target object in each target image, wherein the target objects include construction machinery or power transmission cables.

[0065] In this embodiment, the contour extraction method for each construction machine and each transmission cable is the same. The position information can be the pixel coordinates of the contour point in the image coordinate system, and the image feature can be the spatial semantic feature information of the target object in the target image, which can be obtained by first extracting the features of the target object to obtain a feature vector, then extracting a multi-scale feature vector of the contour point position from the feature vector, and obtaining the vector after normalizing and merging the multi-scale feature vectors.

[0066] In this embodiment, in order to obtain the contour of each target object in each target image, each target object in each target image can be segmented using a pre-trained instance segmentation model to obtain a segmentation result for each target object, and then the position information of each contour point of each target object is extracted from the segmentation result. At the same time, during the operation of the instance segmentation model, feature vectors of multiple different scale features of the target image will be obtained, and the image features at each contour point can be obtained based on the multi-scale feature vectors at the position information of each contour point.

[0067] It should be noted that, in a scenario where it can be determined that there is only one target object of the same category in the target image, a semantic segmentation model can also be used to segment the target object and extract multi-scale features.

[0068] In this embodiment, in order to facilitate stereo matching of the same target object in two target images, binocular calibration and stereo correction can be performed before segmentation of the target object using the instance segmentation model. In this embodiment, the images taken from left and right perspectives by a binocular camera including left and right cameras are used as an example to illustrate the implementation of calibration and correction:

[0069] First, binocular positioning is performed on the binocular cameras in the stereo vision system to obtain the binocular positioning parameters of the left and right cameras, which include the intrinsic parameter matrix K of the left and right cameras. L ,K R And the external parameter matrix R L ,R R ,t L ,t R , and distortion D L ,D R .

[0070] In this embodiment, the intrinsic parameter matrix describes the influence of the internal parameters of the camera on the image formation process, and these parameters do not change with the change of the camera position or viewing angle. The intrinsic parameter matrix generally contains the following elements: Focal Length: Indicates the degree to which the lens converges light, and the commonly used unit is pixel. It reflects the magnification of the camera lens. Principal Point: The coordinates of the center point of the image in the pixel coordinate system. Ideally, this point should be located at the center of the image. Distortion Coefficients: Used to correct lens distortion, such as barrel distortion or pincushion distortion. When shooting with an actual camera, various distortions will occur due to the limitations of the lens manufacturing process, which need to be compensated by these coefficients.

[0071] In this embodiment, the extrinsic parameter matrix describes the position and posture of the camera relative to the world coordinate system. It defines the rotation and translation of the camera. The extrinsic parameter matrix is ​​usually used to transform an object from the world coordinate system to the camera coordinate system, or vice versa.

[0072] Secondly, the binocular camera system is stereo calibrated. The optical center position is obtained and the optical center distance d is calculated through the relative posture relationship between the left and right cameras. The intrinsic and extrinsic parameters of the virtual camera are determined, and then the homography matrix of the left and right cameras relative to the virtual camera is determined.

[0073] In this embodiment, the homography matrix can be used to describe the projection relationship between two planes, that is, how to map points on one plane to another plane. For example, the homography matrix of the left camera relative to the virtual camera is used to describe the projection relationship between points on the image taken by the left camera and points on the image taken by the virtual camera.

[0074] Finally, the binocular positioning parameters and homography matrix can be pre-calibrated in an offline manner. After obtaining the calibrated binocular positioning parameters and homography matrix, in practical applications, the original images taken by the left and right cameras are firstly subjected to distortion correction according to the pre-calibrated binocular positioning parameters of the left and right cameras, and then stereo correction (also called epipolar correction) is performed on the distortion-corrected original images of the left and right cameras according to the homography matrix of the left and right cameras relative to the virtual camera to obtain two target images.

[0075] In this embodiment, distortion correction refers to correcting image distortion caused by the camera lens, while stereo correction refers to aligning images from different perspectives for stereo matching and 3D reconstruction. Performing distortion correction on the target image first and then stereo correction can further improve the accuracy and efficiency of stereo matching, thereby improving the accuracy and efficiency of external damage hidden danger identification.

[0076] After obtaining two target images, the instance segmentation model is first used to perform instance segmentation on each target object in each target image. For each target object, the instance segmentation result includes the category, mask and target constraint box of the target object. The category is used to characterize whether the target object is a construction machine or a transmission cable. The mask represents the binary segmentation map of the target object. The pixel value 0 represents the background and 1 represents the target object. The target constraint box represents the minimum bounding rectangle of the target object, which can be represented by the pixel coordinates of the two vertices of the upper left corner and the lower right corner of the minimum bounding rectangle. In order to ensure the segmentation effect, the instance segmentation models can be trained for construction machinery and transmission cables respectively to obtain the segmentation results of construction machinery and transmission cables for each target image.

[0077] Step S103, based on the position information and image features of each contour point of the same target object in the two target images, stereo matching is performed on each contour point of the same target object, and the three-dimensional coordinates of each contour point of the same target object are determined, thereby obtaining the three-dimensional coordinates of all contour points of the construction machinery and the three-dimensional coordinates of all contour points of the transmission cables.

[0078] In this embodiment, in order to determine the same target object in the two target images, it is necessary to match each target object in the two target images. To this end, first, one of the target images is used as the main view and the other target image is used as the slave view, and the reprojection matrix from the slave view to the main view is calculated. Then, for each target object in the slave view, it is projected to the main view. The specific method can be to project the coordinates of its constraint box to the main view to obtain its coordinate vector s1 in the main view. Next, s1 and the original target objects in the main view are segmented to obtain the coordinate vectors t1~tn of the constraint box for matching, where n is the number of target objects. The specific matching method is to calculate the L2 norm between each coordinate vector from s1 to t1~tn, and obtain the target object in the slave view corresponding to s1 according to the nearest neighbor matching algorithm. Match the target object corresponding to the nearest coordinate vector in t1~tn, thereby obtaining a matching pair of segmentation results of the same target object in the two target images, each matching pair includes the segmentation results of the left and right views, and the two respectively represent the detection results of the same target object in the left and right perspectives in the world coordinate system. Finally, a matching pair of segmentation results of the construction machinery and a matching pair of segmentation results of the same transmission cable are obtained.

[0079] In this embodiment, stereo matching is used to recover the depth information of the scene from one or more pairs of stereo images. By comparing images taken from different perspectives, the stereo matching algorithm can find corresponding points and calculate the disparity map based on the differences between these corresponding points. In stereo matching, this embodiment avoids using all pixel points in the same target object coverage area, but only uses the contour points of the target object, thereby greatly reducing the amount of stereo matching data and improving the efficiency of stereo matching.

[0080] In this embodiment, the disparity of each contour point of the same target object is obtained through stereo matching, and the depth of each contour point can be calculated according to the disparity of each contour point, and the three-dimensional coordinates of each contour point can be calculated according to the depth of each contour point. Each target object is either a construction machine or a transmission cable, and finally the three-dimensional coordinates of all contour points of the construction machine and the three-dimensional coordinates of all contour points of the transmission cable can be obtained.

[0081] Step S104, identifying external damage hazards of the transmission cables based on the three-dimensional coordinates of all contour points of the construction machinery and the three-dimensional coordinates of all contour points of the transmission cables.

[0082] The above method provided in this embodiment uses the contour points of the target object in the two target images for matching when matching the same target object in two target images taken from different perspectives, thereby reducing the amount of matching data and improving matching efficiency. At the same time, it uses position information and image information for matching to ensure the accuracy of matching.

[0083] In this embodiment, in order to more efficiently obtain the contour points at the same position of the same target object in two target images, and then perform accurate stereo matching based on the information of the contour points at the same position to obtain accurate three-dimensional coordinates of each contour point of the same target object, this embodiment provides an implementation method:

[0084] Firstly, for each target object, any one of the two target images is used as the main view, the other one is used as the secondary view, and the contour points of the target object in the main view and the secondary view are used as the first contour point and the second contour point respectively;

[0085] In this embodiment, taking the target images as a left view and a right view, the left view can be used as the main view and the right view as the slave view, or the right view can be used as the main view and the left view as the slave view. This embodiment is not limited to this.

[0086] Secondly, for any target first contour point among all first contour points of the target object, each target second contour point on the same horizontal polar line as the target first contour point is determined from all second contour points;

[0087] In this embodiment, the processing method for each contour point of all the first contour points of the target object is the same, and this embodiment is described using any one of the first contour points of the target object.

[0088] If the two target images have been subjected to distortion correction and stereo correction before, the method for determining each target second contour point of the target first contour point here can be: taking the contour point whose horizontal axis coordinate in the second contour point matches the horizontal axis coordinate of the target first contour point as the target second contour point, and the horizontal axis coordinate matching can be that the horizontal axis coordinates are the same or the difference in the horizontal axis coordinates is less than a preset difference.

[0089] If the two target images have not been subjected to distortion correction and stereo correction before, the method for determining each target second contour point of the target first contour point here can be: calculating the epipolar line of the target first contour point, and then determining the target second contour point based on the intersection of the epipolar line and the target image to which the second contour point belongs.

[0090] In this embodiment, the stereo matching algorithm can be based on the SGBM (Semi-Global Block Matching) algorithm. When performing stereo matching, only contour points are matched, and non-contour points do not participate in the matching. This method is called contour constraint. The method of using horizontal polar lines for matching during stereo matching is called polar line constraint. The contour constraint and polar line constraint can greatly reduce the amount of matching data, reducing the space complexity and time complexity.

[0091] It should be noted that in addition to the SGBM algorithm, the DP (Dynamic Programming) algorithm, BP (Belief Propagation) and the like can also be used.

[0092] Third, each matching cost is calculated based on the position information and image features of the target first contour point and each target second contour point;

[0093] In this embodiment, the matching cost is used to characterize the degree of matching between two contour points. The greater the matching cost, the smaller the probability that the two contour points match, and vice versa. In order to make full use of the position information and image features of the target first contour point and each target second contour point to improve the accuracy of matching, this embodiment provides an implementation method: calculate each initial cost based on the position information of the target first contour point and each target second contour point; calculate the weight of each initial cost based on the image features of the target first contour point and each target second contour point; calculate each matching cost based on each initial cost and its weight. For the target first contour point and any target second contour point, the matching cost can be calculated using the following formula:

[0094] C' (p,q)=<F(p),F(q)> ·C(p, q), where C' represents the matching cost, C represents the initial cost, F represents the spatial semantic representation, i.e., the image feature, p and q represent the position coordinates of the first contour point of the target and the second contour point of the target respectively, <*,*> represents the vector inner product, and · represents multiplication.

[0095] Fourth, determining the disparity of the first contour point of the target according to the first contour point of the target and the second contour point of the target corresponding to the minimum matching cost among all matching costs;

[0096] In this embodiment, the target second contour point with the smallest matching cost means the best match with the target first contour point, and the disparity of the target first contour point determined thereby is more accurate.

[0097] Fifth, according to the parallax of the first contour point of the target, the three-dimensional coordinates of the first contour point of the target are calculated;

[0098] In this embodiment, the disparity represents the horizontal displacement of the same physical point in two images of different viewing angles, and the depth information of the physical point can be calculated through the disparity. The disparity of the first target contour point is the disparity of the physical point corresponding to the first target contour point, and the three-dimensional coordinates of the first target contour point are the disparity of the contour point of the target object corresponding to the first target contour point. It should be noted that the contour point of the target object corresponding to the first target contour point and the most matching second target contour point are the same.

[0099] The method of calculating the three-dimensional coordinates of the first contour point of the target may be: firstly, according to the parallax of the first contour point of the target, the depth of the contour point of the target object corresponding to the first contour point of the target may be calculated using the calculation formula: D = d·f L / S, where d is the distance from the optical center, f L is the focal length of the corrected main view camera, and S is the parallax of the first contour point of the target. Then, according to the pixel coordinates and depth of the first contour point of the target, as well as the camera internal parameters, the contour points corresponding to the first contour point of the target object are reconstructed in three dimensions by inverse projection to obtain the three-dimensional coordinates of the first contour point of the target, that is, the three-dimensional coordinates of the contour point of the target object corresponding to the first contour point of the target.

[0100] Sixth, all first contour points of the target object are traversed to finally obtain the three-dimensional coordinates of each contour point of the target object.

[0101] In this embodiment, after obtaining the three-dimensional coordinates of all contour points of the construction machinery and the three-dimensional coordinates of all contour points of the transmission cables, when there are multiple transmission cables, in order to identify the hidden danger of external damage as quickly as possible, the target transmission cable closest to the construction machinery can be first identified, and then whether there is a hidden danger of external damage can be determined based on the distance between the two. This embodiment provides an implementation method:

[0102] (1) Determine the target transmission cable closest to the construction machinery based on the three-dimensional coordinates of all contour points of the construction machinery and the three-dimensional coordinates of all contour points of each transmission cable;

[0103] In this embodiment, the target transmission cable closest to the construction machinery can be identified by roughly estimating the distance between the construction machinery and each transmission cable. Since the outline shape of the construction machinery is not a regular geometric shape, the construction machinery is represented by the centroid of the construction machinery, and the transmission cable can be fitted into a fitting line. The distance between the centroid and the fitting line of each transmission cable is used to represent the distance between the construction machinery and the transmission cable, thereby screening out the target transmission cable closest to the construction machinery. One way to determine the target transmission cable is:

[0104] Firstly, the centroid of the construction machinery is calculated according to the three-dimensional coordinates of all the contour points of the construction machinery;

[0105] In this embodiment, the centroid refers to a special point inside a plane figure or a three-dimensional figure, which can be regarded as the "average position" of the figure. Since the geometric shape formed by the three-dimensional coordinates of all contour points of the construction machinery is usually irregular, the approximate centroid of the construction machinery can be obtained through geometric calculation.

[0106] Secondly, according to the three-dimensional coordinates of all contour points of each transmission cable, the fitting line of each transmission cable projected on the two-dimensional plane is obtained;

[0107] In this embodiment, the fitting line represents the approximate shape of the transmission cable on the plane. In practical applications, the transmission cable may be fitted into a straight line or a curve. This embodiment provides a method for obtaining the fitting line: plane fitting is performed according to the three-dimensional coordinates of all contour points of each transmission cable to obtain the two-dimensional plane where each transmission cable is located; each contour point of each transmission cable is projected onto the two-dimensional plane to obtain each projection coordinate of each contour point of each transmission cable in the two-dimensional plane; the correlation coefficient representing the linear correlation between the projection coordinates of all contour points of each transmission cable is calculated; line fitting is performed according to the correlation coefficient and the projection coordinates of all contour points of each transmission cable to obtain the fitting line of each transmission cable projected onto the two-dimensional plane.

[0108] In this embodiment, the plane fitting method includes, but is not limited to, RANSAC robust fitting algorithm (RANdomSAmple Consensus), MSAC (M-estimator Sample Consensus), etc. The representation of the two-dimensional plane can be: Ax+By+Cz+D=0, where ABCD is the coefficient and xyz is the three-dimensional coordinate variable.

[0109] In this embodiment, different values ​​of the correlation coefficient characterize different degrees of linear correlation between the projection coordinates of all contour points of the transmission cable. If the correlation coefficient characterizes linear correlation, it means that the fitting line of the transmission cable on the two-dimensional plane is closer to a straight line. Otherwise, it means that the fitting line of the transmission cable on the two-dimensional plane is closer to a curve. Since some transmission cables are closer to a straight line and some are closer to a curve, in order to fit them more in line with actual scenarios, different fitting methods are performed for different situations, and the obtained fitting line can be closer to the actual transmission cable. One fitting method for different situations is: if the correlation coefficient is greater than a preset threshold, a straight line fitting is performed on the projection coordinates to obtain a fitting straight line and the fitting straight line is used as the fitting line; if the correlation coefficient is less than or equal to the preset threshold, a quadratic polynomial fitting is performed on the projection coordinates to obtain a fitting curve and the fitting curve is used as the fitting line.

[0110] In this embodiment, the preset threshold can be set according to the needs of the actual scene, for example, the preset threshold is 0.98. If the correlation coefficient is greater than 0.98, the RANSAC robust fitting algorithm can be used for straight line fitting to obtain a fitting line, otherwise, a quadratic parabola fitting is used to obtain a quadratic polynomial curve representation.

[0111] Third, the relative distance between the construction machine and each transmission cable is calculated based on the centroid of the construction machine and the fitting line of each transmission cable;

[0112] In this embodiment, the relative distance between the construction machinery and each transmission cable is calculated in the same way. The relative distance between the construction machinery and any transmission cable can be calculated as follows: plane fitting is performed based on the three-dimensional coordinates of all contour points of each transmission cable to obtain a two-dimensional plane where each transmission cable is located, and the distance d1 from the centroid to the two-dimensional plane is calculated; the centroid O of the construction machinery is projected on the two-dimensional plane to obtain the projected centroid O' of the centroid, and the distance d2 from O' to the transmission cable in the two-dimensional plane is calculated. If the fitted line of the transmission cable is a straight line, the distance from point O' to the fitted line can be directly used to obtain d2. If the fitted line of the transmission cable is a curve, a distance equation can be established, and the pole coordinates can be obtained by solving a cubic equation, and then d2 is calculated; based on d1 and d2, the relative distance d3 is calculated using the Pythagorean theorem.

[0113] Finally, the transmission cable corresponding to the minimum distance among the multiple relative distances is taken as the target transmission cable.

[0114] (2) Calculate the distance between the three-dimensional coordinates of each contour point of the construction machinery and the target transmission cable to obtain multiple point-line distances;

[0115] In this embodiment, all contour points of the construction machinery are traversed, and for one contour point, the distance between the three-dimensional coordinates of the contour point and the target transmission cable is calculated to obtain a point-line distance, and finally multiple point-line distances are obtained. For one contour point, a point-line distance is calculated as follows: plane fitting is performed according to the three-dimensional coordinates of all contour points of the target transmission cable to obtain the target two-dimensional plane where the target transmission cable is located, and the distance d1' from the contour point a to the target two-dimensional plane is calculated; the contour point a is projected on the target two-dimensional plane to obtain the projection a' of the contour point a, and the distance d2' from a' to the target transmission cable in the target two-dimensional plane is calculated. If the fitting line of the target transmission cable is a straight line, the distance from point a' to the fitting line can be directly used to obtain d2'. If the fitting line of the target transmission cable is a curve, a distance equation can be established, and its pole coordinates are obtained by solving a cubic equation, and then d2' is calculated; based on d1' and d2', the point-line Euclidean distance d3' is calculated.

[0116] (3) The minimum distance among all point-line distances is used as the reference distance between the construction machinery and the target transmission cable;

[0117] (4) Identify external damage risks of target transmission cables based on the reference distance.

[0118] In this embodiment, external damage hazard identification can be performed according to the reference distance by setting an external damage hazard threshold. For example, when the reference distance is greater than the external damage hazard threshold, it is determined that there is an external damage hazard risk; otherwise, it is determined that there is no external damage hazard risk.

[0119] It should be noted that when it is determined that there is a risk of external damage, an alarm message can be sent or displayed, which may include the endpoint coordinates of the target transmission cable and information about the threatening construction machinery (including but not limited to the instance category, confidence, contour, constraint box, etc. of the construction machinery).

[0120] In order to demonstrate the whole process of identifying the hidden danger of external damage of power transmission cables, this embodiment provides a specific example of the whole process. Figure 2 , Figure 2 This is an example diagram of the overall process of the method for identifying hidden dangers of external damage to power transmission cables provided in this embodiment. Figure 2 In the example, the left view is used as the main view and the right view is used as the slave view. The processing flow is as follows:

[0121] (1) Perform binocular localization and stereo correction based on the left view and the right view to obtain a corrected left view and a corrected right view;

[0122] (2) Perform instance segmentation on the corrected left view and the corrected right view, including construction machinery instance segmentation and power transmission cable instance segmentation, to obtain left view target instances and right view target instances;

[0123] (3) Match the two instances;

[0124] (4) For the matched instances, fast binocular stereo matching is performed based on the contour features to obtain a target contour depth map, wherein each instance has a target contour depth map, and the target contour depth map includes a depth value of each contour point of the corresponding instance;

[0125] (5) Reconstruct the target contour 3D based on the instance matching results and the target contour depth map;

[0126] (6) Fitting the transmission cable based on the result of the three-dimensional reconstruction, wherein the fitting of the transmission cable includes the plane fitting where the transmission cable is located and the line fitting of the transmission cable;

[0127] (7) The closest distance between the construction machinery and the transmission cable is measured based on the fitting results, and finally the external damage hazards between the construction machinery and the transmission cable are identified based on the closest distance measurement results. The closest distance measurement between the construction machinery and the transmission cable includes first matching the nearest neighbor transmission cable of the construction machinery, and then measuring the distance between the outline of the construction machinery and the nearest neighbor transmission cable.

[0128] In order to execute the corresponding steps in the above embodiment and each possible implementation method, the following provides a method for implementing a device 100 for identifying hidden dangers of external damage to a power transmission cable. Figure 3 , Figure 3 This is a block diagram of the device for identifying hidden dangers of external damage of a transmission cable provided in this embodiment. It should be noted that the basic principles and technical effects of the device for identifying hidden dangers of external damage of a transmission cable 100 provided by the present invention are the same as those of the corresponding above-mentioned embodiments, and are not mentioned in this embodiment for the sake of brief description.

[0129] The transmission cable external damage hidden danger identification device 100 includes an acquisition module 110 , an extraction module 120 , a matching module 130 and an identification module 140 .

[0130] An acquisition module 110 is used to acquire two target images obtained by photographing the construction machinery and the power transmission cable from different viewing angles;

[0131] An extraction module 120 is used to extract contours of each target image to obtain position information and image features of each contour point of each target object in each target image, wherein the target object includes a construction machine or a transmission cable;

[0132] The matching module 130 is used to perform stereo matching on each contour point of the same target object according to the position information and image features of each contour point of the same target object in the two target images, determine the three-dimensional coordinates of each contour point of the same target object, and obtain the three-dimensional coordinates of all contour points of the construction machinery and the three-dimensional coordinates of all contour points of the transmission cable;

[0133] The identification module 140 is used to identify external damage hazards of the transmission cable based on the three-dimensional coordinates of all contour points of the construction machinery and the three-dimensional coordinates of all contour points of the transmission cable.

[0134] In an optional implementation manner, the matching module 130 is specifically configured to:

[0135] For each target object, any one of the two target images is used as a main view, the other one is used as a secondary view, and the contour points of the target object in the main view and the secondary view are used as the first contour point and the second contour point respectively;

[0136] For any target first contour point among all first contour points of the target object, determine each target second contour point on the same horizontal polar line as the target first contour point from all second contour points;

[0137] Calculate each matching cost according to the position information and image features of the target first contour point and each target second contour point;

[0138] Determine the disparity of the first contour point of the target according to the first contour point of the target and the second contour point of the target corresponding to the minimum matching cost among all matching costs;

[0139] Calculate the three-dimensional coordinates of the first contour point of the target according to the parallax of the first contour point of the target;

[0140] All first contour points of the target object are traversed, and finally the three-dimensional coordinates of each contour point of the target object are obtained.

[0141] In an optional implementation manner, the matching module 130 is used to calculate each matching cost according to the position information and image features of the target first contour point and each target second contour point:

[0142] Calculate each initial cost according to the position information of the target first contour point and each target second contour point;

[0143] Calculate the weight of each initial cost according to the image features of the target first contour point and each target second contour point;

[0144] Based on each initial cost and its weight, each matching cost is calculated.

[0145] In an optional embodiment, there are multiple transmission cables, and the identification module 140 is used to identify the external damage hidden dangers of the transmission cables according to the three-dimensional coordinates of all contour points of the construction machinery and the three-dimensional coordinates of all contour points of the transmission cables, and is specifically used to:

[0146] Determine the target transmission cable closest to the construction machinery based on the three-dimensional coordinates of all contour points of the construction machinery and the three-dimensional coordinates of all contour points of each transmission cable;

[0147] Calculate the distance between the three-dimensional coordinates of each contour point of the construction machinery and the target transmission cable to obtain multiple point-line distances;

[0148] The minimum distance among all point-line distances is used as the reference distance between the construction machinery and the target transmission cable;

[0149] Identify external damage risks of target transmission cables based on reference distance.

[0150] In an optional embodiment, the identification module 140 is used to determine the target transmission cable closest to the construction machine according to the three-dimensional coordinates of all contour points of the construction machine and the three-dimensional coordinates of all contour points of each transmission cable, and is specifically used to:

[0151] Calculate the centroid of the construction machinery according to the three-dimensional coordinates of all contour points of the construction machinery;

[0152] According to the three-dimensional coordinates of all contour points of each transmission cable, a fitting line of each transmission cable projected on a two-dimensional plane is obtained;

[0153] Calculate the relative distance between the construction machine and each transmission cable based on the centroid of the construction machine and the fitting line of each transmission cable;

[0154] The transmission cable corresponding to the minimum distance among the multiple relative distances is taken as the target transmission cable.

[0155] In an optional embodiment, the identification module 140 is used to obtain a fitting line of each transmission cable projected on a two-dimensional plane according to the three-dimensional coordinates of all contour points of each transmission cable, specifically for:

[0156] Perform plane fitting according to the three-dimensional coordinates of all contour points of each transmission cable to obtain a two-dimensional plane where each transmission cable is located;

[0157] Projecting each contour point of each transmission cable onto a two-dimensional plane to obtain each projection coordinate of each contour point of each transmission cable in the two-dimensional plane;

[0158] Calculate the correlation coefficient that characterizes the linear correlation between the projected coordinates of all contour points of each transmission cable;

[0159] Line fitting is performed according to the correlation coefficient and the projection coordinates of all contour points of each transmission cable to obtain a fitting line of each transmission cable projected on a two-dimensional plane.

[0160] In an optional embodiment, the identification module 140 is used to perform line fitting according to the correlation coefficient and the projection coordinates of all contour points of each transmission cable to obtain a fitting line of each transmission cable projected on a two-dimensional plane, specifically for:

[0161] If the correlation coefficient is greater than a preset threshold, a straight line is fitted to the projection coordinates to obtain a fitting line and the fitting line is used as the fitting line;

[0162] If the correlation coefficient is less than or equal to a preset threshold, a quadratic polynomial is fitted to the projection coordinates to obtain a fitting curve and the fitting curve is used as the fitting line.

[0163] The embodiment of the present invention also provides a block diagram of an electronic device 10, which implements the method for identifying hidden dangers of power transmission cable damage in the above embodiment. Figure 4 , Figure 4 This is a block diagram of an electronic device 10 provided in this embodiment. The electronic device 10 includes a processor 11, a memory 12 and a bus 13. The processor 11 and the memory 12 are connected via the bus 13.

[0164] The processor 11 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the transmission cable external damage hidden danger identification method of the above embodiment may be completed by the hardware integrated logic circuit or software instructions in the processor 11. The above processor 11 may be a general-purpose processor, including a CPU (Central Processing Unit), NP (Network Processor), etc.; it may also be a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Logic Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0165] The memory 12 is used to store a program for implementing a method for identifying hidden dangers of external damage to a power transmission cable. The program may be a software function module stored in the memory 12 in the form of software or firmware or solidified in the OS (Operating System) of the electronic device 10 .

[0166] After receiving the execution instruction, the processor 11 executes the program to implement the method for identifying hidden dangers of external damage to a power transmission cable according to the aforementioned embodiment.

[0167] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for identifying hidden dangers of external damage to a power transmission cable as described in the aforementioned embodiment is implemented.

[0168] In summary, an embodiment of the present invention provides a method, device, electronic device and storage medium for identifying hidden dangers of external damage of a transmission cable, the method comprising: obtaining two target images obtained by photographing a construction machine and a transmission cable from different perspectives; performing contour extraction on each target image to obtain position information and image features of each contour point of each target object in each target image, wherein the target objects include construction machinery or transmission cables; performing stereo matching on each contour point of the same target object based on the position information and image features of each contour point of the same target object in the two target images to determine the three-dimensional coordinates of each contour point of the same target object, and obtain the three-dimensional coordinates of all contour points of the construction machine and the three-dimensional coordinates of all contour points of the transmission cable; identifying hidden dangers of external damage of the transmission cable based on the three-dimensional coordinates of all contour points of the construction machine and the three-dimensional coordinates of all contour points of the transmission cable. Compared with the prior art, this embodiment has at least the following advantages: (1) The contour points of the target object in the two target images are used for matching, which reduces the amount of matching data and improves the matching efficiency. At the same time, the position information and image information are used for matching to ensure the accuracy of the matching; (2) Three-dimensional space perception is realized based on binocular vision, and an instance segmentation algorithm is adopted to take into account the instance information of target detection (including but not limited to category, contour, constraint box) and the geometric information of semantic segmentation (including but not limited to straight line fitting, curve fitting, centroid, etc.), thereby realizing accurate and efficient three-dimensional space perception; (3) Fast stereo matching is realized through contour constraints and epipolar constraints, and sparse depth estimation and three-dimensional reconstruction are performed only on the contour points of the instance of the target object, which can simultaneously reduce the time complexity and space complexity. Complexity, to meet the requirements of real-time and lightweight applications; (4) In the matching cost calculation link of stereo matching, the multi-scale feature vector of the instance segmentation model is introduced as the high-dimensional spatial semantic representation of the target point to improve the matching accuracy; (5) The distance measurement is decomposed into two steps: determining the nearest neighbor matching of the target transmission line and the distance measurement with the matched target transmission line. The shortest distance calculation between the target point and the spatial line is realized based on spatial analytic geometry, thereby realizing a distance measurement algorithm with linear time complexity; (6) Different fitting methods are used for fitting of transmission lines with different characteristics to ensure the accuracy of fitting and ultimately improve the accuracy of external threat identification; (7) The RANSAC algorithm is used to fit the line representation, which effectively handles the noise area of ​​the input and improves the robustness of the algorithm.

[0169] The above are only various embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for identifying hidden dangers of external damage to power transmission cables, characterized in that: The method comprises: Acquire two target images obtained by photographing a construction machine and a power transmission cable from different viewing angles, wherein there are multiple power transmission cables; Performing contour extraction on each of the target images respectively to obtain position information and image features of each contour point of each target object in each of the target images, wherein the target object includes the construction machinery or the power transmission cable; According to the position information and image features of each contour point of the same target object in the two target images, stereo matching is performed on each contour point of the same target object to determine the three-dimensional coordinates of each contour point of the same target object, and the three-dimensional coordinates of all the contour points of the construction machinery and the three-dimensional coordinates of all the contour points of the transmission cable are obtained; Calculating the centroid of the construction machinery according to the three-dimensional coordinates of all contour points of the construction machinery; According to the three-dimensional coordinates of all the contour points of each of the transmission cables, a fitting line of each of the transmission cables projected on a two-dimensional plane is obtained, wherein the fitting line is obtained by performing line fitting according to a correlation coefficient and the projection coordinates of all the contour points of each of the transmission cables on the two-dimensional plane where each of the transmission cables is located, wherein the correlation coefficient represents the linear correlation between the projection coordinates of all the contour points of each of the transmission cables; Calculating the relative distance between the construction machine and each of the transmission cables according to the centroid of the construction machine and the fitting line of each of the transmission cables; Taking the transmission cable corresponding to the minimum distance among the plurality of relative distances as the target transmission cable; Calculating the distance between the three-dimensional coordinates of each of the contour points of the construction machinery and the target transmission cable to obtain a plurality of point-line distances; Using the minimum distance among all the point-line distances as the reference distance between the construction machine and the target transmission cable; The target power transmission cable is identified for potential external damage according to the reference distance.

2. The method for identifying hidden dangers of external damage of power transmission cables according to claim 1, characterized in that: The step of performing stereo matching on each contour point of the same target object according to the position information and image features of each contour point of the same target object in the two target images to determine the three-dimensional coordinates of each contour point of the same target object comprises: For each of the target objects, use any one of the two target images as a main view, use the other one as a secondary view, and use contour points of the target object in the main view and the secondary view as first contour points and second contour points, respectively; For any target first contour point among all the first contour points of the target object, determine each target second contour point on the same horizontal polar line as the target first contour point from all the second contour points; Calculating each matching cost according to the position information and image features of the target first contour point and each target second contour point; Determine the disparity of the target first contour point according to the target first contour point and the target second contour point corresponding to the minimum matching cost among all the matching costs; Calculating the three-dimensional coordinates of the first contour point of the target according to the disparity of the first contour point of the target; All the first contour points of the target object are traversed to finally obtain the three-dimensional coordinates of each contour point of the target object.

3. The method for identifying hidden dangers of external damage of power transmission cables according to claim 2, characterized in that: The step of calculating each matching cost according to the position information and image features of the target first contour point and each target second contour point comprises: Calculating each initial cost according to the position information of the target first contour point and each target second contour point; Calculating a weight of each initial cost according to the image features of the target first contour point and each target second contour point; According to each initial cost and its weight, each matching cost is calculated.

4. The method for identifying hidden dangers of external damage of power transmission cables according to claim 1, characterized in that: The step of obtaining a fitting line projected on a two-dimensional plane by each transmission cable according to the three-dimensional coordinates of all the contour points of each transmission cable comprises: Performing plane fitting according to the three-dimensional coordinates of all the contour points of each of the transmission cables to obtain a two-dimensional plane where each of the transmission cables is located; Projecting each of the contour points of each of the transmission cables onto the two-dimensional plane to obtain each projection coordinate of each of the contour points of each of the transmission cables in the two-dimensional plane; calculating the correlation coefficient; Line fitting is performed according to the correlation coefficient and the projection coordinates of all the contour points of each of the transmission cables to obtain a fitting line of each of the transmission cables projected on a two-dimensional plane.

5. The method for identifying hidden dangers of external damage of power transmission cables according to claim 4, characterized in that: The step of performing line fitting according to the correlation coefficient and the projection coordinates of all the contour points of each of the transmission cables to obtain a fitting line projected on a two-dimensional plane for each of the transmission cables comprises: If the correlation coefficient is greater than a preset threshold, a straight line fitting is performed on the projection coordinates to obtain a fitting straight line and the fitting straight line is used as the fitting line; If the correlation coefficient is less than or equal to the preset threshold, a quadratic polynomial fitting is performed on the projection coordinates to obtain a fitting curve and the fitting curve is used as the fitting line.

6. A device for identifying hidden dangers of external damage to power transmission cables, characterized in that: The device comprises: An acquisition module, used to acquire two target images obtained by photographing the construction machinery and the power transmission cable from different viewing angles, wherein there are multiple power transmission cables; An extraction module, used to extract contours of each target image respectively, to obtain position information and image features of each contour point of each target object in each target image, wherein the target object includes the construction machinery or the transmission cable; A matching module, for performing stereo matching on each contour point of the same target object according to the position information and image features of each contour point of the same target object in the two target images, determining the three-dimensional coordinates of each contour point of the same target object, and obtaining the three-dimensional coordinates of all the contour points of the construction machinery and the three-dimensional coordinates of all the contour points of the transmission cable; The identification module is used to: calculate the centroid of the construction machinery according to the three-dimensional coordinates of all the contour points of the construction machinery; obtain the fitting line of each transmission cable projected on a two-dimensional plane according to the three-dimensional coordinates of all the contour points of each transmission cable, wherein the fitting line is obtained by line fitting according to the correlation coefficient and the projection coordinates of all the contour points of each transmission cable on the two-dimensional plane where each transmission cable is located, and the correlation coefficient represents the linear correlation between the projection coordinates of all the contour points of each transmission cable; calculate the relative distance between the construction machinery and each transmission cable according to the centroid of the construction machinery and the fitting line of each transmission cable; take the transmission cable corresponding to the minimum distance among multiple relative distances as the target transmission cable; calculate the distance between the three-dimensional coordinates of each contour point of the construction machinery and the target transmission cable to obtain multiple point-line distances; take the minimum distance among all the point-line distances as the reference distance between the construction machinery and the target transmission cable; and identify the hidden danger of external damage to the target transmission cable according to the reference distance.

7. An electronic device, characterized in that: It comprises a processor and a memory, the memory is used to store a program, and the processor is used to implement the method for identifying hidden dangers of external damage of a transmission cable as described in any one of claims 1 to 5 when executing the program.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for identifying hidden dangers of external damage of a power transmission cable as described in any one of claims 1 to 5 is implemented.

Images